Webcam capture with OpenCV, Tkinter and multiprocessing: Solved!

First working prototype of webcam capture with OpenCV, Tkinter and multiprocessing.

Warning: It's ugly due to a Tkinter/multiprocessing bug in Linux/Mac.
http://bugs.python.org/issue5527#msg195480

import cv2
import multiprocessing

def cam_loop(the_q, event):
    width, height = 800, 600
    cap = cv2.VideoCapture(0)
    cap.set(cv2.CAP_PROP_FRAME_WIDTH, width)
    cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height)

    while True:
        _ , img = cap.read()
        if img is not None:
            img = cv2.flip(img, 1)
            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGBA)
            the_q.put(img)
            event.set()


if __name__ == '__main__':
    try:
        logger = multiprocessing.log_to_stderr()
        logger.setLevel(multiprocessing.SUBDEBUG)

        the_q = multiprocessing.Queue(1)

        event = multiprocessing.Event()
        cam_process = multiprocessing.Process(target=cam_loop,args=(the_q, event))
        cam_process.start()

        # Bug in Tkinter, must be imported after processes are forked:
        import Tkinter as tk
        from PIL import Image, ImageTk
        class TkGui(tk.Tk):
            def __init__(self):
                tk.Tk.__init__(self, None)

                self.parent = None
                self.bind('', lambda e: self.quit())

                self.lmain = tk.Label(self)
                self.lmain.pack()

            def update_frame(self, img):
                img = Image.fromarray(img)
                imgtk = ImageTk.PhotoImage(image=img)
                self.lmain.imgtk = imgtk
                self.lmain.configure(image=imgtk)
                self.update()

        def gui_mainloop(the_q, event):
            gui = TkGui()
            while True:
                event.wait()
                img = the_q.get()
                gui.update_frame(img)

        gui_mainloop(the_q, event)

        cam_process.join()

    except KeyboardInterrupt:
        cam_process.terminate()

The idea is to have a camera-capture process running in parallel to the gui one.

So, the "cam_loop" method is transformed into a separate process (#27), images are captured from the webcam in a loop, a bit of make-up for them(#13-14),  puts one of them into a queue,  and sends a signal to the event saying: hey! there is something here.

At the same time, the main process has been importing the gui, creates a simple Tkinter root, enters in a hand made "mainloop", and awaits for the signal from the cam_process(#53). Once it receives it, tells the gui to update, so it shows the new image received.

The events are not really needed, but they seem to be a nice way of not wasting cycles in the gui loop, that is, only refresh when there is a frame waiting.

Goods news is that CPU usage is less than previous tests, around 55-60%:

Webcam capture with OpenCV, Tkinter and multiprocessing: __THE_PROCESS_HAS_FORKED

child process calling self.run()The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.

This is what you get when you try to put the three pieces together, OpenCV, Tkinter and multiprocessing.

Take a simple example:

import multiprocessing
import cv2

import Tkinter as tk    # HERE1

def cam_loop(the_q):
    while True:
        the_q.put('foo in the queue')

def show_loop(the_q):
    cv2.VideoCapture(0)  # HERE2
    while True:
        from_queue = the_q.get()
        print from_queue

if __name__ == '__main__':
    try:
        the_q = multiprocessing.Queue(1)

        cam_process = multiprocessing.Process(target=cam_loop,args=(the_q, ))
        cam_process.start()

        show_process = multiprocessing.Process(target=show_loop,args=(the_q, ))
        show_process.start()

        cam_process.join()
        show_loop.join()
    except KeyboardInterrupt:
        cam_process.terminate()
        show_process.terminate()

Odd, even though not really using Tkinter in any place of the code, just by importing it, if you try to run this, it will throw the error above.
Comment out the "import Tkinter", and code will work OK.

Oddx2, it will not break if the line HERE2 is commented out. That is, you can import OpenCV, but when trying to use it inside the process, it will snap.

Seems like cv2 and Tkinter are not really good friends when living in multiprocessing fields.

Found some references:
http://stackoverflow.com/a/19082049/1956309
http://bugs.python.org/issue5527#msg194848

The solution, or workaround to be more precise, is to change the Tikinter import to somewhere after the fork is done, like:

import multiprocessing
import cv2

def cam_loop(the_q):
    while True:
        the_q.put('foo in the queue')

def show_loop(the_q):
    cv2.VideoCapture(0)
    while True:
        from_queue = the_q.get()
        print from_queue

if __name__ == '__main__':

    try:
        the_q = multiprocessing.Queue(1)

        cam_process = multiprocessing.Process(target=cam_loop,args=(the_q, ))
        cam_process.start()

        show_process = multiprocessing.Process(target=show_loop,args=(the_q, ))
        show_process.start()

        import Tkinter as tk
        cam_process.join()
        show_loop.join()
    except KeyboardInterrupt:
        cam_process.terminate()
        show_process.terminate()

Next:
A. flee from Tkinter, explore other guis.
B. Keep trying to make any sense of this.

Webcam capture with OpenCV and multiprocessing II

OpenCV with multiprocessing for a webcam capture using shared memory via Namespace:

import multiprocessing
import cv2


def cam_loop(namespace, event):
    cap = cv2.VideoCapture(0)

    while True:
        _ , img = cap.read()
        if img is not None:
            namespace.value = img
            event.set()

def show_loop(the_q, event):
    cv2.namedWindow('pepe')

    while True:
        event.wait()
        from_queue = namespace.value
        cv2.imshow('pepe', from_queue)
        k = cv2.waitKey(1)
        if k == ord('q') or k == 27:
            break

if __name__ == '__main__':

    logger = multiprocessing.log_to_stderr()
    logger.setLevel(multiprocessing.SUBDEBUG)

    mgr = multiprocessing.Manager()
    namespace = mgr.Namespace()

    event = multiprocessing.Event()

    cam_process = multiprocessing.Process(target=cam_loop,args=(namespace, event))
    cam_process.start()

    show_process = multiprocessing.Process(target=show_loop,args=(namespace, event))
    show_process.start()

    cam_process.join()
    show_process.join()

Difficult to say what is the performance compared with  pipes or queues.

The cpu usage is high, 70+% if you sum up all processes, and I'm not even doing any image processing, nor gui.

I'm beginning to wonder if the multiprocessing via is going to be suitable for the core aspects of behave.

Next: bring Tkinter into this multiprocessing-opencv-webcam-capture scenario.

Webcam capture with OpenCV and multiprocessing.

Well, it seems that multiprocessing is the way to go when you want to squeeze the cores of your cpu.

Some reading on the subject:
https://docs.python.org/2/library/multiprocessing.html
http://pymotw.com/2/multiprocessing/index.html

And basic working version of OpenCV with multiprocessing for a webcam capture using pipes:

import multiprocessing
import cv2


def cam_loop(pipe_parent):
    cap = cv2.VideoCapture(0)

    while True:
        _ , img = cap.read()
        if img is not None:
            pipe_parent.send(img)

def show_loop(pipe_child):
    cv2.namedWindow('pepe')

    while True:
        from_queue = pipe_child.recv()
        cv2.imshow('pepe', from_queue)
        cv2.waitKey(1)

if __name__ == '__main__':

    logger = multiprocessing.log_to_stderr()
    logger.setLevel(multiprocessing.SUBDEBUG)

    pipe_parent, pipe_child = multiprocessing.Pipe()

    cam_process = multiprocessing.Process(target=cam_loop,args=(pipe_parent, ))
    cam_process.start()

    show_process = multiprocessing.Process(target=show_loop,args=(pipe_child, ))
    show_process.start()

    cam_process.join()
    show_loop.join()

This code will create one main process and two child ones, one for each function, plus a pipe that is how the images that one captures, the other consumes.

What about using a queue?:

import multiprocessing
import cv2


def cam_loop(the_q):
    cap = cv2.VideoCapture(0)

    while True:
        _ , img = cap.read()
        if img is not None:
            the_q.put(img)

def show_loop(the_q):
    cv2.namedWindow('pepe')

    while True:
        from_queue = the_q.get()
        cv2.imshow('pepe', from_queue)
        cv2.waitKey(1)

if __name__ == '__main__':

    logger = multiprocessing.log_to_stderr()
    logger.setLevel(multiprocessing.SUBDEBUG)

    the_q = multiprocessing.Queue()

    cam_process = multiprocessing.Process(target=cam_loop,args=(the_q, ))
    cam_process.start()

    show_process = multiprocessing.Process(target=show_loop,args=(the_q, ))
    show_process.start()

    cam_process.join()
    show_loop.join()

The memory of cam_loop child process will grow and grow without limit. My educated guess is that the queue has no limit, so it will just get fatter until it eats all system memory.
How to fix this? limiting the size of the queue at instantiation moment:

    the_q = multiprocessing.Queue(1)

Warning!!!: the bigger the int, the more lag is added to the web capture... it-is-spooky!

Next: compare performance with shared memory.

Warning!:
        cv2.waitKey(1)
A good deal of time can be spent troubleshooting if you forget this, that allows the opencv to process events.

Webcam capture with OpenCV and Tkinter: multithreading

Python, Tkinter, OpenCV, alright! lets check multithreading:

On multithreading with Python, recommended lecture:
http://www.python-course.eu/threads.php
Threading with queues:
http://www.nryoung.org/blog/2013/2/28/python-threading/
Discussion:
http://stackoverflow.com/questions/2846653/python-multithreading-for-dummies

Starting point for multithreading with Tkinter:
http://stackoverflow.com/questions/459083/how-do-you-run-your-own-code-alongside-tkinters-event-loop

Maybe threading is not the answer, GIL:
http://www.jeffknupp.com/blog/2012/03/31/pythons-hardest-problem/

"in any Python program, no matter how many threads and how many processors are present, only one thread is being executed at any time."

So, multiprocessing is the way to go.

Webcam capture with OpenCV and Tkinter: widget.after(...)

So, how to show the webcam using OpenCV and Tkinter?

There seem to be two alternatives, one is using the widget.after(delay, function) method, the other one is multithreading/multiprocessing.

For widget.after option, this is the basic code:

import Tkinter as tk
import cv2
from PIL import Image, ImageTk

width, height = 800, 600
cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, width)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height)

root = tk.Tk()
root.bind('', lambda e: root.quit())
lmain = tk.Label(root)
lmain.pack()

def show_frame():
    _, frame = cap.read()
    frame = cv2.flip(frame, 1)
    cv2image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGBA)
    img = Image.fromarray(cv2image)
    imgtk = ImageTk.PhotoImage(image=img)
    lmain.imgtk = imgtk
    lmain.configure(image=imgtk)
    lmain.after(10, show_frame)

show_frame()
root.mainloop()

The after methods are ways of telling Tkinter to call functions after it has processed the events in the 'mainloop'.
When show_frame (Line 25) is called, a new frame is processed and the final call lmain.after(10, show_frame) outside the mainloop, will register 'show_frame' as after method to be called in the loop with the delay in milliseconds passed (10).

Later on, when inside the mainloop (Line 26), there is a check for if there are any 'after' methods at the end of the event processing, in this case it will find 'show_frame' and call it, the method will 'hook' itself to the mainloop, so it will be called semi-recursively.

So, think like this: the 'show_frame' will tell Tkinter mainloop: 'hey! I'm here, call me back in 10 milliseconds', tkinter will do its normal life, processing events, etc, and then after 10 milliseconds will be obedient and will call 'show_frame'. Method runs its code and at the end does the sneaky thing of saying again 'hey!, I'm here, call me...'

So, in each of those calls to the method, we take a frame captured by OpenCV's camera capturer. And that is the way you are able to capture frames and display them in Tkinker.

There is a bonus track also in all this:

   lmain.imgtk = imgtk

In spite of appearances it is not a superfluous piece of code, it is much needed, since the pseudo-recursive nature of the after call, it would garbage-collect the variable imgtk, we have to anchor it somewhere so it does not get deleted. We could also set it as "global" :|

Last word on widget.after: watch out for performance, the trick is on the delays set to the 'after' method. In this simple code if the delay of 10 is changed to 1, you can see the video capture with constant jumps in frames. With the value of 10 or even 100, it works smoothly.

I've got best results with the delay parameter when is set to more or less the time that takes the VideoCapture to take a frame. That is, we are talking about 'FPS' :) .

Refs to understand the after method:
http://effbot.org/tkinterbook/widget.htm#Tkinter.Widget.after-method

A bit more advanced, good read:

http://stupidpythonideas.blogspot.ie/2013/10/why-your-gui-app-freezes.html

http://stupidpythonideas.blogspot.ie/2013/10/solving-callbacks-for-python-guis.html

New Behave version, user input issues.

New version of Behave is available.

The main difference for the user is that now it allows to auto adjust the face lower limit.

When you press space in the app, it will tell you to sit with your back straight for 5 seconds, during that time it will save your face position, calculate average position and size, and set the a lower limit for you, no manual input needed. And enforce it nicely :)

The code has also evolved a good deal, what started as basic functionality with initialisation of needed bits with a while(true) loop, moved towards being more modular via creating functions, and then pursuing further modularity using classes, and as last stage: expanding the user input towards a gui-event based paradigm.

What I'm finding is that more and more the user interface expands, the more complex the code becomes, losing part of its original orthogonality, that is, when I need to add a new handle for specific event, I usually need to touch other parts of the code to accommodate the new functionality.

What is happening?

The application is especially vulnerable to events that require different handling depending on the context... The code is littered with a multitude of global variables and flags that serve only one purpose -- handling the context.... With this representation, the context of the computation is represented ambiguously, so it is difficult to tell precisely in which mode the application is at any given time. Actually, the application has no notion of any single mode of operation, but rather tightly coupled and overlapping conditions of operation determined by values of the global variables and flags.

Found in Who moved my state, that reveals the weaknesses of event-driven programming and advocates the use of finite state machines.

As I see it, Behave's code should move from a main loop reacting to user keystrokes and mouse clicks, towards two separate modules with their respective loops in different threads.  One would run the frame capturing and what to do with it, the other would create the gui and facilitate start/stop and interact/modify with the first.

I'll be digging into finite state machines in the next days and how to implement them with Behave.

Some reading done:
http://infohost.nmt.edu/tcc/help/pubs/tkinter/tkinter.pdf
http://www.astro.washington.edu/users/rowen/ROTKFolklore.html
http://www.astro.washington.edu/users/rowen/TkinterSummary.html#Resources
http://www.ferg.org/thinking_in_tkinter/all_programs.html
Tutorial on guis using Tkinter and Python:
http://www.alan-g.me.uk/l2p/tutevent.htm
http://www.alan-g.me.uk/l2p/tutgui.htm

Finite state machine implementation in Python:
http://www.python-course.eu/finite_state_machine.php
These guys have also a Tkinter tutorial:
http://www.python-course.eu/python_tkinter.php