Face detection with OpenCV, Tkinter and multiprocessing: Performace

So it works: code for face detection using multiple processes in parallel with Tkinter; but what about performance?

Dramatis personæ:

- 21773 is the 'Master' process that creates the pipes and two child processes:

- 21774 plays as Tkinter.

- 21777 role is OpenCv, camera capture and face detection.

Activity with face detection and showing the capture in the gui:

CPU is 73% more or less, you can notice the system is working hard.


Activity without showing the frames in the gui:



Overall CPU reduces to aprox. 64%.
Master (21773 ) process spikes up to 60% (from 26%) --> NOK

Tkinter (21774) reduces usage from 22% to 6%. --> OK

OpenCv (21773) does not vary --> OK

Something is not right here, Master should not be working harder if there are no frames to put in the gui.

Most probably its loop runs more than it needs to.

Activity without the capture process, that is, only the gui running and the controller:



Minimal activity --> OK

Ideas to improve performance:
- Show webcam only in gray colours.
- Reduce resolution, these tests are capturing at 640x480
- Reduce the after() call frequency, so there are less updates requested to Tkinter.

Webcam capture with OpenCV and multiprocessing, gui crossroads.

So, right now I have a basic version of Behave with multiprocessing and I ask myself: is Tkinter the right choice?

Good comparative with different Python guis:
http://www.pythoncentral.io/introduction-python-gui-development/

Options
http://wxpython.org/what.php
http://zetcode.com/gui/pyqt4/
http://kivy.org/
http://www.cosc.canterbury.ac.nz/greg.ewing/python_gui/
Behave needs a lightweight gui, simple and flexible, since there is nothing out there that gives much more than what I have, I'll continue tinkering.

About the 90's look of Tkinter, it seems you can give Tkinter a more modern one:
https://docs.python.org/dev/library/tkinter.ttk.html#module-tkinter.ttk

But with Python 3.x, maybe this is a good moment for me to move away from my comfort zone, 2.6 and 2.7 ?

Webcam capture with OpenCV, Tkinter and multiprocessing: parallel processes for webcam and gui.

This post shows how to use multiprocessing to create two processes working in parallel, one for OpenCV that does webcam capture, and the other with Tkinter to show the frames captured by the first.

This is the module called 'proc_capturer.py' with OpenCV code, capturing frames from the webcam and putting them in the queue.

def cam_loop(the_q, event):
    import cv2
    width, height = 800, 600
    cap = cv2.VideoCapture(0)
    cap.set(cv2.CAP_PROP_FRAME_WIDTH, width)
    cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height)

    while True:
        _ , img = cap.read()
        if img is not None:
            img = cv2.flip(img, 1)
            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGBA)
            the_q.put(img)
            event.set()

And the Tkinter's module 'proc_tk_gui.py', starting the gui, receiving the frames from the queue and processing them in a hand-made 'mainloop':

def gui_mainloop(the_q, event):
    import Tkinter as tk
    from PIL import Image, ImageTk

    class TkGui(tk.Tk):
        def __init__(self):
            tk.Tk.__init__(self, None)

            self.parent = None

            self.lmain = tk.Label(self)

            self.lmain.pack()

        def update_frame(self, img):
            img = Image.fromarray(img)
            imgtk = ImageTk.PhotoImage(image=img)
            self.lmain.imgtk = imgtk
            self.lmain.configure(image=imgtk)
            self.update()

    gui = TkGui()

    while True:
        event.wait()
        img = the_q.get()
        gui.update_frame(img)

Finally, your 'multiprocess-orchestrator' module:

import multiprocessing
from proc_tk_gui import gui_mainloop
from proc_capturer import cam_loop

if __name__ == '__main__':
    logger = multiprocessing.log_to_stderr()
    logger.setLevel(multiprocessing.SUBDEBUG)

    q_frames_captured = multiprocessing.Queue(1)
    e_frame_captured = multiprocessing.Event()

    p_cap = multiprocessing.Process(target=cam_loop,args=(q_frames_captured, e_frame_captured))
    p_gui = multiprocessing.Process(target=gui_mainloop,args=(q_frames_captured, e_frame_captured))

    try:
        p_cap.start()
        p_gui.start()

        p_cap.join()
        p_gui.join()

    except KeyboardInterrupt:
        p_cap.terminate()
        p_gui.terminate()

I like the idea of having a master module with the multiprocessing logic, and two independent parallel processes only linked with queues and event handlers.
This could develop into a more finite-state machine approach for my project, behave.

Thanks for this good inspiration:
http://www.briansimulator.org/docs/examples-multiprocessing_multiple_runs_with_gui.html


Edit:
It seems from comments in forums that doing your own update loop for tkinter is not really a good thing. The problem is that even though you can force Tkinter to "refresh the window", when it comes to handle events like mouse or button clicking without the mainloop(): you-are-fried. So, mainloop version of 'proc_tk_gui.py' using my dreaded "tkinter after method":

def gui_mainloop(the_q, event):
    import Tkinter as tk
    from PIL import Image, ImageTk

    class TkGui(tk.Tk):
        def __init__(self):
            tk.Tk.__init__(self, None)

            self.parent = None

            self.lmain = tk.Label(self)

            self.lmain.pack()

        def update_frame(self, the_q, the_e):
            img = the_q.get()
            img = Image.fromarray(img)
            imgtk = ImageTk.PhotoImage(image=img)
            self.lmain.imgtk = imgtk        # remember you need to anchor the image!!!
            self.lmain.configure(image=imgtk)
            self.lmain.after(10, self.update_frame, the_q, the_e)

    gui = TkGui()
    gui.update_frame(img, event)
    gui.mainloop()

Webcam capture with OpenCV, Tkinter and multiprocessing: Solved!

First working prototype of webcam capture with OpenCV, Tkinter and multiprocessing.

Warning: It's ugly due to a Tkinter/multiprocessing bug in Linux/Mac.
http://bugs.python.org/issue5527#msg195480

import cv2
import multiprocessing

def cam_loop(the_q, event):
    width, height = 800, 600
    cap = cv2.VideoCapture(0)
    cap.set(cv2.CAP_PROP_FRAME_WIDTH, width)
    cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height)

    while True:
        _ , img = cap.read()
        if img is not None:
            img = cv2.flip(img, 1)
            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGBA)
            the_q.put(img)
            event.set()


if __name__ == '__main__':
    try:
        logger = multiprocessing.log_to_stderr()
        logger.setLevel(multiprocessing.SUBDEBUG)

        the_q = multiprocessing.Queue(1)

        event = multiprocessing.Event()
        cam_process = multiprocessing.Process(target=cam_loop,args=(the_q, event))
        cam_process.start()

        # Bug in Tkinter, must be imported after processes are forked:
        import Tkinter as tk
        from PIL import Image, ImageTk
        class TkGui(tk.Tk):
            def __init__(self):
                tk.Tk.__init__(self, None)

                self.parent = None
                self.bind('', lambda e: self.quit())

                self.lmain = tk.Label(self)
                self.lmain.pack()

            def update_frame(self, img):
                img = Image.fromarray(img)
                imgtk = ImageTk.PhotoImage(image=img)
                self.lmain.imgtk = imgtk
                self.lmain.configure(image=imgtk)
                self.update()

        def gui_mainloop(the_q, event):
            gui = TkGui()
            while True:
                event.wait()
                img = the_q.get()
                gui.update_frame(img)

        gui_mainloop(the_q, event)

        cam_process.join()

    except KeyboardInterrupt:
        cam_process.terminate()

The idea is to have a camera-capture process running in parallel to the gui one.

So, the "cam_loop" method is transformed into a separate process (#27), images are captured from the webcam in a loop, a bit of make-up for them(#13-14),  puts one of them into a queue,  and sends a signal to the event saying: hey! there is something here.

At the same time, the main process has been importing the gui, creates a simple Tkinter root, enters in a hand made "mainloop", and awaits for the signal from the cam_process(#53). Once it receives it, tells the gui to update, so it shows the new image received.

The events are not really needed, but they seem to be a nice way of not wasting cycles in the gui loop, that is, only refresh when there is a frame waiting.

Goods news is that CPU usage is less than previous tests, around 55-60%:

Webcam capture with OpenCV, Tkinter and multiprocessing: __THE_PROCESS_HAS_FORKED

child process calling self.run()The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.

This is what you get when you try to put the three pieces together, OpenCV, Tkinter and multiprocessing.

Take a simple example:

import multiprocessing
import cv2

import Tkinter as tk    # HERE1

def cam_loop(the_q):
    while True:
        the_q.put('foo in the queue')

def show_loop(the_q):
    cv2.VideoCapture(0)  # HERE2
    while True:
        from_queue = the_q.get()
        print from_queue

if __name__ == '__main__':
    try:
        the_q = multiprocessing.Queue(1)

        cam_process = multiprocessing.Process(target=cam_loop,args=(the_q, ))
        cam_process.start()

        show_process = multiprocessing.Process(target=show_loop,args=(the_q, ))
        show_process.start()

        cam_process.join()
        show_loop.join()
    except KeyboardInterrupt:
        cam_process.terminate()
        show_process.terminate()

Odd, even though not really using Tkinter in any place of the code, just by importing it, if you try to run this, it will throw the error above.
Comment out the "import Tkinter", and code will work OK.

Oddx2, it will not break if the line HERE2 is commented out. That is, you can import OpenCV, but when trying to use it inside the process, it will snap.

Seems like cv2 and Tkinter are not really good friends when living in multiprocessing fields.

Found some references:
http://stackoverflow.com/a/19082049/1956309
http://bugs.python.org/issue5527#msg194848

The solution, or workaround to be more precise, is to change the Tikinter import to somewhere after the fork is done, like:

import multiprocessing
import cv2

def cam_loop(the_q):
    while True:
        the_q.put('foo in the queue')

def show_loop(the_q):
    cv2.VideoCapture(0)
    while True:
        from_queue = the_q.get()
        print from_queue

if __name__ == '__main__':

    try:
        the_q = multiprocessing.Queue(1)

        cam_process = multiprocessing.Process(target=cam_loop,args=(the_q, ))
        cam_process.start()

        show_process = multiprocessing.Process(target=show_loop,args=(the_q, ))
        show_process.start()

        import Tkinter as tk
        cam_process.join()
        show_loop.join()
    except KeyboardInterrupt:
        cam_process.terminate()
        show_process.terminate()

Next:
A. flee from Tkinter, explore other guis.
B. Keep trying to make any sense of this.

Webcam capture with OpenCV and multiprocessing II

OpenCV with multiprocessing for a webcam capture using shared memory via Namespace:

import multiprocessing
import cv2


def cam_loop(namespace, event):
    cap = cv2.VideoCapture(0)

    while True:
        _ , img = cap.read()
        if img is not None:
            namespace.value = img
            event.set()

def show_loop(the_q, event):
    cv2.namedWindow('pepe')

    while True:
        event.wait()
        from_queue = namespace.value
        cv2.imshow('pepe', from_queue)
        k = cv2.waitKey(1)
        if k == ord('q') or k == 27:
            break

if __name__ == '__main__':

    logger = multiprocessing.log_to_stderr()
    logger.setLevel(multiprocessing.SUBDEBUG)

    mgr = multiprocessing.Manager()
    namespace = mgr.Namespace()

    event = multiprocessing.Event()

    cam_process = multiprocessing.Process(target=cam_loop,args=(namespace, event))
    cam_process.start()

    show_process = multiprocessing.Process(target=show_loop,args=(namespace, event))
    show_process.start()

    cam_process.join()
    show_process.join()

Difficult to say what is the performance compared with  pipes or queues.

The cpu usage is high, 70+% if you sum up all processes, and I'm not even doing any image processing, nor gui.

I'm beginning to wonder if the multiprocessing via is going to be suitable for the core aspects of behave.

Next: bring Tkinter into this multiprocessing-opencv-webcam-capture scenario.

Webcam capture with OpenCV and multiprocessing.

Well, it seems that multiprocessing is the way to go when you want to squeeze the cores of your cpu.

Some reading on the subject:
https://docs.python.org/2/library/multiprocessing.html
http://pymotw.com/2/multiprocessing/index.html

And basic working version of OpenCV with multiprocessing for a webcam capture using pipes:

import multiprocessing
import cv2


def cam_loop(pipe_parent):
    cap = cv2.VideoCapture(0)

    while True:
        _ , img = cap.read()
        if img is not None:
            pipe_parent.send(img)

def show_loop(pipe_child):
    cv2.namedWindow('pepe')

    while True:
        from_queue = pipe_child.recv()
        cv2.imshow('pepe', from_queue)
        cv2.waitKey(1)

if __name__ == '__main__':

    logger = multiprocessing.log_to_stderr()
    logger.setLevel(multiprocessing.SUBDEBUG)

    pipe_parent, pipe_child = multiprocessing.Pipe()

    cam_process = multiprocessing.Process(target=cam_loop,args=(pipe_parent, ))
    cam_process.start()

    show_process = multiprocessing.Process(target=show_loop,args=(pipe_child, ))
    show_process.start()

    cam_process.join()
    show_loop.join()

This code will create one main process and two child ones, one for each function, plus a pipe that is how the images that one captures, the other consumes.

What about using a queue?:

import multiprocessing
import cv2


def cam_loop(the_q):
    cap = cv2.VideoCapture(0)

    while True:
        _ , img = cap.read()
        if img is not None:
            the_q.put(img)

def show_loop(the_q):
    cv2.namedWindow('pepe')

    while True:
        from_queue = the_q.get()
        cv2.imshow('pepe', from_queue)
        cv2.waitKey(1)

if __name__ == '__main__':

    logger = multiprocessing.log_to_stderr()
    logger.setLevel(multiprocessing.SUBDEBUG)

    the_q = multiprocessing.Queue()

    cam_process = multiprocessing.Process(target=cam_loop,args=(the_q, ))
    cam_process.start()

    show_process = multiprocessing.Process(target=show_loop,args=(the_q, ))
    show_process.start()

    cam_process.join()
    show_loop.join()

The memory of cam_loop child process will grow and grow without limit. My educated guess is that the queue has no limit, so it will just get fatter until it eats all system memory.
How to fix this? limiting the size of the queue at instantiation moment:

    the_q = multiprocessing.Queue(1)

Warning!!!: the bigger the int, the more lag is added to the web capture... it-is-spooky!

Next: compare performance with shared memory.

Warning!:
        cv2.waitKey(1)
A good deal of time can be spent troubleshooting if you forget this, that allows the opencv to process events.