Face detection with OpenCV, Tkinter and multiprocessing: Performace

So it works: code for face detection using multiple processes in parallel with Tkinter; but what about performance?

Dramatis personæ:

- 21773 is the 'Master' process that creates the pipes and two child processes:

- 21774 plays as Tkinter.

- 21777 role is OpenCv, camera capture and face detection.

Activity with face detection and showing the capture in the gui:

CPU is 73% more or less, you can notice the system is working hard.


Activity without showing the frames in the gui:



Overall CPU reduces to aprox. 64%.
Master (21773 ) process spikes up to 60% (from 26%) --> NOK

Tkinter (21774) reduces usage from 22% to 6%. --> OK

OpenCv (21773) does not vary --> OK

Something is not right here, Master should not be working harder if there are no frames to put in the gui.

Most probably its loop runs more than it needs to.

Activity without the capture process, that is, only the gui running and the controller:



Minimal activity --> OK

Ideas to improve performance:
- Show webcam only in gray colours.
- Reduce resolution, these tests are capturing at 640x480
- Reduce the after() call frequency, so there are less updates requested to Tkinter.

Webcam capture with OpenCV and multiprocessing, gui crossroads.

So, right now I have a basic version of Behave with multiprocessing and I ask myself: is Tkinter the right choice?

Good comparative with different Python guis:
http://www.pythoncentral.io/introduction-python-gui-development/

Options
http://wxpython.org/what.php
http://zetcode.com/gui/pyqt4/
http://kivy.org/
http://www.cosc.canterbury.ac.nz/greg.ewing/python_gui/
Behave needs a lightweight gui, simple and flexible, since there is nothing out there that gives much more than what I have, I'll continue tinkering.

About the 90's look of Tkinter, it seems you can give Tkinter a more modern one:
https://docs.python.org/dev/library/tkinter.ttk.html#module-tkinter.ttk

But with Python 3.x, maybe this is a good moment for me to move away from my comfort zone, 2.6 and 2.7 ?

Webcam capture with OpenCV, Tkinter and multiprocessing: parallel processes for webcam and gui.

This post shows how to use multiprocessing to create two processes working in parallel, one for OpenCV that does webcam capture, and the other with Tkinter to show the frames captured by the first.

This is the module called 'proc_capturer.py' with OpenCV code, capturing frames from the webcam and putting them in the queue.

def cam_loop(the_q, event):
    import cv2
    width, height = 800, 600
    cap = cv2.VideoCapture(0)
    cap.set(cv2.CAP_PROP_FRAME_WIDTH, width)
    cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height)

    while True:
        _ , img = cap.read()
        if img is not None:
            img = cv2.flip(img, 1)
            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGBA)
            the_q.put(img)
            event.set()

And the Tkinter's module 'proc_tk_gui.py', starting the gui, receiving the frames from the queue and processing them in a hand-made 'mainloop':

def gui_mainloop(the_q, event):
    import Tkinter as tk
    from PIL import Image, ImageTk

    class TkGui(tk.Tk):
        def __init__(self):
            tk.Tk.__init__(self, None)

            self.parent = None

            self.lmain = tk.Label(self)

            self.lmain.pack()

        def update_frame(self, img):
            img = Image.fromarray(img)
            imgtk = ImageTk.PhotoImage(image=img)
            self.lmain.imgtk = imgtk
            self.lmain.configure(image=imgtk)
            self.update()

    gui = TkGui()

    while True:
        event.wait()
        img = the_q.get()
        gui.update_frame(img)

Finally, your 'multiprocess-orchestrator' module:

import multiprocessing
from proc_tk_gui import gui_mainloop
from proc_capturer import cam_loop

if __name__ == '__main__':
    logger = multiprocessing.log_to_stderr()
    logger.setLevel(multiprocessing.SUBDEBUG)

    q_frames_captured = multiprocessing.Queue(1)
    e_frame_captured = multiprocessing.Event()

    p_cap = multiprocessing.Process(target=cam_loop,args=(q_frames_captured, e_frame_captured))
    p_gui = multiprocessing.Process(target=gui_mainloop,args=(q_frames_captured, e_frame_captured))

    try:
        p_cap.start()
        p_gui.start()

        p_cap.join()
        p_gui.join()

    except KeyboardInterrupt:
        p_cap.terminate()
        p_gui.terminate()

I like the idea of having a master module with the multiprocessing logic, and two independent parallel processes only linked with queues and event handlers.
This could develop into a more finite-state machine approach for my project, behave.

Thanks for this good inspiration:
http://www.briansimulator.org/docs/examples-multiprocessing_multiple_runs_with_gui.html


Edit:
It seems from comments in forums that doing your own update loop for tkinter is not really a good thing. The problem is that even though you can force Tkinter to "refresh the window", when it comes to handle events like mouse or button clicking without the mainloop(): you-are-fried. So, mainloop version of 'proc_tk_gui.py' using my dreaded "tkinter after method":

def gui_mainloop(the_q, event):
    import Tkinter as tk
    from PIL import Image, ImageTk

    class TkGui(tk.Tk):
        def __init__(self):
            tk.Tk.__init__(self, None)

            self.parent = None

            self.lmain = tk.Label(self)

            self.lmain.pack()

        def update_frame(self, the_q, the_e):
            img = the_q.get()
            img = Image.fromarray(img)
            imgtk = ImageTk.PhotoImage(image=img)
            self.lmain.imgtk = imgtk        # remember you need to anchor the image!!!
            self.lmain.configure(image=imgtk)
            self.lmain.after(10, self.update_frame, the_q, the_e)

    gui = TkGui()
    gui.update_frame(img, event)
    gui.mainloop()

Webcam capture with OpenCV, Tkinter and multiprocessing: Solved!

First working prototype of webcam capture with OpenCV, Tkinter and multiprocessing.

Warning: It's ugly due to a Tkinter/multiprocessing bug in Linux/Mac.
http://bugs.python.org/issue5527#msg195480

import cv2
import multiprocessing

def cam_loop(the_q, event):
    width, height = 800, 600
    cap = cv2.VideoCapture(0)
    cap.set(cv2.CAP_PROP_FRAME_WIDTH, width)
    cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height)

    while True:
        _ , img = cap.read()
        if img is not None:
            img = cv2.flip(img, 1)
            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGBA)
            the_q.put(img)
            event.set()


if __name__ == '__main__':
    try:
        logger = multiprocessing.log_to_stderr()
        logger.setLevel(multiprocessing.SUBDEBUG)

        the_q = multiprocessing.Queue(1)

        event = multiprocessing.Event()
        cam_process = multiprocessing.Process(target=cam_loop,args=(the_q, event))
        cam_process.start()

        # Bug in Tkinter, must be imported after processes are forked:
        import Tkinter as tk
        from PIL import Image, ImageTk
        class TkGui(tk.Tk):
            def __init__(self):
                tk.Tk.__init__(self, None)

                self.parent = None
                self.bind('', lambda e: self.quit())

                self.lmain = tk.Label(self)
                self.lmain.pack()

            def update_frame(self, img):
                img = Image.fromarray(img)
                imgtk = ImageTk.PhotoImage(image=img)
                self.lmain.imgtk = imgtk
                self.lmain.configure(image=imgtk)
                self.update()

        def gui_mainloop(the_q, event):
            gui = TkGui()
            while True:
                event.wait()
                img = the_q.get()
                gui.update_frame(img)

        gui_mainloop(the_q, event)

        cam_process.join()

    except KeyboardInterrupt:
        cam_process.terminate()

The idea is to have a camera-capture process running in parallel to the gui one.

So, the "cam_loop" method is transformed into a separate process (#27), images are captured from the webcam in a loop, a bit of make-up for them(#13-14),  puts one of them into a queue,  and sends a signal to the event saying: hey! there is something here.

At the same time, the main process has been importing the gui, creates a simple Tkinter root, enters in a hand made "mainloop", and awaits for the signal from the cam_process(#53). Once it receives it, tells the gui to update, so it shows the new image received.

The events are not really needed, but they seem to be a nice way of not wasting cycles in the gui loop, that is, only refresh when there is a frame waiting.

Goods news is that CPU usage is less than previous tests, around 55-60%:

Webcam capture with OpenCV, Tkinter and multiprocessing: __THE_PROCESS_HAS_FORKED

child process calling self.run()The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.

This is what you get when you try to put the three pieces together, OpenCV, Tkinter and multiprocessing.

Take a simple example:

import multiprocessing
import cv2

import Tkinter as tk    # HERE1

def cam_loop(the_q):
    while True:
        the_q.put('foo in the queue')

def show_loop(the_q):
    cv2.VideoCapture(0)  # HERE2
    while True:
        from_queue = the_q.get()
        print from_queue

if __name__ == '__main__':
    try:
        the_q = multiprocessing.Queue(1)

        cam_process = multiprocessing.Process(target=cam_loop,args=(the_q, ))
        cam_process.start()

        show_process = multiprocessing.Process(target=show_loop,args=(the_q, ))
        show_process.start()

        cam_process.join()
        show_loop.join()
    except KeyboardInterrupt:
        cam_process.terminate()
        show_process.terminate()

Odd, even though not really using Tkinter in any place of the code, just by importing it, if you try to run this, it will throw the error above.
Comment out the "import Tkinter", and code will work OK.

Oddx2, it will not break if the line HERE2 is commented out. That is, you can import OpenCV, but when trying to use it inside the process, it will snap.

Seems like cv2 and Tkinter are not really good friends when living in multiprocessing fields.

Found some references:
http://stackoverflow.com/a/19082049/1956309
http://bugs.python.org/issue5527#msg194848

The solution, or workaround to be more precise, is to change the Tikinter import to somewhere after the fork is done, like:

import multiprocessing
import cv2

def cam_loop(the_q):
    while True:
        the_q.put('foo in the queue')

def show_loop(the_q):
    cv2.VideoCapture(0)
    while True:
        from_queue = the_q.get()
        print from_queue

if __name__ == '__main__':

    try:
        the_q = multiprocessing.Queue(1)

        cam_process = multiprocessing.Process(target=cam_loop,args=(the_q, ))
        cam_process.start()

        show_process = multiprocessing.Process(target=show_loop,args=(the_q, ))
        show_process.start()

        import Tkinter as tk
        cam_process.join()
        show_loop.join()
    except KeyboardInterrupt:
        cam_process.terminate()
        show_process.terminate()

Next:
A. flee from Tkinter, explore other guis.
B. Keep trying to make any sense of this.

Webcam capture with OpenCV and multiprocessing II

OpenCV with multiprocessing for a webcam capture using shared memory via Namespace:

import multiprocessing
import cv2


def cam_loop(namespace, event):
    cap = cv2.VideoCapture(0)

    while True:
        _ , img = cap.read()
        if img is not None:
            namespace.value = img
            event.set()

def show_loop(the_q, event):
    cv2.namedWindow('pepe')

    while True:
        event.wait()
        from_queue = namespace.value
        cv2.imshow('pepe', from_queue)
        k = cv2.waitKey(1)
        if k == ord('q') or k == 27:
            break

if __name__ == '__main__':

    logger = multiprocessing.log_to_stderr()
    logger.setLevel(multiprocessing.SUBDEBUG)

    mgr = multiprocessing.Manager()
    namespace = mgr.Namespace()

    event = multiprocessing.Event()

    cam_process = multiprocessing.Process(target=cam_loop,args=(namespace, event))
    cam_process.start()

    show_process = multiprocessing.Process(target=show_loop,args=(namespace, event))
    show_process.start()

    cam_process.join()
    show_process.join()

Difficult to say what is the performance compared with  pipes or queues.

The cpu usage is high, 70+% if you sum up all processes, and I'm not even doing any image processing, nor gui.

I'm beginning to wonder if the multiprocessing via is going to be suitable for the core aspects of behave.

Next: bring Tkinter into this multiprocessing-opencv-webcam-capture scenario.

Webcam capture with OpenCV and multiprocessing.

Well, it seems that multiprocessing is the way to go when you want to squeeze the cores of your cpu.

Some reading on the subject:
https://docs.python.org/2/library/multiprocessing.html
http://pymotw.com/2/multiprocessing/index.html

And basic working version of OpenCV with multiprocessing for a webcam capture using pipes:

import multiprocessing
import cv2


def cam_loop(pipe_parent):
    cap = cv2.VideoCapture(0)

    while True:
        _ , img = cap.read()
        if img is not None:
            pipe_parent.send(img)

def show_loop(pipe_child):
    cv2.namedWindow('pepe')

    while True:
        from_queue = pipe_child.recv()
        cv2.imshow('pepe', from_queue)
        cv2.waitKey(1)

if __name__ == '__main__':

    logger = multiprocessing.log_to_stderr()
    logger.setLevel(multiprocessing.SUBDEBUG)

    pipe_parent, pipe_child = multiprocessing.Pipe()

    cam_process = multiprocessing.Process(target=cam_loop,args=(pipe_parent, ))
    cam_process.start()

    show_process = multiprocessing.Process(target=show_loop,args=(pipe_child, ))
    show_process.start()

    cam_process.join()
    show_loop.join()

This code will create one main process and two child ones, one for each function, plus a pipe that is how the images that one captures, the other consumes.

What about using a queue?:

import multiprocessing
import cv2


def cam_loop(the_q):
    cap = cv2.VideoCapture(0)

    while True:
        _ , img = cap.read()
        if img is not None:
            the_q.put(img)

def show_loop(the_q):
    cv2.namedWindow('pepe')

    while True:
        from_queue = the_q.get()
        cv2.imshow('pepe', from_queue)
        cv2.waitKey(1)

if __name__ == '__main__':

    logger = multiprocessing.log_to_stderr()
    logger.setLevel(multiprocessing.SUBDEBUG)

    the_q = multiprocessing.Queue()

    cam_process = multiprocessing.Process(target=cam_loop,args=(the_q, ))
    cam_process.start()

    show_process = multiprocessing.Process(target=show_loop,args=(the_q, ))
    show_process.start()

    cam_process.join()
    show_loop.join()

The memory of cam_loop child process will grow and grow without limit. My educated guess is that the queue has no limit, so it will just get fatter until it eats all system memory.
How to fix this? limiting the size of the queue at instantiation moment:

    the_q = multiprocessing.Queue(1)

Warning!!!: the bigger the int, the more lag is added to the web capture... it-is-spooky!

Next: compare performance with shared memory.

Warning!:
        cv2.waitKey(1)
A good deal of time can be spent troubleshooting if you forget this, that allows the opencv to process events.

Webcam capture with OpenCV and Tkinter: multithreading

Python, Tkinter, OpenCV, alright! lets check multithreading:

On multithreading with Python, recommended lecture:
http://www.python-course.eu/threads.php
Threading with queues:
http://www.nryoung.org/blog/2013/2/28/python-threading/
Discussion:
http://stackoverflow.com/questions/2846653/python-multithreading-for-dummies

Starting point for multithreading with Tkinter:
http://stackoverflow.com/questions/459083/how-do-you-run-your-own-code-alongside-tkinters-event-loop

Maybe threading is not the answer, GIL:
http://www.jeffknupp.com/blog/2012/03/31/pythons-hardest-problem/

"in any Python program, no matter how many threads and how many processors are present, only one thread is being executed at any time."

So, multiprocessing is the way to go.

Webcam capture with OpenCV and Tkinter: widget.after(...)

So, how to show the webcam using OpenCV and Tkinter?

There seem to be two alternatives, one is using the widget.after(delay, function) method, the other one is multithreading/multiprocessing.

For widget.after option, this is the basic code:

import Tkinter as tk
import cv2
from PIL import Image, ImageTk

width, height = 800, 600
cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, width)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height)

root = tk.Tk()
root.bind('', lambda e: root.quit())
lmain = tk.Label(root)
lmain.pack()

def show_frame():
    _, frame = cap.read()
    frame = cv2.flip(frame, 1)
    cv2image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGBA)
    img = Image.fromarray(cv2image)
    imgtk = ImageTk.PhotoImage(image=img)
    lmain.imgtk = imgtk
    lmain.configure(image=imgtk)
    lmain.after(10, show_frame)

show_frame()
root.mainloop()

The after methods are ways of telling Tkinter to call functions after it has processed the events in the 'mainloop'.
When show_frame (Line 25) is called, a new frame is processed and the final call lmain.after(10, show_frame) outside the mainloop, will register 'show_frame' as after method to be called in the loop with the delay in milliseconds passed (10).

Later on, when inside the mainloop (Line 26), there is a check for if there are any 'after' methods at the end of the event processing, in this case it will find 'show_frame' and call it, the method will 'hook' itself to the mainloop, so it will be called semi-recursively.

So, think like this: the 'show_frame' will tell Tkinter mainloop: 'hey! I'm here, call me back in 10 milliseconds', tkinter will do its normal life, processing events, etc, and then after 10 milliseconds will be obedient and will call 'show_frame'. Method runs its code and at the end does the sneaky thing of saying again 'hey!, I'm here, call me...'

So, in each of those calls to the method, we take a frame captured by OpenCV's camera capturer. And that is the way you are able to capture frames and display them in Tkinker.

There is a bonus track also in all this:

   lmain.imgtk = imgtk

In spite of appearances it is not a superfluous piece of code, it is much needed, since the pseudo-recursive nature of the after call, it would garbage-collect the variable imgtk, we have to anchor it somewhere so it does not get deleted. We could also set it as "global" :|

Last word on widget.after: watch out for performance, the trick is on the delays set to the 'after' method. In this simple code if the delay of 10 is changed to 1, you can see the video capture with constant jumps in frames. With the value of 10 or even 100, it works smoothly.

I've got best results with the delay parameter when is set to more or less the time that takes the VideoCapture to take a frame. That is, we are talking about 'FPS' :) .

Refs to understand the after method:
http://effbot.org/tkinterbook/widget.htm#Tkinter.Widget.after-method

A bit more advanced, good read:

http://stupidpythonideas.blogspot.ie/2013/10/why-your-gui-app-freezes.html

http://stupidpythonideas.blogspot.ie/2013/10/solving-callbacks-for-python-guis.html

New Behave version, user input issues.

New version of Behave is available.

The main difference for the user is that now it allows to auto adjust the face lower limit.

When you press space in the app, it will tell you to sit with your back straight for 5 seconds, during that time it will save your face position, calculate average position and size, and set the a lower limit for you, no manual input needed. And enforce it nicely :)

The code has also evolved a good deal, what started as basic functionality with initialisation of needed bits with a while(true) loop, moved towards being more modular via creating functions, and then pursuing further modularity using classes, and as last stage: expanding the user input towards a gui-event based paradigm.

What I'm finding is that more and more the user interface expands, the more complex the code becomes, losing part of its original orthogonality, that is, when I need to add a new handle for specific event, I usually need to touch other parts of the code to accommodate the new functionality.

What is happening?

The application is especially vulnerable to events that require different handling depending on the context... The code is littered with a multitude of global variables and flags that serve only one purpose -- handling the context.... With this representation, the context of the computation is represented ambiguously, so it is difficult to tell precisely in which mode the application is at any given time. Actually, the application has no notion of any single mode of operation, but rather tightly coupled and overlapping conditions of operation determined by values of the global variables and flags.

Found in Who moved my state, that reveals the weaknesses of event-driven programming and advocates the use of finite state machines.

As I see it, Behave's code should move from a main loop reacting to user keystrokes and mouse clicks, towards two separate modules with their respective loops in different threads.  One would run the frame capturing and what to do with it, the other would create the gui and facilitate start/stop and interact/modify with the first.

I'll be digging into finite state machines in the next days and how to implement them with Behave.

Some reading done:
http://infohost.nmt.edu/tcc/help/pubs/tkinter/tkinter.pdf
http://www.astro.washington.edu/users/rowen/ROTKFolklore.html
http://www.astro.washington.edu/users/rowen/TkinterSummary.html#Resources
http://www.ferg.org/thinking_in_tkinter/all_programs.html
Tutorial on guis using Tkinter and Python:
http://www.alan-g.me.uk/l2p/tutevent.htm
http://www.alan-g.me.uk/l2p/tutgui.htm

Finite state machine implementation in Python:
http://www.python-course.eu/finite_state_machine.php
These guys have also a Tkinter tutorial:
http://www.python-course.eu/python_tkinter.php

Solved: cx_Freeze and OpenCV, app is portable!

Just managed to port my python OpenCv face detection app, Behave, to another mac and run ok using cx_freeze.

The trick is to copy the /lib directory from the opencv installation, into the application directory, then tell cx_freeze to use it.

In the base directory of your app:

mkdir lib
cp /usr/opencv-64/lib .

Change /usr/opencv-64 for the path to where opencv has been installed.

And then modify your setup.py:

from cx_Freeze import setup, Executable

setup(  name = "behave",
        version = "0.1",
        description = "Behave, face detection to help you",
        options = {'build_exe':
                    {'includes': ['numpy'],
                        'include_files': ['cascades/', 'lib/']}},
        executables = [Executable("capturebasic.py")])

The las entry in 'include_files' is the bit that tells OpenCv to bundle the whole folder with the app.
I'm sure there must be a more elegant way of doing this, but this one works: happy days!

cx_Freeze and OpenCV, fixing numpy imports and cascades as data files.

After all the problems found trying cx_Freeze in Mac to create the app in 32-bits (ref) I decided to bend like a bamboo in the wind, and move to OpenCV-64 bit.

Now, a couple of problems found:

When packaging the app with:

python2.7 setup.py bdist_dmg

An error would arise trying to execute it:

open build/exe.macosx-10.6-intel-2.7/capturebasic
"cx-freeze numpy.core.multiarray failed to import"

This error is solved by adding numpy in your setup.py, the reason behind is that even though OpenCv's Python wrapper uses numpy, cx_freeze fails to detect it in its dependency crawler.

setup.py then looks like:

from cx_Freeze import setup, Executable
setup(  name = "behave",
        version = "0.1",
        description = "Behave, face detection to help you",
        options = {'build_exe':
                    {'includes': ['numpy']}},
        executables = [Executable("capturebasic.py", )])

The second problem I encounter after that is that the cascades directory, where the xml files that the face detector algorithm uses, were not included in the cx_freeze app's directory.
The error you can find is similar to:

cv2.error: ../cascadedetect.cpp:1578: error: (-215) !empty() in function detectMultiScale

That is, instantiating "cv2.CascadeClassifier(path_to_xml_file)" will give no errors if you pass a wrong path, but when it is used later in the code to actually detect something, it will complain.

How to solve this? you need to add the cascade files in the project:

from cx_Freeze import setup, Executable
setup(  name = "behave",
        version = "0.1",
        description = "Behave, face detection to help you",
        options = {'build_exe':
                    {'includes': ['numpy'],
                    'include_files': ['cascades/haarcascade_frontalface_alt.xml']}},
        executables = [Executable("capturebasic.py", )])

One more step is required, since we are using those haarcascade files as data files inside cx_freeze, we need to tell the app where to find them, adding a function to our code:

def find_data_file(filename):
    if getattr(sys, 'frozen', False):
        # The application is frozen
        datadir = os.path.dirname(sys.executable)
    else:
        # The application is not frozen
        # Change this bit to match where you store your data files:
        datadir = os.path.dirname(__file__)

    return os.path.join(datadir, filename)

And using it any time we try to access the cascades we are using.
Inside the code it looks like:

self.face_classifier = classifiers.CascadeClassifier(
           find_data_file('cascades/haarcascade_frontalface_alt.xml'))

More information here:
http://cx-freeze.readthedocs.org/en/latest/faq.html#using-data-files

How to install OpenCv in Mac Os 10.8.5

Easy way that simply works, this will install 64bit version of opencv, using python2.7.

First, install git and get opencv from github:

git clone https://github.com/Itseez/opencv.git

Then:

cd opencv
mkdir 64-bit
cd 64-bit
make -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/opencv-64 -D PYTHON_EXECUTABLE:FILEPATH=/usr/local/bin/python2.7 -D INSTALL_PYTHON_EXAMPLES:BOOL=ON ..
make -j 8  #j 8 speeds up things by doing parallel jobs
sudo make install

CMAKE_INSTALL_PREFIX is the path where OpenCv libs will be installed.

FILEPATH is the path to the python you want OpenCv to use.

We need to set some env variables for when opencv library is imported inside our python code, in bold:

vim ~/.bash_profile
###-------
export PYTHONPATH=/usr/opencv-64/lib/python2.7/site-packages:/usr/local/lib/python2.7/site-packages/:/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages:
export DYLD_LIBRARY_PATH=/usr/opencv-64/lib/

Last part (DYLD...) tells python where to find opencv libs when importing.

And that's it, you should be able to python; import cv2.

mach-o, but wrong architecture Part II

I've tried to use cx-freeze with the setup.py instead of the script, same same.
Ref:
http://cx-freeze.readthedocs.org/en/latest/distutils.html#distutils

How to install cx-freeze from source to use a 32 bit version of python, in this case python2.7-32.
1. Download source from:
http://cx-freeze.sourceforge.net/
2. Untar it. tar xfv or jus doubleclick
3. build/install:

python2.7-32 setup.py build
python2.7-32 setup.py install

Then you can run cx-freeze in your 32 bit version:

python2.7-32 setup.py bdist_dmg

 No matter what, the app generated is still a 64bit version:

$ python2.7-32 setup.py bdist_dmg
$ file capturebasic
capturebasic: Mach-O 64-bit executable x86_64
$ arch -i386 ./capturebasic
arch: posix_spawnp: ./capturebasic: Bad CPU type in executable

I think the problem is that I'm installing the 64bit version of cx-freeze, even if I am running python2.7-32 setup.py build.

So, what next?
- keep banging my head against this?
- move to 64bit opencv and forget?

mach-o, but wrong architecture

Using cx-freeze to create a package to distribute behave, some problems with it.

>> python2.7-32 /Library/Frameworks/Python.framework/Versions/2.7/bin/cxfreeze capturebasic.py
[..]
>> ./capturebasic
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 27, in <module>
    exec code in m.__dict__
  File "capturebasic.py", line 1, in <module>
ImportError: dlopen(/Users/Luis/Dropbox/PyProjects/behave/dist/cv2.so, 2): no suitable image found.  Did find:
/Users/Luis/Dropbox/PyProjects/behave/dist/cv2.so: mach-o, but wrong architecture

This is the same error that I got when opencv has been built and installed in its 32-version and then you try to use from within a 64bits Python version.

I even tried to build/install cx-freeze in 32 bit:

  python2.7-32 setup.py build
  python2.7-32 setup.py install 

But same errors.

My best guess at the moment is that cxfreeze puts the 64-bit version in the app by default, and cannot import cv2 from opencv.

Options:
- try to force cxfreeze into python2.7-32
- re-install opencv, the reason why I installed the 32bit version was because it was the one compatible with pygame, since I'm not using pygame, maybe is a good chance to move to 64.

OpenCv, calculate average fps in python

After having left the app to run for a couple of hours, I noticed that the memory had more than doubled.

From 70MB to 150MB of "Real Memory" as Mac's Activity Monitor puts it.

What was happening is that I had created a list that was growing with each frame in order to calculate the average of the frames.

I decided to get rid of this, first creating a circular counter in Python using a generator that resets to 0 when reaches a max value. Then initiating a fixed-size list that the generator will fill the fps in a circular manner.

It looks like this:

def circular_counter(max):
    """helper function that creates an eternal counter till a max value"""
    x = 0
    while True:
        if x == max:
            x = 0
        x += 1
        yield x

class CvTimer(object):
    def __init__(self):
        self.tick_frequency = cv2.getTickFrequency()
        self.tick_at_init = cv2.getTickCount()
        self.last_tick = self.tick_at_init
        self.fps_len = 100
        self.l_fps_history = [ 10 for x in range(self.fps_len)]
        self.fps_counter = circular_counter(self.fps_len)
    def reset(self):
        self.last_tick = cv2.getTickCount()
    def get_tick_now(self):
        return cv2.getTickCount()
    @property
    def fps(self):
        fps = self.tick_frequency / (self.get_tick_now() - self.last_tick)
        self.l_fps_history[self.fps_counter.next() - 1] = fps
        return fps
    @property
    def avg_fps(self):
        return sum(self.l_fps_history) / float(self.fps_len)

In your frame-by-frame while loop:

#Timecv:
cv2.putText(self.a_frame, "fps=%s avg=%s" % (timer.fps, timer.avg_fps),
            (10, 75), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255,255,255))

And finally in the app window:

Behave, declaration of principles

The idea of controlling yourself in front of computer helped by an app was born around three years ago while cycling to work. It was a sunny morning, precious situation in Ireland, riding on Strawberry Beds road direction Leixlip.

It was clear that had to use machine learning techniques, computer vision and Python. 

Three years have passed, today there is a running app that does the basics of what I had in mind, and it works.

It works so well that I use it as I code, the more I use it, the more need to code it to improve it, the more I code it, the more I need it... an unexpected vicious circle.

Behave is open source and free, its aim is to help people, serve as support to maintain a healthier back and habits in front of a computer. 

Because I firmly believe that health should be a right and not a benefit nor a business, this software is and it will be free.

The dream is to reach schools where youngsters are starting to spend time in front of the screen and to prevent them from back problems, and other habits, as they grow up :)

That's it, it is written.

Performance with opencv and python, fps reduced

So, I decided to sharpen the knifes and get real data on what affects performance in running Behave.

The problem: 

A refactor in python code using opencv, made the program running slower.

The refactor consisted in transforming basic code using functions, "Functional Code":
https://github.com/Kieleth/behave/commit/fda558d61c11720d346a107c174a052f97c2ee3b

Into more abstracted "Code with classes":
https://github.com/Kieleth/behave/commit/503e698dbcbea2c7224fae72c12a94832552c606

Difference was observed from +-12fps to +-5fps. Functionality is the same.

The weapons:

- Python performance tips:
https://wiki.python.org/moin/PythonSpeed/PerformanceTips
- Python profiler:
http://docs.python.org/2/library/profile.html
- Opencv optimisation guide:
http://answers.opencv.org/question/755/object-detection-slow/#760

Process:

First of all was to create a branch from the functional code in git, this allows to explore both possibilities and move between them.

Now, lets get some data:

Functional code:

It runs smoothly at around 10fps.

Added a break in the code, that is triggered after 20 seconds of running the app:

profile:

python2.7-32 -m profile -s cumulative capturebasic.py

         16009 function calls (15918 primitive calls) in 16.698 seconds
   Ordered by: cumulative time
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   16.698   16.698 profile:0(<code object <module> at 0x445a88, file "capturebasic.py", line 1>)
        1    0.041    0.041   16.697   16.697 capturebasic.py:1(<module>)
        1    0.253    0.253   16.465   16.465 capturebasic.py:10(launch)
      129    6.311    0.049    6.311    0.049 :0(waitKey)
      129    2.615    0.020    2.615    0.020 :0(imshow)
      129    2.554    0.020    2.554    0.020 :0(detectMultiScale)
      129    1.867    0.014    1.867    0.014 :0(read)
      129    1.124    0.009    1.124    0.009 :0(flip)
      129    0.688    0.005    0.688    0.005 :0(cvtColor)
      129    0.602    0.005    0.602    0.005 :0(equalizeHist)
        1    0.205    0.205    0.205    0.205 :0(VideoCapture)...

cProfile:

python2.7-32 -m cProfile -s cumulative capturebasic.py 

         16184 function calls (16093 primitive calls) in 20.487 seconds
   Ordered by: cumulative time
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.044    0.044   20.488   20.488 capturebasic.py:1(<module>)
        1    0.073    0.073   20.331   20.331 capturebasic.py:10(launch)
      141    8.175    0.058    8.175    0.058 {cv2.waitKey}
      141    3.450    0.024    3.450    0.024 {cv2.imshow}
      141    2.565    0.018    2.565    0.018 {method 'detectMultiScale' of 'cv2.CascadeClassifier' objects}
      141    1.903    0.013    1.903    0.013 {method 'read' of 'cv2.VideoCapture' objects}
        1    1.309    1.309    1.309    1.309 {cv2.VideoCapture}
      141    1.219    0.009    1.219    0.009 {cv2.flip}
      141    0.592    0.004    0.592    0.004 {cv2.equalizeHist}
      141    0.565    0.004    0.565    0.004 {cv2.cvtColor}
        1    0.303    0.303    0.303    0.303 {method 'release' of 'cv2.VideoCapture' objects}
        1    0.115    0.115    0.115    0.115 {cv2.CascadeClassifier}

Found:

• profile for some reason only runs 16.7 seconds, instead of the full 20. Explanation could be this: "which adds significant overhead to profiled programs" from http://docs.python.org/2/library/profile.html#introduction-to-the-profilers
• the waitKey takes a good deal of time? seems normal since has a 1ms delay every time is called, explained in: http://docs.opencv.org/modules/highgui/doc/user_interface.html
• cv.imshow would be worth it to compare to other window handler, pygame or others.

Added a break in the code that exits the program after 100 frames:

Using linux "time":

#3 passes: 

time python2.7-32 capturebasic.py 

real 0m15.977s,       real 0m15.411s,       real 0m15.398s

Code with classes:

Running sluggish at 5fps.

Break at 20 seconds of running the app:

python2.7-32 -m cProfile -s cumulative capturebasic.py
         13488 function calls (13397 primitive calls) in 21.944 seconds
   Ordered by: cumulative time
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.160    0.160   21.946   21.946 capturebasic.py:1(<module>)
       52    0.001    0.000    8.801    0.169 classifiers.py:34(__init__)
       52    0.003    0.000    8.800    0.169 classifiers.py:4(__init__)
       52    8.797    0.169    8.797    0.169 {cv2.CascadeClassifier}
       52    0.001    0.000    5.382    0.104 gui.py:17(should_quit)
       52    5.382    0.103    5.382    0.103 {cv2.waitKey}
       52    1.768    0.034    1.768    0.034 {cv2.imshow}
       52    0.001    0.000    1.679    0.032 classifiers.py:21(detect_multiscale)
       52    1.678    0.032    1.678    0.032 {method 'detectMultiScale' of 'cv2.CascadeClassifier' objects}
        1    1.178    1.178    1.178    1.178 {cv2.VideoCapture}
       52    0.952    0.018    0.952    0.018 {method 'read' of 'cv2.VideoCapture' objects}
       52    0.002    0.000    0.838    0.016 utils.py:15(prepare_frame_for_detection)
       52    0.000    0.000    0.714    0.014 utils.py:11(flip_frame)
       52    0.714    0.014    0.714    0.014 {cv2.flip}
       52    0.423    0.008    0.423    0.008 {cv2.cvtColor}
       52    0.413    0.008    0.413    0.008 {cv2.equalizeHist}
        1    0.330    0.330    0.330    0.330 {method 'release' of 'cv2.VideoCapture' objects}
        1    0.003    0.003    0.107    0.107 __init__.py:106(<module>)
        1    0.000    0.000    0.050    0.050 add_newdocs.py:9(<module>)
        2    0.003    0.002    0.042    0.021 __init__.py:1(<module>)
        1    0.002    0.002    0.037    0.037 __init__.py:15(<module>)
        2    0.007    0.004    0.029    0.014 __init__.py:2(<module>)
        1    0.000    0.000    0.028    0.028 type_check.py:3(<module>)
       52    0.001    0.000    0.016    0.000 gui.py:11(display_faces)

Found:

• "{cv2.CascadeClassifier}" is executed 52 times, that is, every frame, whereas in the functional code is only executed 1.

Break after 100 frames:

time python2.7-32 capturebasic.py

real 0m27.675s,     real 0m25.162s,      real 0m26.867s

Solution:

In the refactored code I was instantiating the face detector "FaceClassifier" class in every pass of the loop:

while(True):
    ...
    face_classifier = FaceClassifier()
    faces_list = face_classifier.detect_multiscale(frame_prepared)

Taking it out of the while:

face_classifier = FaceClassifier()

while(True):
    ...
    faces_list = face_classifier.detect_multiscale(frame_prepared)

And fps are back on track:

Cleaned up camera.
         18736 function calls (18645 primitive calls) in 21.654 seconds
   Ordered by: cumulative time
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.092    0.092   21.656   21.656 capturebasic.py:1(<module>)
      168    0.008    0.000    9.017    0.054 gui.py:17(should_quit)
      168    9.008    0.054    9.008    0.054 {cv2.waitKey}
      168    3.317    0.020    3.317    0.020 {cv2.imshow}
      168    0.020    0.000    2.745    0.016 classifiers.py:21(detect_multiscale)
      168    2.725    0.016    2.725    0.016 {method 'detectMultiScale' of 'cv2.CascadeClassifier' objects}
      168    2.077    0.012    2.077    0.012 {method 'read' of 'cv2.VideoCapture' objects}
      168    0.001    0.000    1.322    0.008 utils.py:11(flip_frame)
      168    1.320    0.008    1.320    0.008 {cv2.flip}
      168    0.015    0.000    1.277    0.008 utils.py:15(prepare_frame_for_detection)
        1    1.165    1.165    1.165    1.165 {cv2.VideoCapture}
      168    0.648    0.004    0.648    0.004 {cv2.equalizeHist}
      168    0.613    0.004    0.613    0.004 {cv2.cvtColor}
        1    0.332    0.332    0.332    0.332 {method 'release' of 'cv2.VideoCapture' objects}
        1    0.002    0.002    0.104    0.104 __init__.py:106(<module>) 

Timings as well:

time python2.7-32 capturebasic.py
real 0m14.860s,    real 0m17.444s,    real 0m13.883s

This is a bug, but a good one, since it has forced me to dig into performance tools used with opencv. I'm pretty sure this will not be the last time to have to use them in the project.

FPS reduced!

A nasty surprise, a refactor in the code to move it from exploratory-functional to class-abstracted, has ended up with a reduction on fps in the capture.

In simple terms, the exploratory code which was a while loop and some processing inside was giving averages of 12fps. 

The "better" code, with classes, modular, extensible, cleaner, etc, gives 5fps.

This means that class interaction in the python code affects performance.

Optimisation is needed, some links to follow:

http://docs.opencv.org/trunk/doc/py_tutorials/py_core/py_optimization/py_optimization.html#optimization-techniques

https://wiki.python.org/moin/PythonSpeed/PerformanceTips

http://pynash.org/2013/03/06/timing-and-profiling.html

Viola-Jones, Haar features, Cascades, plan.

A bit of theory:
http://en.wikipedia.org/wiki/Viola%E2%80%93Jones_object_detection_framework
http://en.wikipedia.org/wiki/AdaBoost

Slides:
http://www.slideshare.net/wolf/avihu-efrats-viola-and-jones-face-detection-slides/

Viola-Jones explained:
https://sites.google.com/site/5kk73gpu2012/assignment/viola-jones-face-detection#TOC-Image-Pyramid

Integral Images
http://en.wikipedia.org/wiki/Summed_area_table
Haar:http://en.wikipedia.org/wiki/Haar-like_features

Good to understand.
http://docs.opencv.org/trunk/doc/py_tutorials/py_objdetect/py_face_detection/py_face_detection.html

Future, for training the cascades:
http://docs.opencv.org/doc/user_guide/ug_traincascade.html

The nicety is that since this is a working prototype, is controlling my back even as I write this post.

What are next steps?

1. Incorporate GUI:
  - selection of threshold to check face position (drag and drop?)
  - customization of alerts when alarm is triggered (idea is to drop some scripts in folder, to make it extensible.
  - Turn camera on-off to improve performance.
  - Menu to select tweaks in performance (investigate how to show real FPS)
    - face detection tweaks
    - general camera capture tweaks.
    - Auto-find best setup for the computer.

2. Add nail bitting checking.

3. Make it installable (bundle opencv-required libs inside the project).

Future:
- "Teaching" option, so user tells Behave to learn things to detect.

Behaving

Digging into opencv and theory behind object detection/recognition.

Project to detect hand gestures, great stuff.
https://www.youtube.com/watch?v=8Vr08EYBN04

And, face detection using Haar cascades explained:
https://www.youtube.com/watch?v=sWTvK72-SPU

Behave, alive.

After some months of proper procrastination, finally Behave has a raw working version:

https://github.com/Kieleth/behave

I decided the re-do the version I had previously, it was giving problems with latency in the capture and image processing. So, I cut short and simplified as much as I could.

Also the fact that OpenCv documentation online is back, helped a lot, some nice articles for basics:

http://docs.opencv.org/trunk/doc/py_tutorials/py_gui/py_video_display/py_video_display.html

http://docs.opencv.org/trunk/doc/py_tutorials/py_objdetect/py_face_detection/py_face_detection.html

At the moment it runs smoothly thanks to some minor tweaks in the face-detection call:

    #Recognition:
    scale_factor = 1.3
    min_neigh = 4
    flags = cv2.CASCADE_SCALE_IMAGE
    minSize = (200, 200)

    faces = face_cascade.detectMultiScale(gray, scale_factor, min_neigh,minSize=minSize, flags=flags)

Where:

    cv2.cv.CV_HAAR_SCALE_IMAGE: Scales each windowed image region to match the feature data. (The default approach is the opposite: scale the feature data to match the window.) Scaling the image allows for certain optimizations on modern hardware. This flag must not be combined with others.

• scaleFactor – Parameter specifying how much the image size is reduced at each image scale.
• minNeighbors – Parameter specifying how many neighbors each candidate rectangle should have to retain it.
• flags – Parameter with the same meaning for an old cascade as in the function cvHaarDetectObjects. It is not used for a new cascade.
• minSize – Minimum possible object size. Objects smaller than that are ignored.
• maxSize – Maximum possible object size. Objects larger than that are ignored

The idea is to find optimal parameters automatically using frame times, but that's another day's work :D

I'll be working in adding functionality and cleaning up the first version of the code (in exploratory mode still).

"Bus error: 10" Cont.

It seems the best option is to make cv again, so, lets get the latest version:

# in a nice place you want to do your clone:
git clone https://github.com/Itseez/opencv.git opencv
cd opencv
mkdir makethis; cd makethis 

 # select 2.4 branch:
Luiss-iMac:opencv Luis$ git branch -a
* master
  remotes/origin/2.4
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
Luiss-iMac:opencv Luis$ git checkout 2.4

#This installs the 32 bit version:

cmake -G "Unix Makefiles" -D CMAKE_OSX_ARCHITECTURES=i386 -D CMAKE_C_FLAGS=-m32 -D CMAKE_CXX_FLAGS=-m32 -D CMAKE_INSTALL_PREFIX=/usr/local -D PYTHON_EXECUTABLE:FILEPATH=/usr/local/bin/python2.7-32 -D INSTALL_PYTHON_EXAMPLES:BOOL=ON .. 

# for the 64 version:
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local PYTHON_EXECUTABLE:FILEPATH=/usr/local/bin/python2.7 -D INSTALL_PYTHON_EXAMPLES:BOOL=ON ..  

make -j8

#Go a get a beer... 

 Success looks like:

BUILD SUCCESSFUL
Total time: 5 seconds
[100%] Built target opencv_test_java 

sudo make install

Now, you might think that it's done:

Luiss-iMac:make5 Luis$ python2.7-32
Python 2.7.5 (v2.7.5:ab05e7dd2788, May 13 2013, 13:18:45)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/cv.py", line 1, in <module>
    from cv2 import *
ImportError: dlopen(/usr/local/lib/python2.7/site-packages/cv2.so, 2): Symbol not found: __ZN2cv10PCAComputeERKNS_11_InputArrayERKNS_12_OutputArrayES5_d
  Referenced from: /usr/local/lib/python2.7/site-packages/cv2.so
  Expected in: lib/libopencv_core.3.0.dylib
 in /usr/local/lib/python2.7/site-packages/cv2.so

What's wrong now? well, this is an old fight I had with opencv, I managed to repel the enemy by:

          export DYLD_LIBRARY_PATH=/usr/opencv/lib/ 

That is, tell the Dynamic Linker

https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/dyld.1.html


Where to find the libs needed for cv2.so, how to find it?:

Luiss-iMac:site-packages Luis$ otool -L cv2.so

\cv2.so:

     cv2.so (compatibility version 0.0.0, current version 0.0.0)

     /System/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.2)

     lib/libopencv_core.2.4.dylib (compatibility version 2.4.0, current version 2.4.7)

After this (you can include the env in your profile), and remember to use python-32 if you have installed OpenCV in 32 bits, you can use cv again:

Luiss-iMac:site-packages Luis$ python-32
Python 2.7.5 (v2.7.5:ab05e7dd2788, May 13 2013, 13:18:45)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv
>>>

EDIT:
An updated version on how to (re-)install OpenCv and/or set your DYLD_LIBRARY_PATH can be found here.