About a year ago, I created a Wall-E robot that does object and face recognition. It uses Arduino as the controller and need to communicate with a computer that runs the face detection program to track the target. Raspberry Pi face recognition has become very popular recently.

---

Some of the links on this page are affiliate links. I receive a commission (at no extra cost to you) if you make a purchase after clicking on one of these affiliate links. This helps support the free content for the community on this website. Please read our Affiliate Link Policy for more information.

With the powerful processor on Raspberry Pi, I can connect it with the Arduino using i2c on the robot and run the object recognition program on-board. It could become a truly independent, intelligent Wall-E robot!

However, building such a robot will be a project for near future. In this article, I will be showing you how to do basic Object Recognition on the Raspberry Pi using Python and OpenCV. I will also show you a simple open loop face tracking application using pan-tilt servos to turn the camera around.

Note: Please be careful about the indentation in the Python codes, sometimes my blog decides to mess this up randomly.

I wrote an article on how to use SSH and VNC to control and monitor the Raspberry Pi, and that’s what I will be using in this project.

Installing OpenCV For Python

To install OpenCV for Python, all you have to do is use apt-get like below:

sudo apt-get install python-opencv

To test the installation of OpenCV, run this Python script, it will switch on your camera for video streaming if it is working.

[sourcecode language=”python”]
import cv

cv.NamedWindow(“w1”, cv.CV_WINDOW_AUTOSIZE)
camera_index = 0
capture = cv.CaptureFromCAM(camera_index)

def repeat():
global capture #declare as globals since we are assigning to them now
global camera_index
frame = cv.QueryFrame(capture)
cv.ShowImage(“w1″, frame)
c = cv.WaitKey(10)
if(c==”n”): #in “n” key is pressed while the popup window is in focus
camera_index += 1 #try the next camera index
capture = cv.CaptureFromCAM(camera_index)
if not capture: #if the next camera index didn’t work, reset to 0.
camera_index = 0
capture = cv.CaptureFromCAM(camera_index)

while True:
repeat()
[/sourcecode]

Simple Example of Raspberry Pi Face Recognition

This example is a demonstration for Raspberry Pi face recognition using haar-like features. it finds faces in the camera and puts a red square around it. I am surprised how fast the detection is given the limited capacity of the Raspberry Pi (about 3 to 4 fps). Although it’s still much slower than a laptop, but it would still be useful in some robotics applications.

You will need to download this trained face file:

http://stevenhickson-code.googlecode.com/svn/trunk/AUI/Imaging/face.xml

[sourcecode language=”python”]
#!/usr/bin/python

The program finds faces in a camera image or video stream and displays a red box around them.

import sys
import cv2.cv as cv
from optparse import OptionParser

min_size = (20, 20)
image_scale = 2
haar_scale = 1.2
min_neighbors = 2
haar_flags = 0

def detect_and_draw(img, cascade):
# allocate temporary images
gray = cv.CreateImage((img.width,img.height), 8, 1)
small_img = cv.CreateImage((cv.Round(img.width / image_scale),
cv.Round (img.height / image_scale)), 8, 1)

# convert color input image to grayscale
cv.CvtColor(img, gray, cv.CV_BGR2GRAY)

# scale input image for faster processing
cv.Resize(gray, small_img, cv.CV_INTER_LINEAR)
cv.EqualizeHist(small_img, small_img)

if(cascade):
t = cv.GetTickCount()
faces = cv.HaarDetectObjects(small_img, cascade, cv.CreateMemStorage(0),
haar_scale, min_neighbors, haar_flags, min_size)
t = cv.GetTickCount() – t
print “time taken for detection = %gms” % (t/(cv.GetTickFrequency()*1000.))
if faces:
for ((x, y, w, h), n) in faces:
# the input to cv.HaarDetectObjects was resized, so scale the
# bounding box of each face and convert it to two CvPoints
pt1 = (int(x * image_scale), int(y * image_scale))
pt2 = (int((x + w) * image_scale), int((y + h) * image_scale))
cv.Rectangle(img, pt1, pt2, cv.RGB(255, 0, 0), 3, 8, 0)

cv.ShowImage(“video”, img)

if __name__ == ‘__main__’:

parser = OptionParser(usage = “usage: %prog [options] [filename|camera_index]”)
parser.add_option(“-c”, “–cascade”, action=”store”, dest=”cascade”, type=”str”, help=”Haar cascade file, default %default”, default = “../data/haarcascades/haarcascade_frontalface_alt.xml”)
(options, args) = parser.parse_args()

cascade = cv.Load(options.cascade)

if len(args) != 1:
parser.print_help()
sys.exit(1)

input_name = args[0]
if input_name.isdigit():
capture = cv.CreateCameraCapture(int(input_name))
else:
capture = None

cv.NamedWindow(“video”, 1)

#size of the video
width = 160
height = 120

if width is None:
width = int(cv.GetCaptureProperty(capture, cv.CV_CAP_PROP_FRAME_WIDTH))
else:
cv.SetCaptureProperty(capture,cv.CV_CAP_PROP_FRAME_WIDTH,width)

if height is None:
height = int(cv.GetCaptureProperty(capture, cv.CV_CAP_PROP_FRAME_HEIGHT))
else:
cv.SetCaptureProperty(capture,cv.CV_CAP_PROP_FRAME_HEIGHT,height)

if capture:
frame_copy = None
while True:

frame = cv.QueryFrame(capture)
if not frame:
cv.WaitKey(0)
break
if not frame_copy:
frame_copy = cv.CreateImage((frame.width,frame.height),
cv.IPL_DEPTH_8U, frame.nChannels)

if frame.origin == cv.IPL_ORIGIN_TL:
cv.Copy(frame, frame_copy)
else:
cv.Flip(frame, frame_copy, 0)

detect_and_draw(frame_copy, cascade)

if cv.WaitKey(10) >= 0:
break
else:
image = cv.LoadImage(input_name, 1)
detect_and_draw(image, cascade)
cv.WaitKey(0)

cv.DestroyWindow(“video”)

[/sourcecode]

To run this program, type in this command in your VNC Viewer’s terminal:

python facedetect.py --cascade=face.xml 0

The number at the end represents the number of your video device.

Face Tracking in Raspberry Pi with pan-tilt Servos

In this example I will be using the Wall-E Robot‘s camera and pan-tilt servo head.

The idea is simple. Raspberry Pi detects the position of the face, sends a command to the Arduino. Arduino will convert the command into servo position and turn the camera. I am using i2c to connect Raspberry Pi and Arduino.

Note: I am still trying to optimize the code for this example, so the result is still not great, but it gives you the idea how it works. I will come back and update this post as soon as I am happy with the result.

Arduino Source Code

Here is the Arduino code. Note that this code uses a very dummy and basic open loop control method, I only use this because of its simplicity. For a more optimal control method, please see Color Tracking Using PID.

In this example it basically waits for commands from the Raspberry Pi and turn the head around. The commands are expected to be integer 1, 2, 3 or 4, each represents a direction that it should turn the camera to. While it’s turning, the variable ‘state’ will be set to zero, so Raspberry Pi will stop detecting or sending any more command to avoid turning the camera too far, because of the delays.

Like I mentioned at the beginning, this is an open loop control system, and still has a lot of room to improve. I made it so simple just for some people to pick up more easily.

[sourcecode language=”cpp”]
#include <Wire.h>
#define SLAVE_ADDRESS 0x04
byte command;
byte state;

// Servo code
#include <Servo.h>

Servo servoNeckX;
Servo servoNeckY;

const byte servoNeckX_pin = 3;
const byte servoNeckY_pin = 4;

const int lrServoMax = 2300; // looking right
const int lrServoMin = 700;
const int udServoMax = 2100; // looking down
const int udServoMin = 750; // looking up

int posX = 1500;
int posY = 1300;

// End of Servo code

void setup() {

servoNeckX.attach(servoNeckX_pin);
servoNeckY.attach(servoNeckY_pin);

servoNeckX.writeMicroseconds(posX);
delay(100);
servoNeckY.writeMicroseconds(posY);
delay(100);

// initialize i2c as slave
Wire.begin(SLAVE_ADDRESS);

// define callbacks for i2c communication
Wire.onReceive(receiveData);
Wire.onRequest(sendData);

Serial.begin(9600); // start serial for output
Serial.println(“Ready!”);

state = 1;
}

void loop() {
delay(20);
}

// callback for received data
void receiveData(int byteCount){

while(Wire.available()) {
state = 0; // moving servos
command = Wire.read();
Serial.print(“command received: “);
Serial.println(command);

switch (command){

case 1:
// lift head (-Y)
posY = constrain(posY-20, udServoMin, udServoMax);
servoNeckY.writeMicroseconds(posY);
break;

case 2:
// lower head (+Y)
posY = constrain(posY+20, udServoMin, udServoMax);
servoNeckY.writeMicroseconds(posY);
break;

case 3:
// turn head left (+X)
posX = constrain(posX+20, lrServoMin, lrServoMax);
servoNeckX.writeMicroseconds(posX);
break;

case 4:
// turn head right (-X)
posX = constrain(posX-20, lrServoMin, lrServoMax);
servoNeckX.writeMicroseconds(posX);
break;
}
state = 1; // finished moving servos

}
}

// callback for sending data

void sendData(){
Wire.write(state);
}

[/sourcecode]

The Python Code is similar to the first example. I added some code necessary for i2c communication and a few lines after a face is detected, so it sends commands to the Arduino.

[sourcecode language=”python”]

#!/usr/bin/python

import sys
import cv2.cv as cv
from optparse import OptionParser

min_size = (20, 20)
image_scale = 2
haar_scale = 1.2
min_neighbors = 2
haar_flags = 0

def detect_and_draw(img, cascade):
# allocate temporary images
gray = cv.CreateImage((img.width,img.height), 8, 1)
small_img = cv.CreateImage((cv.Round(img.width / image_scale), cv.Round (img.height / image_scale)), 8, 1)

# convert color input image to grayscale
cv.CvtColor(img, gray, cv.CV_BGR2GRAY)

# scale input image for faster processing
cv.Resize(gray, small_img, cv.CV_INTER_LINEAR)
cv.EqualizeHist(small_img, small_img)

if(cascade):
t = cv.GetTickCount()
faces = cv.HaarDetectObjects(small_img, cascade, cv.CreateMemStorage(0),
haar_scale, min_neighbors, haar_flags, min_size)
t = cv.GetTickCount() – t
print “time taken for detection = %gms” % (t/(cv.GetTickFrequency()*1000.))
if faces:
for ((x, y, w, h), n) in faces:
# the input to cv.HaarDetectObjects was resized, so scale the
# bounding box of each face and convert it to two CvPoints
pt1 = (int(x * image_scale), int(y * image_scale))
pt2 = (int((x + w) * image_scale), int((y + h) * image_scale))
cv.Rectangle(img, pt1, pt2, cv.RGB(255, 0, 0), 3, 8, 0)

cv.ShowImage(“video”, img)

if __name__ == ‘__main__’:

parser = OptionParser(usage = “usage: %prog [options] [filename|camera_index]”)
parser.add_option(“-c”, “–cascade”, action=”store”, dest=”cascade”, type=”str”, help=”Haar cascade file, default %default”, default = “../data/haarcascades/haarcascade_frontalface_alt.xml”)
(options, args) = parser.parse_args()

cascade = cv.Load(options.cascade)

if len(args) != 1:
parser.print_help()
sys.exit(1)

input_name = args[0]
if input_name.isdigit():
capture = cv.CreateCameraCapture(int(input_name))
else:
capture = None

cv.NamedWindow(“video”, 1)

#size of the video
width = 160
height = 120

if width is None:
width = int(cv.GetCaptureProperty(capture, cv.CV_CAP_PROP_FRAME_WIDTH))
else:
cv.SetCaptureProperty(capture,cv.CV_CAP_PROP_FRAME_WIDTH,width)

if height is None:
height = int(cv.GetCaptureProperty(capture, cv.CV_CAP_PROP_FRAME_HEIGHT))
else:
cv.SetCaptureProperty(capture,cv.CV_CAP_PROP_FRAME_HEIGHT,height)

if capture:
frame_copy = None

while True:

frame = cv.QueryFrame(capture)
if not frame:
cv.WaitKey(0)
break
if not frame_copy:
frame_copy = cv.CreateImage((frame.width,frame.height),
cv.IPL_DEPTH_8U, frame.nChannels)

if frame.origin == cv.IPL_ORIGIN_TL:
cv.Copy(frame, frame_copy)
else:
cv.Flip(frame, frame_copy, 0)

detect_and_draw(frame_copy, cascade)

if cv.WaitKey(10) >= 0:
break
else:
image = cv.LoadImage(input_name, 1)
detect_and_draw(image, cascade)
cv.WaitKey(0)

cv.DestroyWindow(“video”)

[/sourcecode]

Possible Raspberry Pi Face Recognition Improvement

For face recognition on an embedded system, I think LBP is a better choice, because it does all the calculations in integers. Haar uses floats, whick is a killer for embedded/mobile. LBP is a few times faster, but about 10-20% less accurate than Haar.