Wall E Robot (object/sound recognition, AI)

by Oscar

Wall E Robot

This project is aimed to bring the Wall E robot from the movie into life! it will have a camera in one of his eyes, the information will be passed and processed in the computer, and commands are then sent via bluetooth back the Wall E robot. It can also recognised sounds, and can be controlled manually as a spy robot.

banggood

The source code you see in this project are written in C++ and with Qt Framework and OpenCV library.

=============================================
Flow chart

commands
PC ——–>——–>——–> Robot (Arduino)
PC < ——<——– Bluetooth ——> Air —–> Bluetooth —–> Arduino
Serial
communication

[ 27/08/2012 ] Test 1 – Colour Tracking

[ 05/09/2012 ] Test 2 – Servos and motor driver

[ 08/09/2012 ] Test 3 – Controlling Wall-E from PC

[ 11/09/2012 ] Test 4 – Speech Recognition

=============================================
update 05/08/2012

I will need a easy way to create user interface for the control program, there are a couple of options: SDL, Qt.

SDL the convenient library or tool that handle “Create a Window” , “Load a Image File” , “Play Sound and Music” , “Load Font in ttf Format” , “TCP/IP Protocol” , “Using a Gamepad , a Keyboard , and a Mouse” for you. So you can concentrate on creating your own world with these handy tools. It gives you the relative easy way to make a multi-media windows program.
https://sites.google.com/site/sdlgamer/

Also need to learn how to user serial ports.
The tutorial of Win32 Serial Port Programming
https://sites.google.com/site/arduinorccar/communication

=============================================
update 08/08/2012

I think i have most of the project planned, and come up with quite a few functionalities.

For Object tracking and recognition, I will write the code myself with OpenCV in C++. And the program will run on a PC, images are transmitted from Wall E robot using the wireless Wecam, and after processing the corresponding cammands will be sent back to Wall-E via bluetooth.

I have been looking very hard for programming solution for speech recognition, and hope someone has already wriiten a API or some sort. And Iaccidentally bumpped into a YouTube Video showing a much simpler way of doing this – EasyVR Arduino Shield! So I might use that instead of writing codes myself!

=============================================
update 10/08/2012

wall e robot

Wall E robot arrived! :)
I should have started this week, but just before I was going to take the video for my lastest version of the hexapod robot, one of the servos broke!! I guess i just had to wait…

It’s a great toy for 8’s, it’s only got one motor, which means it can only turn left, or go forward. Moves its hands as well, but that’s pretty much it. Here is a video showing roughly the same one:

http://www.youtube.com/watch?v=VEoh8Iws-kk

=============================================
update 12/08/2012

Still waiting for the servo gear to arrive, I am so bored, so I started working on the robot hardware.

I took it apart and amazed by how well it works considering it’s only got so few components.

It was quite dirty since it’s second hand. I had to wash every piece of it with soup water!
I will leave the assembling another day.

=============================================
update 14/08/2012

Finally found the time to look at the robot pieces and could get started to assemble it.

I recycled the motors and motor driver from my previous robot (Wally Object tracking robot).

It was quite challenging to modify the wall e robot to fit the servos. but i did at the end ^.^.
I will start coding another day!

=============================================
update 17/08/2012

[sourcecode language=”cpp”]

colour tracking code!

capwebcam.read(matOriginal);

if(matOriginal.empty() == true) return;
inRange(matOriginal, cv::Scalar(0,0,175), cv::Scalar(100,100,256), matProcessed);
GaussianBlur(matProcessed, matProcessed, cv::Size(9,9), 1.5);
cv::HoughCircles(matProcessed, vecCircles, CV_HOUGH_GRADIENT, 2, matProcessed.rows/4, 100, 50, 10, 400);

for(itrCircles = vecCircles.begin(); itrCircles != vecCircles.end(); itrCircles++){
ui->txtXYRadius->appendPlainText(QString(“ball position x =”) +
QString::number((*itrCircles)[0]).rightJustified(4, ‘ ‘) +
QString(“, y =”) +
QString::number((*itrCircles)[1]).rightJustified(4, ‘ ‘) +
QString(“, radius =”) +
QString::number((*itrCircles)[2], ‘f’, 3).rightJustified(7, ‘ ‘));

cv::circle(matOriginal, cv::Point((int)(*itrCircles)[0], (int)(*itrCircles)[1]), 3, cv::Scalar(0,255,0), CV_FILLED);
cv::circle(matOriginal, cv::Point((int)(*itrCircles)[0], (int)(*itrCircles)[1]), (int)(*itrCircles)[2], cv::Scalar(0,0,255), 3);

}

// Convert OpenCV image to QImage

cv::cvtColor(matOriginal, matOriginal, CV_BGR2RGB);

QImage qimgOriginal((uchar*)matOriginal.data, matOriginal.cols, matOriginal.rows, matOriginal.step, QImage::Format_RGB888);
QImage qimgProcessed((uchar*)matProcessed.data, matProcessed.cols, matProcessed.rows, matProcessed.step, QImage::Format_Indexed8);

// update label on form

ui->lblOriginal->setPixmap(QPixmap::fromImage(qimgOriginal));
ui->lblProcessed->setPixmap(QPixmap::fromImage(qimgProcessed));

[/sourcecode]

=============================================
update 24/08/2012

The Servo Gear finally arrived! (actually the servo did, i guess they must have send me the wrong thing, ^.^ )

Anyway, i immediately fixed the hexapod robot, and started making the video, hope I can finally begin working on the coding for Wall E robot.

=============================================
update 27/08/2012

As a starting point, I wrote a Qt program to detect colour (red), and send out command via serial port to arduino, to turn Wall E’s head to follow the object. I will extend the object that can be tracked to faces, certain objects, light source etc..

I struggled so much at the beginning, because everytime i connect to arduino via serial port, it freezes the video. I later realized it’s the thread issue. when the program is waiting for data from serial port (or reading, or writing? i am not sure), it actually hangs the thread, so I decided to modify both serialport class, and video class to have their own thread when running.

Some people suggest it’s not a very good idea to use thread if we don’t have a formal education on this subject. And I did find it confusing how to start with thread, because some say we shouldn’t make QThread subclass and we should instead move a object into a thread. But since it’s in the official documentation that we should make it subclass, I followed the latter.

I am still very new to Qt and OpenCV, since I only started learning these a few days back, and I was already thinking about multithreading, and I now realized how crazy that was!

With frustrations, I spent the whole weekend and my bank holiday just debugging the code. I dropped my diet routine, my exercises, and my movies! But I won at the end. Altought it is still not as good as I would expect, tracking is quite slow and inaccurate, and the head shakes a lot, at least it works ^.^

I will look around for some better algorithm, at the meantime might add a few more functionality in the program like adjusting the video properties, and better threading coding…

see you now..

=============================================
update 28/08/2012

The whole reason i spent so much time coding the colour tracking was because, i needed to write a program that does multithreading, and that’s because i need to listen to the serial port for input from arduino while processing video.

I need to confirm arduino has completed the previous command before i send another one out. but still, it’s not fast enough.

I saw someone has done a project similar to this, but he doesn’t listen to signal from arduino, but send out commands from computer every frame he processes. and the result is actually better than mine!

i am thinking, with enough delay between each frame, there might be a possibility that this could work. I can also say good bye to the confusing multi-threading programming too!

I should also stop sending commands when there is nothing detected.

should calculate the middle point of the detected image, so it will work regardless the size of the detected image.

i might try it out tomorrow.

=============================================
update 29/08/2012

So I tried sending commands without feedback signal from arduino, and it works great! I modified the code based on my initial Qt program, using single thread, it wasn’t lagging at all! So now I know what was slowing the program down, must be the ‘serial port data listener’. For whatever reason either I am using it wrong, or by nature it is blocking other processes, I should avoid using it. But in the future when I add the command recognition functionality, I will need to somehow send data back to computer, to run certain applications, for example, if I want Wall-E to track Faces, I would say ‘Wall E, follow faces’, and arduino will send the computer the command, and open ‘track faces application’.

=============================================
update 01/09/2012

Assembling the eye.

=============================================
update 02/09/2012

Improving the Tracking computer programs:

1. options to the device.2. options to each object tracking.
this is very useful when it comes to camera devices: http://docs.opencv.org/modules/highgui/doc/reading_and_writing_images_and_video.html?highlight=cv_cap_prop_contrast
re-writing the colour tracking algorithm, here is how it works in the new version:

1. Preprocess the image using cv::inRange(), with necessary colour bounds to isolate a desired colour. It might be a good idea to transform to a color-space like HSV or YCbCr for more stable color bounds because chrominance and luminance are better separated. You can use cvtColor() for this. I imagine we will only need to worry about the Hue and Saturation channels as the Value channel doesn’t contain any color info (i.e., the Value range would be left untouched [0, 255]).

link to cv::cvtColor()

http://opencv.willowgarage.com/documentation/cpp/miscellaneous_image_transformations.html

but how to determine the min/max colour boundries?

2. smooth the image (pre-process) with GuassianBlur to get rid of some of the jagginess. I used a bigger dilation kernel than erosion kernel (5×5 vs. 3×3) to get rid of some noisy pixels. The smoothing might help this also tweaking the thresholds could make the erosion unnecessary.

3. compare sizes if multiple objects detected, choose the biggest one.

=============================================
update 05/09/2012

Finally I have some time to sit down and continue my project! I finished the inside layout and tidied up all the cablings tonight. I Also tested the servos and motor driver, all seem working fine!

But i just have to say how much i hate soldering right now!! I literally spent 2 hours trying to solder a switch to some long cables. The first try, i found the cables was a bit out of contacts, so I put hot glue to stick them together, didn’t help. So I tried very hard, to take the glue off, and replace all the cables with new ones, and soldered again.

This time was better. I then installed it on the robot, and hot-glued it on. Only found that the switch itself isn’t working properly, I have to occasionally push the plastic bit. I guess i might have damaged the switch when I was taking the hot glue off…

There is a reason I love programming so much! When I was at Uni doing projects, I always left all the soldering and cabling works to my lab partner, and I would take care of all the coding and maths. I just don’t have the hands to do these things i guess… :(

=============================================
update 08/09/2012

We can now control Wall E robot from PC.

We can also use it as a spy robot :)

=============================================
update 11/09/2012

Added Speech recognition, so i can now control wall-e with my voice.

I am using EasyVR arudino Shield. The shield is great, easy to use, and work quite well actually. But it’s intolerate to background noise, even it’s just a little.

At first, I trained the robot with commands, and it’s all working fine. But when I switched on the robot, it starts ignoring my commands, being unresponsive. I later realize it’s the noise from the servo and motor that interferes the input from the Mic.

So I re-trained the robot with the motor and servos on, to simulate the noisy environment. And really, it works much better, although still not ideal.

To conclude, I don’t recommend using EasyVR on a robot that is sitting next to servos and motors, or operate in a noisy environment.

For future work, i will migrate the speech recognition to the computer site, and send commands via bluetooth. (actually that’s even better because now I don’t need to send signal back from Wall E robot to computer anymore, so communication will be strickly one way, see last few updates about the bluetooth listener issue)

Source Code Download And Fund Raising

To raise money for new robot parts and the time spent on improving the codes, I am starting a fund raise for this project. To say thank you, anyone who donate more than $10, I will send you the source code of this project, and any future update version of the source code.

Amount does not matter, every little helps. You can donate through paypal via the link below (Donate button).
Thank you!

Leave a Comment

By using this form, you agree with the storage and handling of your data by this website. Note that all comments are held for moderation before appearing.

32 comments

Sergio 22nd September 2022 - 5:46 pm

Hola Óscar.
Me llamo Sergio y me ha impresionado el proyecto de Wall E.

No sé casi nada de robótica pero vuelo drones ,y helicópteros RC.
No sé si ayudará mucho.
Qué es lo necesario para construir un Wall E
Gracias por todo desde España

Reply
elena 29th August 2021 - 2:18 pm

Hi Oscar
I have donated, please send me.
thanks

Reply
Oscar 29th August 2021 - 4:12 pm

Sent, thank you very much!

Reply
N 29th March 2021 - 9:19 am

I have donated, please send me the source code

Reply
Oscar 29th March 2021 - 9:31 pm

Hi, I didn’t find any transaction from your email address.

Reply
Kevr102 8th May 2020 - 3:59 pm

Hi,
Dont know if this post is still open, do you still have the code?

Regards

Reply
Andrea 7th April 2020 - 4:13 am

I sent you 10 euros

Reply
Andrea 1st April 2020 - 1:21 pm

hi, compliments for the project. would be interested in the code, please tell where I should transfer the 10 euros. Thanks a lot

Reply
fadi 27th February 2020 - 1:56 pm

hi sir this is amazing project i’m study in my last year and i’m working on some thing like ur project could u plz send me code ?
i’m from algeria

Reply
Leonardo 22nd April 2018 - 1:08 am

Where did you buy this copy of Wall-e?

Reply
Oscar 23rd April 2018 - 4:42 pm

it was 2nd hand on eBay. Got it years ago so I don’t think it’s easily available again.

Reply
Dennis A. 12th December 2017 - 1:46 pm

Hi Oscar, i think it is a verry good project and i start to work on my own version. Donate is sone. Have a nice time.

Reply
Oscar 12th December 2017 - 2:14 pm

thank you Dennis, I just sent you the code :)

Reply
shaker 4th June 2017 - 12:02 am

can make all of that robot in one robot ” robot multi task” ?

Reply
Stephine 2nd August 2016 - 7:19 am

Hi! Good day! Sir, I would like to ask help from you. I am actually building a project that is almost same concept as yours but for some reasons, I can’t ask the codes from you ‘coz of your terms and conditions that I must need to donate atleast $10. I’m actually I’m just a poor guy that needs to finish my studies and I need to make a proposal. So if you don’t mind sir please do send me a mail if we could make some other conditions aside from using money.

Reply
Maurizio 9th November 2015 - 9:39 am

Hi Oscar,
Many thanks for this cool conversion project for pixar Wall-E!
Just donated 10 euros via paypal to your account to say thank you and support your great projects…. it’s a pleasure to read about them on your blog ^^
Now I’m hoping to get arduino code/stuff as for Wall-E soon and start to build mine!

I follow you! ^^
greetings from Italy!
Nasoftz

Reply
San 8th November 2015 - 2:11 am

Hi Oscar,

I really want to build this project and I am shopping WALL-E online. Would you mind providing which model is it? Turns out that there are multiple version of remote controlled WALL-E on the market.

Really appreciate it.
Thanks
San

Reply
Frederic 9th October 2015 - 6:26 pm

Hi Oscar,
another great gadget you built/modded, thank you.
I´m still figuring out the hexapod code so I don´t need another new project yet but Wall-E is my kids all-time favourite movie robot (Johnny 5 left aside, who he is the Uber-robot but just out of my price range big time).

But cheers for the great instructables!
Frederic

Reply
Oscar 12th October 2015 - 11:02 am

Hi Frederic
Same here! I really enjoyed building this robot! wish i have more time to keep working on my Wall-E… he’s still sitting in my bedroom :D

Reply
Mike 7th September 2015 - 11:31 pm

Hi Oscar

Just came across this post as I was going to start my own project with Wall-e. Then I started exploring the quadcopter tutorials – absolutely great stuff. Getting ready to retire so I am lining up my projects. I hope the offer still holds for a copy of the software for wall-e. Love moding code – makes life easier. Either way just donated 10 lbs via paypal to support the site.

Thanks for posting this great stuff.

Reply
Oscar 10th September 2015 - 10:28 am

Thanks Mike for the donation!
Sorry only read your comment, I will email you the code shortly.

Reply
Dom 18th June 2015 - 7:15 pm

you are my hero! Beatifull!!

Reply
Dj_Garfield 13th May 2015 - 2:18 pm

Amazing project !!! Even if it dates a little , how goes Mr Wall E ?
You made a very big thing , but you still have to teach him to dance :) HAHA :)

Reply
Oscar 18th May 2015 - 2:52 pm

haha… wall-E is standing at the corner in my room, staring at me when i am busy building quadcopters :D
Poor wall-E ….

Reply
Frank 28th September 2014 - 10:02 am

Hi!

Thanks for your great explanation and congratulations for this project!

I was checking at ebay.com this Wall-E toy but I’m not sure which one can be useful as the cheap look like the tracks can’t move, and the ones that move are expensive… could you tell me which can I select? Thanks!

Kind regards
Frank

Reply
Oscar 28th September 2014 - 10:08 am

To be honest, I didn’t know which wall-e toy to buy, because I had no idea! the one I have is a random second hand toy I bough from ebay, only costed me $5. But that was almost two years ago…
I think any wall-e toy with a wired remote control should work. but make sure the size is big enough to put servos, arduino board, and battery in it.

Reply
Fadi 3rd November 2013 - 9:37 am

It is a great effort ….. thank you very much for sharing.
Actually I am a beginner in this domain and I will be a gratitude if you can provide the list of parts that you use …:(

Reply
Oscar 5th November 2013 - 10:44 am

parts that i used:

2 motors
1 motor driver
1 wireless camera
4 9g servos (2 for hands 2 for the head)
1 Arduino
1 Voice recognition arduino shield
1 wall-E toy (cheap from ebay)

Reply
Vick 25th July 2013 - 3:30 pm

Very nice ! Are you using wireless camera ?

Thanks

Reply
Oscar 25th July 2013 - 3:49 pm

yes, I was!

Reply
saran raj 2nd February 2016 - 4:49 am

Hi Oscar
hey i am new to this domain i doing my final year project in this domain so plaese help me to do that .
my first problem is how to find the object using image then how can i give input to motors

Reply
joerh99 7th May 2013 - 9:14 am

太酷了!!!

Cool , and your website looks much better than me /…

Reply