Gesture-Controlled Robotic Car

A simple robotic car controlled by hand gestures using:

Arduino Uno
Two continuous-rotation servo motors
Python
OpenCV
MediaPipe
USB serial communication

The webcam detects the direction of the user's hand. Python converts the hand direction into a command and sends it to the Arduino. The Arduino then controls the two servo motors to move the car forward, backward, or stop.

1. Project Overview

This project combines computer vision, Python, Arduino, and servo motors.

The complete flow is:

Hand gesture
    ↓
Webcam captures video
    ↓
OpenCV reads each video frame
    ↓
MediaPipe detects the hand
    ↓
Python calculates the hand angle
    ↓
Python decides: Forward, Backward, or Stop
    ↓
Python sends F, B, or S through USB serial
    ↓
Arduino receives the command
    ↓
Arduino controls the servo motors
    ↓
The car moves

2. Features

Hand-gesture control
Forward movement
Backward movement
Stop command
Real-time hand tracking
Real-time angle display
Serial communication between Python and Arduino
Simple cardboard chassis
Beginner-friendly electronics setup

3. Components Required

Component	Quantity	Purpose
Arduino Uno	1	Receives commands and controls the servos
Continuous-rotation servo motors	2	Rotate the left and right wheels
Breadboard	1	Distributes 5V and GND connections
Jumper wires	Several	Connect Arduino, breadboard, and servos
Cardboard	As required	Used as the car body
Cardboard wheels	2	Attached to the servo horns
USB cable	1	Powers Arduino and carries Python commands
Computer or laptop	1	Runs Python and processes webcam video
Webcam	1	Captures hand gestures

Important: This project assumes that the two servo motors are continuous-rotation servos. A normal positional servo usually cannot rotate continuously.

4. Role of Each Component

Arduino Uno

The Arduino is the hardware controller.

It:

Receives commands from Python
Reads one command at a time
Controls the left servo
Controls the right servo
Stops the servos when it receives S

The Arduino does not detect the hand. The computer performs the hand detection.

Servo Motors

The servo motors act as wheel motors.

One servo drives the left wheel
One servo drives the right wheel

Because the motors are mounted on opposite sides, they must rotate in opposite directions for the car to move straight.

For continuous-rotation servos:

Around 90 = Stop
Below 90  = Rotate in one direction
Above 90  = Rotate in the opposite direction

The exact stop value may be slightly different for different servos.

Breadboard

The breadboard does not create power.

It only distributes connections.

Example:

Arduino 5V
    ↓
Breadboard positive rail
    ├── Left servo power
    └── Right servo power

The same applies to GND.

Jumper Wires

Jumper wires carry:

5V power
Ground connection
Servo control signals

USB Cable

The USB cable has two jobs:

Upload the Arduino code
Carry live commands from Python to Arduino

It may also power the Arduino.

Computer

The computer:

Runs the Python program
Opens the webcam
Detects the hand
Calculates the hand angle
Sends commands to the Arduino

Webcam

The webcam captures the user's hand.

The Python program processes the live video frame by frame.

5. Understanding Power, Ground, and Signal

Each servo normally has three wires.

Wire	Purpose
Power	Supplies electrical energy, usually 5V
Ground	Completes the electrical path
Signal	Tells the servo how to move

The simplest way to understand the connection is:

5V → Servo → GND

The signal wire does not provide enough power to run the motor. It only carries instructions.

Why GND Is Needed

Electricity needs a complete path.

Arduino 5V → Servo → Arduino GND

Without GND, the electrical path is incomplete and the servo cannot work correctly.

Does GND Switch the Servo Off?

No.

GND is required while the servo is powered.

The servo stops because the Arduino sends a stop instruction:

servo.write(90);

The servo still has 5V and GND connected.

6. Wiring Connections

Left Servo

Servo Wire	Connect To
Power	Breadboard 5V rail
Ground	Breadboard GND rail
Signal	Arduino pin 9

Right Servo

Servo Wire	Connect To
Power	Breadboard 5V rail
Ground	Breadboard GND rail
Signal	Arduino pin 10

Breadboard Power

Arduino Pin	Breadboard
5V	Positive rail
GND	Ground rail

Simple Wiring Diagram

Arduino 5V
    ↓
Breadboard positive rail
    ├── Left servo power
    └── Right servo power

Arduino GND
    ↓
Breadboard ground rail
    ├── Left servo ground
    └── Right servo ground

Arduino pin 9  → Left servo signal
Arduino pin 10 → Right servo signal

7. Important Power Note

Two servo motors can draw more current than the Arduino can safely provide.

The breadboard does not increase the available power. It only distributes it.

For a small lightweight prototype, Arduino USB power may work. However, symptoms of insufficient power include:

Servo shaking
Weak movement
Arduino restarting
Serial disconnection
Random movement

For a more reliable project, use a separate regulated 5V power supply for the servos.

External 5V positive → Both servo power wires
External GND         → Both servo ground wires
Arduino GND          → External GND
Arduino pins 9, 10   → Servo signal wires

The Arduino ground and external power ground must be connected together.

8. Software Used

Arduino IDE

Used to:

Write Arduino code
Compile the code
Upload the code to the Arduino Uno

Python

Used to:

Read the webcam
Detect the hand
Calculate the hand angle
Send movement commands

OpenCV

OpenCV handles the camera and image display.

It:

Opens the webcam
Reads each video frame
Flips the frame
Converts BGR to RGB
Draws lines and text
Displays the final window

MediaPipe

MediaPipe understands the hand inside the image.

It:

Detects the hand
Tracks the hand
Finds 21 hand landmark points
Gives the position of the wrist and finger joints

Simple difference:

OpenCV = Gets and displays the image
MediaPipe = Detects the hand inside the image

PySerial

PySerial connects Python to Arduino through the USB serial port.

It sends commands such as:

F
B
S

9. Why Video Is Processed as Images

A video is a fast sequence of images called frames.

Example:

Frame 1
Frame 2
Frame 3
Frame 4
...

OpenCV reads one frame at a time:

success, frame = cap.read()

Each frame is processed separately.

Because many frames are processed every second, it looks like live video.

10. Why the Camera Frame Is Flipped

This line flips the image horizontally:

frame = cv2.flip(frame, 1)

It creates a mirror-like view.

Without flipping:

You move your hand right
The screen may show it moving left

With flipping:

You move your hand right
The screen also shows it moving right

The flip mainly makes the controls feel natural.

11. BGR and RGB

A digital image uses three color channels:

R = Red
G = Green
B = Blue

OpenCV normally stores images as:

BGR = Blue, Green, Red

MediaPipe expects:

RGB = Red, Green, Blue

Therefore, the frame is converted:

rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

The actual image does not change. Only the order of the color values changes.

12. Hand Landmarks Used

MediaPipe detects 21 hand landmarks.

This project uses:

Landmark 0 = Wrist
Landmark 9 = Base of the middle finger

The program draws a line between these two points.

Wrist → Middle finger base

The direction of this line represents the orientation of the hand.

13. Arduino Code

Create a file named:

robot_car.ino

Use this code:

#include <Servo.h>

Servo leftServo;
Servo rightServo;

void setup() {
  Serial.begin(9600);

  leftServo.attach(9);
  rightServo.attach(10);

  leftServo.write(90);
  rightServo.write(90);
}

void loop() {
  if (Serial.available()) {
    char cmd = Serial.read();

    if (cmd == 'F') {
      leftServo.write(80);
      rightServo.write(100);

      Serial.println("FORWARD");
    }

    else if (cmd == 'B') {
      leftServo.write(100);
      rightServo.write(80);

      Serial.println("BACKWARD");
    }

    else if (cmd == 'S') {
      leftServo.write(90);
      rightServo.write(90);

      Serial.println("STOP");
    }
  }
}

14. Arduino Code Explanation

Importing the Servo Library

#include <Servo.h>

This library makes it easy to control servo motors.

Creating Servo Objects

Servo leftServo;
Servo rightServo;

These objects represent the two servo motors.

Starting Serial Communication

Serial.begin(9600);

The Arduino communicates with Python at 9600 baud.

Python must use the same speed.

Connecting Servo Objects to Pins

leftServo.attach(9);
rightServo.attach(10);

The left servo uses pin 9.

The right servo uses pin 10.

Stopping at Startup

leftServo.write(90);
rightServo.write(90);

This sets both continuous-rotation servos near their neutral value.

Checking for a Command

if (Serial.available())

The Arduino checks whether Python has sent data.

Reading the Command

char cmd = Serial.read();

The Arduino reads one character.

Possible commands:

F = Forward
B = Backward
S = Stop

Forward Movement

leftServo.write(80);
rightServo.write(100);

The two motors rotate in opposite directions because they are mounted on opposite sides.

Backward Movement

leftServo.write(100);
rightServo.write(80);

Both motor directions are reversed.

Stop

leftServo.write(90);
rightServo.write(90);

Both servos receive the neutral command.

15. Python Code

Create a file named:

gesture_control.py

Use this code:

import cv2
import mediapipe as mp
import math
import serial
import time

# ---------- Arduino ----------
arduino = serial.Serial("COM3", 9600)

# Wait so Arduino can reset and initialize
time.sleep(2)

# ---------- MediaPipe ----------
mp_hands = mp.solutions.hands
mp_draw = mp.solutions.drawing_utils

hands = mp_hands.Hands(
    max_num_hands=1,
    min_detection_confidence=0.7,
    min_tracking_confidence=0.7
)

cap = cv2.VideoCapture(0)

last_command = ""

while True:
    success, frame = cap.read()

    if not success:
        break

    frame = cv2.flip(frame, 1)

    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

    results = hands.process(rgb)

    command = "S"

    if results.multi_hand_landmarks:
        hand = results.multi_hand_landmarks[0]

        mp_draw.draw_landmarks(
            frame,
            hand,
            mp_hands.HAND_CONNECTIONS
        )

        wrist = hand.landmark[0]
        middle_base = hand.landmark[9]

        h, w, _ = frame.shape

        x1 = int(wrist.x * w)
        y1 = int(wrist.y * h)

        x2 = int(middle_base.x * w)
        y2 = int(middle_base.y * h)

        cv2.line(
            frame,
            (x1, y1),
            (x2, y2),
            (0, 255, 0),
            3
        )

        angle = math.degrees(
            math.atan2(y2 - y1, x2 - x1)
        )

        cv2.putText(
            frame,
            f"Angle: {int(angle)}",
            (20, 50),
            cv2.FONT_HERSHEY_SIMPLEX,
            1,
            (0, 255, 0),
            2
        )

        if angle < -110:
            command = "F"

        elif angle > -50:
            command = "B"

        else:
            command = "S"

        cv2.putText(
            frame,
            f"CMD: {command}",
            (20, 100),
            cv2.FONT_HERSHEY_SIMPLEX,
            1,
            (0, 0, 255),
            2
        )

    if command != last_command:
        arduino.write(command.encode())
        print("Sent:", command)
        last_command = command

    cv2.imshow("Robot Control", frame)

    if cv2.waitKey(1) & 0xFF == 27:
        break

cap.release()
arduino.close()
cv2.destroyAllWindows()

16. Python Code Explanation

Importing Libraries

import cv2
import mediapipe as mp
import math
import serial
import time

Library	Purpose
`cv2`	Camera and image processing
`mediapipe`	Hand detection
`math`	Angle calculation
`serial`	Communication with Arduino
`time`	Adds a short delay

Connecting Python to Arduino

arduino = serial.Serial("COM3", 9600)

COM3 is the Arduino port
9600 is the baud rate

The COM port may be different on another computer.

Waiting for Arduino

time.sleep(2)

The Arduino often restarts when the serial connection opens.

This delay gives it time to initialize.

Creating the Hand Detector

hands = mp_hands.Hands(
    max_num_hands=1,
    min_detection_confidence=0.7,
    min_tracking_confidence=0.7
)

This configuration:

Detects one hand
Requires 70% detection confidence
Requires 70% tracking confidence

Opening the Webcam

cap = cv2.VideoCapture(0)

0 usually means the default webcam.

Reading a Frame

success, frame = cap.read()

success tells whether the frame was captured
frame contains the image

Flipping the Frame

frame = cv2.flip(frame, 1)

This creates a mirror-like view.

Converting BGR to RGB

rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

OpenCV gives BGR images.

MediaPipe expects RGB images.

Detecting the Hand

results = hands.process(rgb)

MediaPipe analyzes the frame and returns hand landmarks.

Default Stop Command

command = "S"

If no hand is detected, the program sends Stop.

This is a useful safety behavior.

Selecting Hand Points

wrist = hand.landmark[0]
middle_base = hand.landmark[9]

These two points define the hand direction.

Converting Coordinates

MediaPipe gives positions between 0 and 1.

x1 = int(wrist.x * w)
y1 = int(wrist.y * h)

The code converts them into actual image pixel positions.

Calculating the Angle

angle = math.degrees(
    math.atan2(y2 - y1, x2 - x1)
)

This calculates the direction from the wrist to the middle-finger base.

Gesture Decision

if angle < -110:
    command = "F"

elif angle > -50:
    command = "B"

else:
    command = "S"

The angle is divided into three areas.

Angle	Command
Less than `-110`	Forward
Greater than `-50`	Backward
Between `-110` and `-50`	Stop

Sending Only Changed Commands

if command != last_command:
    arduino.write(command.encode())

This avoids sending the same command repeatedly in every frame.

Closing the Program

Press the Escape key to stop the program.

The cleanup code:

cap.release()
arduino.close()
cv2.destroyAllWindows()

releases the camera, closes serial communication, and closes the OpenCV window.

17. Command Table

Python Command	Meaning	Arduino Action
`F`	Forward	Left servo 80, right servo 100
`B`	Backward	Left servo 100, right servo 80
`S`	Stop	Both servos 90

18. Installing the Software

Install Python

Install Python 3 on your computer.

Check the installation:

python --version

Install Python Packages

Run:

pip install opencv-python mediapipe pyserial

Optional requirements.txt:

opencv-python
mediapipe
pyserial

Install from the file:

pip install -r requirements.txt

Install Arduino IDE

Install the Arduino IDE and connect the Arduino Uno.

19. How to Run the Project

Step 1: Build the Car

Attach the two servo motors to the cardboard body
Attach cardboard wheels to the servo horns
Connect the servo wires
Connect Arduino 5V and GND to the breadboard rails

Step 2: Upload Arduino Code

Open robot_car.ino
Select Arduino Uno
Select the correct COM port
Upload the code

Step 3: Find the Arduino COM Port

In Arduino IDE, check:

Tools → Port

Example:

COM3

Update the Python code if necessary:

arduino = serial.Serial("COM3", 9600)

Step 4: Close Arduino Serial Monitor

The Arduino Serial Monitor must be closed before running Python.

Only one program can normally use the same COM port at a time.

Step 5: Run Python

python gesture_control.py

Step 6: Show Your Hand

Move your hand through the gesture zones.

The screen will show:

Hand landmarks
Hand direction line
Calculated angle
Current command

Step 7: Exit

Press the Escape key.

20. Common Problems and Solutions

Problem: Python Cannot Open COM Port

Possible error:

PermissionError
Access is denied

Solution:

Close Arduino Serial Monitor
Check the COM port
Disconnect and reconnect Arduino
Restart the Python program

Problem: Wrong COM Port

Check Arduino IDE:

Tools → Port

Then update:

serial.Serial("COM3", 9600)

Problem: Servos Do Not Stop

The exact neutral value may not be 90.

Try:

leftServo.write(89);
rightServo.write(91);

Adjust each servo separately until it stops.

Problem: Car Moves in the Wrong Direction

Swap the servo values.

Example:

leftServo.write(100);
rightServo.write(80);

Or reverse the physical orientation of one servo.

Problem: Hand Is Not Detected

Try:

Better lighting
Clear background
Move hand closer to the camera
Keep the full hand visible
Lower confidence values slightly

Example:

min_detection_confidence=0.5

Problem: Servo Shakes or Arduino Restarts

The servos may be drawing too much current.

Use a separate regulated 5V supply for the servos.

Problem: Camera Does Not Open

Try another camera number:

cv2.VideoCapture(1)

Problem: Movement Changes Too Quickly

The angle may be close to a boundary.

Possible improvements:

Add larger stop zones
Average several angle values
Add a small delay
Add hysteresis

21. Project Limitations

Current limitations:

Only forward, backward, and stop are supported
No left or right turning
USB cable keeps the robot connected to the computer
Servo speed is fixed
Hand-angle thresholds may require calibration
Arduino USB power may be insufficient for two servos
The cardboard structure may not be mechanically stable

22. Future Improvements

Possible upgrades:

Add left and right turning
Add Bluetooth communication
Add Wi-Fi communication
Use an ESP32
Use a battery
Add separate servo power
Add obstacle detection
Add ultrasonic sensors
Add speed control
Add smoothing for hand angles
Add more gestures
Add a mobile application
Add a proper chassis
Add four wheels
Add a camera directly on the robot

23. Wireless Version Idea

A larger version can replace USB with:

USB power         → Battery or external power supply
USB communication → Bluetooth, Wi-Fi, or radio

Example:

Laptop webcam
    ↓
Python detects gesture
    ↓ Wi-Fi or Bluetooth
Robot receives command
    ↓
Arduino or ESP32 controls motors

24. Suggested Project Folder Structure

gesture-controlled-robot-car/
├── README.md
├── arduino/
│   └── robot_car.ino
├── python/
│   ├── gesture_control.py
│   └── requirements.txt
├── images/
│   ├── robot-car.jpg
│   ├── wiring-diagram.png
│   └── gesture-demo.png
└── docs/
    └── project-documentation.pdf

25. What This Project Teaches

This project introduces:

Arduino programming
Servo motor control
Basic electronics
Power, ground, and signal wiring
Serial communication
Python programming
Computer vision
OpenCV
MediaPipe
Hand landmarks
Angle calculation
Hardware-software integration

26. Safety Notes

Do not connect a 9V battery directly to a 5V servo
Avoid powering high-current motors directly from the Arduino
Check wiring before connecting power
Do not short 5V and GND
Use a regulated power supply
Keep all grounds connected when using separate supplies
Disconnect power before changing wires

27. Final Summary

This project works by combining computer vision and hardware control.

OpenCV captures the camera
MediaPipe detects the hand
Python calculates the hand direction
Python sends F, B, or S
Arduino receives the command
Arduino controls the servos
The car moves

The main idea is simple:

Computer understands the gesture
Arduino controls the hardware

Author

Created as a beginner robotics and computer-vision project.

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
arduino		arduino
python		python
README.MD		README.MD

Folders and files

Latest commit

History

Repository files navigation

Gesture-Controlled Robotic Car

1. Project Overview

2. Features

3. Components Required

4. Role of Each Component

Arduino Uno

Servo Motors

Breadboard

Jumper Wires

USB Cable

Computer

Webcam

5. Understanding Power, Ground, and Signal

Why GND Is Needed

Does GND Switch the Servo Off?

6. Wiring Connections

Left Servo

Right Servo

Breadboard Power

Simple Wiring Diagram

7. Important Power Note

8. Software Used

Arduino IDE

Python

OpenCV

MediaPipe

PySerial

9. Why Video Is Processed as Images

10. Why the Camera Frame Is Flipped

11. BGR and RGB

12. Hand Landmarks Used

13. Arduino Code

14. Arduino Code Explanation

Importing the Servo Library

Creating Servo Objects

Starting Serial Communication

Connecting Servo Objects to Pins

Stopping at Startup

Checking for a Command

Reading the Command

Forward Movement

Backward Movement

Stop

15. Python Code

16. Python Code Explanation

Importing Libraries

Connecting Python to Arduino

Waiting for Arduino

Creating the Hand Detector

Opening the Webcam

Reading a Frame

Flipping the Frame

Converting BGR to RGB

Detecting the Hand

Default Stop Command

Selecting Hand Points

Converting Coordinates

Calculating the Angle

Gesture Decision

Sending Only Changed Commands

Closing the Program

17. Command Table

18. Installing the Software

Install Python

Install Python Packages

Install Arduino IDE

19. How to Run the Project

Step 1: Build the Car

Step 2: Upload Arduino Code

Step 3: Find the Arduino COM Port

Step 4: Close Arduino Serial Monitor

Step 5: Run Python

Step 6: Show Your Hand

Step 7: Exit

20. Common Problems and Solutions

Problem: Python Cannot Open COM Port

Packages