GestureFly 🛸

Real-Time Hand Gesture Controlled Virtual Drone using OpenCV, MediaPipe, Python, WebSockets, and Three.js.

GestureFly is a production-quality, responsive 3D web application where users can pilot a quadcopter in their browser using hand movements captured through a standard webcam. It uses computer vision heuristics to classify gestures on a Python backend and streams commands via WebSockets to a WebGL visualizer.

---

Features

• 🖐 Real-time Hand Tracking: Powered by MediaPipe Hands for high-precision, low-latency landmark tracking.
• 📐 ML-Free Classifier: Geometric heuristics perform gesture classification without needing training data or heavy model files.
• 🧵 Multi-Threaded Architecture: Isolates computer vision loops from WebSocket networking to ensure high framerates (60+ FPS web visualizer, 30 FPS camera tracker).
• 🚀 Three.js Simulation: Realistically animated 3D quadcopter model constructed from primitives, with shadows, glowing collision LEDs, and rotor dynamics.
• 🌪 Realistic Aerodynamics: Drone tilts dynamically into translations (pitch/roll), has physical inertia (acceleration/drag), and features hover bobbing.
• 📟 Futuristic HUD Dashboard: Cockpit overlay showing real-time telemetry (Altitude, Pitch, Roll, Yaw, Speed, active FPS, and gesture confidence indicators).
• ⌨ Keyboard Override Fallback: Standard WASD + Space/Shift + Q/E controls for testing when a webcam is unavailable.
• 🔌 Self-Healing Connection: Reconnects to backend server automatically if connection is lost.

---

Folder Structure

gesture-drone-control/
│
├── backend/
│   ├── main.py                 # Core entry point (FastAPI server + camera thread)
│   ├── hand_tracker.py         # MediaPipe tracking wrapper
│   ├── gesture_classifier.py   # Geometric heuristics classification
│   ├── websocket_server.py     # FastAPI connection state manager
│   ├── config.py               # Constants, thresholds, and server ports
│   ├── requirements.txt        # Pinned python dependencies
│   └── utils.py                # Math helpers, EMA smoothing, and log setup
│
├── frontend/
│   ├── index.html              # Cockpit HUD template
│   ├── style.css               # Futuristic cyberpunk glassmorphic style
│   ├── scene.js                # Three.js viewport, grid, lights, and shadows
│   ├── drone.js                # Drone mesh, kinematics, and keyboard listeners
│   ├── websocket.js            # Self-healing WebSocket client
│   └── app.js                  # Main render coordinator and telemetry updater
│
├── .gitignore                  # Git cache exemptions
├── architecture.md             # In-depth architectural details & algorithms
└── README.md                   # Setup and operations guide

---

Installation & Setup Guide

Ensure you have Python 3.11 or 3.12 installed on your system.

Step 1: Clone or Copy Project Files

Place all generated project files into a folder named gesture-drone-control.

Step 2: Set Up Python Backend

1. Find the directory of gesture-drone-control and Open your terminal or Command Prompt and navigate to the project directory:

   dir gesture-drone-control

   cd C:\Users\Pranav Ram\OneDrive\Desktop\Projects\gesture-drone-control

2. Create a virtual environment:

   python -m venv .venv

3. Activate the virtual environment:
   • Windows (PowerShell):

     .venv\Scripts\Activate.ps1

   • Windows (Command Prompt):

     .venv\Scripts\activate.bat

   • macOS / Linux:

     source .venv/bin/activate

4. Install required python libraries:

   pip install -r backend/requirements.txt

---

How to Run

1. Launch the Backend Server

Run the python app from the root directory:

python -m backend.main

Upon running, you should see:

• A local server starting at ws://127.0.0.1:8000/ws.
• An OpenCV GUI window titled GestureFly - Camera Feed opening to show your mirrored webcam stream with hand skeleton overlays.

Note: You can close the backend at any time by pressing 'q' inside the webcam feed window or by typing Ctrl+C in the terminal.

2. Launch the Web Frontend

Simply open the index.html file in any modern web browser:

• Double click frontend/index.html to open it in Chrome, Edge, or Firefox.
• Or serve it using a local server if preferred:

  # Example: using python's built-in server
  python -m http.server 8080 --directory frontend

  Then navigate to http://localhost:8080 in your browser.

Once open, you'll see the connection indicator in the HUD switch to green (CONNECTED), and you are ready to fly!

---

Flight Operations Guide

Ensure you are in a well-lit room and sitting directly in front of the camera.

Gesture Control Mapping

Gesture	Movement Action	Description
Open Palm	TAKE_OFF	Extend all 5 fingers. Lifts the drone from the ground to the hover altitude.
Closed Fist	HOVER	Fold all 5 fingers. Drone stops translating and hovers in place.
Raise Palm	MOVE_UP	Open Palm + lift hand to the upper 35% of the camera frame.
Lower Palm	MOVE_DOWN	Open Palm + lower hand to the lower 35% of the camera frame.
Tilt Left	TURN_LEFT	Open Palm + tilt fingers left. Drone rotates around Y-axis.
Tilt Right	TURN_RIGHT	Open Palm + tilt fingers right. Drone rotates around Y-axis.
Thumbs Up	SPEED_UP	Extend only thumb pointing up. Increases flight speed multiplier.
2x Open Palms	MOVE_FORWARD	Show both hands in open palm state. Drone flies forward in the heading direction.
2x Closed Fists	LAND	Show both hands closed in fists. Drone descends and lands on the pad.

---

Keyboard Fallback Controls

You can bypass gesture tracking at any time by pressing keys on your keyboard:

• W / S: Pitch forward / backward (movement direction)
• A / D: Roll left / right (translation drift)
• Space / Shift: Ascend / Descend (Altitude control)
• Q / E: Rotate Yaw left / right (Rotation control)

---

Troubleshooting

• AttributeError: module 'mediapipe' has no attribute 'solutions': This happens because newer MediaPipe releases (v0.10.35+) have deprecated and removed the legacy Solutions API (mp.solutions.hands). The project requires mediapipe==0.10.14 to retain this API. If you run into this, make sure your virtual environment is activated and run:

  pip install mediapipe==0.10.14

• Camera fails to launch: Ensure no other app (Zoom, Teams, etc.) is currently using your webcam. You can adjust the camera device index in backend/config.py (CAMERA_INDEX = 1 if using an external webcam).
• FastAPI / WebSocket error: Check if port 8000 is already in use by another local application. If so, change WS_PORT in backend/config.py and wsUrl in frontend/app.js to an open port (e.g. 8081).
• MediaPipe package installation error: Ensure you are using a supported Python version (3.11 or 3.12). On Windows, some packages require the C++ Build Tools installed. Alternatively, upgrade pip: python -m pip install --upgrade pip and retry.