"Every physics student memorizes that 45° gives maximum projectile range. But how many have felt it?"
PhysicsMe is a computer vision-powered educational platform that transforms the student's body into a live physics apparatus. Using real-time pose estimation, students control simulations — launching projectiles with their arm angle, swinging pendulums with their wrist, generating waves with their hand — and receive AI-driven tutoring feedback as they learn.
No hardware. No downloads. Just a webcam and a browser.
Physics education is broken in a specific way: students memorize equations without developing physical intuition. Traditional methods rely on:
- Static diagrams in textbooks that don't move
- Pre-built simulations (PhET, etc.) controlled by sliders and buttons — passive, not embodied
- Expensive lab equipment that many schools can't afford
- Abstract formulas disconnected from sensory experience
Research in kinesthetic learning shows that body-based interaction dramatically improves concept retention — yet almost no physics education tool leverages this.
PhysicsMe uses MediaPipe Pose Estimation to track 33 body landmarks at 30+ FPS directly in the browser. These landmarks drive real-time physics simulations rendered on an HTML Canvas overlay.
The student's body becomes the input device:
- Raise your arm → set a launch angle → watch a projectile follow a parabolic arc
- Swing your arm → drive a pendulum → observe period and amplitude relationships
- Wave your hand → generate a sine wave → visualize frequency and amplitude in real-time
Each module includes gamified challenges and an AI tutor that provides personalized feedback based on the student's actual attempts.
┌─────────────────────────────────────────────────────┐
│ │
│ WEBCAM FEED + CANVAS OVERLAY │
│ │
│ O ← Head │
│ /|\ │
│ / | \ ╭────────────╮ │
│ / | ● →→→→ │ ● │ ← Projectile │
│ / | 45° │ ● │ trajectory │
│ / | │ ● │ │
│ / | │ ● │ │
│ 🦶 🦶 │ 🎯 │ ← Target │
│ ╰────────────╯ │
│ │
│ ┌──────────────────────────────────────────────┐ │
│ │ θ = 45° | v₀ = 12 m/s | Range = 14.7 m │ │
│ │ x = v₀cos(θ)t y = v₀sin(θ)t - ½gt² │ │
│ └──────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────┐ │
│ │ 🎯 CHALLENGE: Hit the target at 10m! │ │
│ │ 💡 AI Hint: "Steeper angles trade range │ │
│ │ for height. Try lowering your arm." │ │
│ └──────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘
Concept: Parabolic trajectory, launch angle, initial velocity, range optimization.
How it works:
- Student stands in front of webcam and raises one arm
- System calculates the angle between shoulder → elbow → wrist
- A virtual ball launches from the hand at that angle
- The trajectory is animated in real-time with a trail
- Physics equations update live on screen
Challenges:
- "Hit the target at 8 meters. What angle do you need?"
- "Launch two balls — which goes farther: 30° or 60°? Why do they land at the same spot?"
- "Find the angle that gives maximum range." (Students discover 45° through experimentation)
Physics covered: x = v₀cos(θ)t, y = v₀sin(θ)t - ½gt², complementary angle theorem, max range at 45°.
Concept: Simple harmonic motion, period, frequency, amplitude.
How it works:
- Student extends their arm and swings it side to side
- A virtual pendulum syncs to their wrist position
- Period (T), frequency (f), and amplitude (A) are calculated and displayed
- A live sine-wave graph plots the oscillation over time
Challenges:
- "Can you swing at exactly 1 Hz?"
- "What happens to the period if you swing wider?" (Students discover amplitude independence)
- "Match the target frequency shown on screen."
Physics covered: T = 2π√(L/g), period-length relationship, amplitude independence of period, SHM graphing.
Concept: Transverse waves, frequency, amplitude, wavelength, interference.
How it works:
- Student moves their hand up and down
- A wave propagates horizontally from their hand across the screen
- Moving faster → higher frequency; moving wider → higher amplitude
- Two-player mode: two hands create two wave sources → visible interference pattern
Challenges:
- "Generate a wave with frequency > 2 Hz."
- "Create constructive interference with a partner."
- "Match the target waveform shown on screen."
Physics covered: v = fλ, transverse wave properties, superposition principle, constructive/destructive interference.
┌──────────────────────────────────────────────────────────────┐
│ BROWSER (Client-Side) │
│ │
│ ┌─────────────┐ ┌──────────────┐ ┌────────────────┐ │
│ │ Webcam │───▶│ MediaPipe │───▶│ Landmark │ │
│ │ Feed │ │ Pose │ │ Processor │ │
│ └─────────────┘ │ (33 points) │ │ (angles, │ │
│ └──────────────┘ │ velocities) │ │
│ └───────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────┐ ┌──────────────┐ ┌────────────────┐ │
│ │ Canvas │◀──│ Renderer │◀──│ Physics │ │
│ │ Overlay │ │ (particles, │ │ Engine │ │
│ │ │ │ trails, │ │ (trajectory, │ │
│ │ │ │ equations) │ │ pendulum, │ │
│ │ │ └──────────────┘ │ waves) │ │
│ └─────────────┘ └───────┬────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────┐ │
│ │ Challenge │ │
│ │ System │ │
│ │ (scoring, │ │
│ │ progression) │ │
│ └───────┬────────┘ │
│ │ │
└──────────────────────────────────────────────────┼────────────┘
│
▼
┌────────────────┐
│ LLM API │
│ (Gemini / │
│ Claude) │
│ AI Tutor │
└────────────────┘
| Layer | Technology | Purpose |
|---|---|---|
| Frontend | React 18 + TypeScript | Application framework |
| Styling | Tailwind CSS | Responsive UI |
| Computer Vision | MediaPipe Pose (@mediapipe/tasks-vision) |
Real-time 33-landmark body tracking at 30+ FPS |
| Rendering | HTML Canvas API | Physics simulation overlay on webcam feed |
| Physics | Vanilla TypeScript (no library) | Trajectory, SHM, and wave calculations |
| AI Tutor | Gemini 2.5 Flash API | Personalized educational feedback |
| Deployment | Vercel | Zero-config static deployment |
- MediaPipe over YOLO/OpenCV: Runs in-browser at 30+ FPS with zero training data. Pose estimation is a solved problem — we use it, not reinvent it.
- Canvas over Three.js/WebGL: 2D physics simulations don't need 3D. Canvas is simpler, faster to develop, and performs well for particle animations.
- No backend for core features: The entire CV pipeline and physics engine run client-side. The only server call is the optional AI tutor, which is non-blocking.
- TypeScript: Type safety for physics calculations prevents subtle bugs (angle in radians vs. degrees, coordinate systems).
physicsme/
├── public/
│ └── index.html
├── src/
│ ├── components/
│ │ ├── App.tsx # Root layout and routing
│ │ ├── Camera.tsx # Webcam feed + MediaPipe init
│ │ ├── CanvasOverlay.tsx # Physics rendering layer
│ │ ├── Dashboard.tsx # Module selection screen
│ │ ├── EquationDisplay.tsx # Live equation panel
│ │ ├── ChallengeCard.tsx # Current mission UI
│ │ ├── TutorPanel.tsx # AI feedback display
│ │ └── ScoreBoard.tsx # Points and progression
│ ├── modules/
│ │ ├── ProjectileModule.ts # Projectile motion logic
│ │ ├── PendulumModule.ts # Pendulum / SHM logic
│ │ └── WaveModule.ts # Wave propagation logic
│ ├── engine/
│ │ ├── PhysicsEngine.ts # Core simulation loop
│ │ ├── LandmarkProcessor.ts # Angle/velocity extraction from pose
│ │ ├── ParticleSystem.ts # Animated dots, trails, glows
│ │ └── Renderer.ts # Canvas drawing utilities
│ ├── services/
│ │ ├── PoseService.ts # MediaPipe wrapper
│ │ └── TutorService.ts # LLM API calls
│ ├── challenges/
│ │ ├── projectileChallenges.ts # Projectile mission definitions
│ │ ├── pendulumChallenges.ts # Pendulum mission definitions
│ │ └── waveChallenges.ts # Wave mission definitions
│ ├── utils/
│ │ ├── math.ts # Trig helpers, unit conversions
│ │ └── constants.ts # Gravity, canvas dimensions, etc.
│ └── main.tsx
├── .env # API keys (not committed)
├── .gitignore
├── package.json
├── tsconfig.json
├── tailwind.config.js
├── vite.config.ts
└── README.md
- Node.js 18+
- A device with a webcam
- A Gemini API key (for the AI tutor feature — optional)
# Clone the repository
git clone https://github.com/SakarZaidan/physicsme.git
cd physicsme
# Install dependencies
npm install
# Set up environment variables
cp .env.example .env
# Add your Gemini API key to .env (optional — core features work without it)
# Start development server
npm run devOpen http://localhost:5173 in your browser. Allow camera access when prompted.
npm run build
npm run previewnpx vercel --prodMediaPipe Pose Landmarker returns 33 normalized (0–1) body landmarks per frame:
// Key landmarks used
const LANDMARKS = {
LEFT_SHOULDER: 11,
LEFT_ELBOW: 13,
LEFT_WRIST: 15,
RIGHT_SHOULDER: 12,
RIGHT_ELBOW: 14,
RIGHT_WRIST: 16,
};Raw landmarks are processed into physics-relevant values:
function calculateArmAngle(shoulder, elbow, wrist): number {
// Returns angle in radians between upper arm and forearm
// Used for: projectile launch angle
}
function calculateAngularVelocity(wristPositions: Point[], dt: number): number {
// Finite difference of wrist angle over last N frames
// Used for: pendulum angular velocity
}
function calculateHandVelocity(wristPositions: Point[], dt: number): Vector2 {
// Velocity vector of hand movement
// Used for: wave frequency and amplitude
}Extracted values feed into module-specific equations:
// Projectile: compute trajectory points
for (let t = 0; t < maxTime; t += dt) {
const x = v0 * Math.cos(theta) * t;
const y = v0 * Math.sin(theta) * t - 0.5 * g * t * t;
trajectory.push({ x, y });
}
// Pendulum: compute position from SHM
const angle = amplitude * Math.cos(2 * Math.PI * frequency * t + phase);
// Wave: propagate sine wave
const y = amplitude * Math.sin(2 * Math.PI * (x / wavelength - frequency * t));The Canvas overlay draws:
- Skeleton connections (subtle, semi-transparent)
- Animated particle trails along computed paths
- Real-time equation display with current values highlighted
- Target markers for active challenges
When a student fails a challenge or requests a hint, the attempt data is sent to the LLM:
const prompt = `
You are a friendly physics tutor for a high school student.
The student attempted a projectile motion challenge:
- Target distance: ${target} meters
- Their launch angle: ${angle}°
- Their result: ${result} meters (${result < target ? 'too short' : 'too far'})
In 2-3 sentences:
1. Explain what happened in simple terms
2. Give a specific hint about what to adjust
3. Connect it to the physics concept
Do not give the exact answer. Guide them to discover it.
`;The tutor responds contextually, adapting to repeated failures and building on previous hints.
Challenges are defined as typed objects for easy extension:
interface Challenge {
id: string;
module: 'projectile' | 'pendulum' | 'wave';
title: string;
description: string;
difficulty: 'beginner' | 'intermediate' | 'advanced';
targetCondition: (state: SimulationState) => boolean;
hints: string[];
conceptTaught: string;
}
// Example
const challenge: Challenge = {
id: 'proj-001',
module: 'projectile',
title: 'Bullseye!',
description: 'Hit the target at 10 meters.',
difficulty: 'beginner',
targetCondition: (state) => Math.abs(state.landingX - 10) < 0.5,
hints: [
'Try angles between 30° and 60°.',
'Remember: 45° gives maximum range.',
],
conceptTaught: 'Range depends on both vertical and horizontal components of velocity.',
};| Metric | Target | Method |
|---|---|---|
| Pose detection FPS | ≥ 25 FPS | MediaPipe WASM backend |
| Simulation update rate | 60 FPS | requestAnimationFrame loop |
| Time to first detection | < 2 seconds | Pre-load model on mount |
| AI tutor response time | < 3 seconds | Gemini 2.5 Flash (streaming) |
| Bundle size | < 5 MB | Tree-shaking + lazy module loading |
PhysicsMe's approach is grounded in established learning theory:
- Embodied Cognition (Lakoff & Johnson, 1999): Physical experience shapes conceptual understanding. Students who gesture while learning physics outperform those who don't.
- Constructivism (Piaget): Learners build knowledge through active experimentation, not passive reception.
- Kinesthetic Learning (Fleming's VARK model): 5–15% of students are primarily kinesthetic learners, underserved by lecture-based instruction.
- Immediate Feedback Loops (Hattie, 2009): Real-time feedback has an effect size of 0.73 — among the highest of any educational intervention.
- Multi-Player Mode — Two students create wave interference patterns together
- Mobile AR Mode — Use phone accelerometer + camera for portable experiments
- Electricity Module — Body pose forms a "circuit" (arms = wires, hands = switches)
- Teacher Dashboard — Track class progress across challenges
- Accessibility Mode — Adapt detection for seated users / limited mobility
- Offline PWA — Full functionality without internet (pre-cache MediaPipe model)
- LMS Integration — Export progress to Google Classroom / Canvas
Contributions are welcome. Please follow these guidelines:
- Fork the repository
- Create a feature branch (
git checkout -b feature/wave-interference) - Commit with clear messages (
git commit -m "Add two-source wave interference rendering") - Push to your fork and open a Pull Request
- Ensure all existing challenges still pass
# .env.example
VITE_GEMINI_API_KEY=your_gemini_api_key_here # Optional — AI tutor onlyThe AI tutor is the only feature requiring an API key. All CV and physics simulation features run entirely client-side with no external dependencies.
MIT License — see LICENSE for details.
- MediaPipe by Google — Pose estimation model
- Vite — Build tooling
- Tailwind CSS — Styling
- Google Gemini — AI tutor reasoning
"We didn't build a simulation. We built an experience."