GestureAI is a real-time chat application that translates American Sign Language (ASL) into text using your webcam. We built it as our final apprenticeship project ("Svenneprøve").
While it works as a standard messaging app with text and image sharing, the main draw is the ASL integration. If you toggle the camera on, a custom AI model reads your hand signs and types the corresponding characters directly into your chat input field.
- ASL Translation: Turn on your webcam, and the custom AI model translates ASL signs into text characters.
- Real-time Chat: Instant messaging powered by WebSockets.
- Media Uploads: Share images directly in the chat.
- Dark/Light Mode: Full theming support.
- Dockerized: The entire stack (frontend, backend, Redis, Postgres) spins up with a single compose file.
The app is split into a SvelteKit frontend and a Python backend, tied together with WebSockets and Redis.
- Frontend (Jesper): Svelte, SvelteKit, TypeScript, WebSockets, TailwindCSS, shadcn-svelte,
- Backend (Viktor): Python (ASL model, API, Analytics page)
- Infrastructure (Viktor & Jesper): Docker, Redis, PostgreSQL, Nginx
You need Docker and Docker Compose installed. You'll also need to configure the environment variables before starting.
-
Clone the repository
git clone https://github.com/GestureAI/gestureai.git cd gestureai -
Set up environment variables Create an
.envfile in the root directory. Based on thedocker-compose.yml, you'll need to define variables like database credentials (DB_USER,DB_PASS,DB_NAME), yourUPLOADTHING_TOKEN, and other backend secrets. -
Run the containers
docker-compose up -d --build
-
Access the app Open
http://localhost:5173/in your browser. Nginx handles the routing automatically.
- Enter a username to join the global chat room.
- Click the Gesture AI badge to enable Gesture Mode and grant webcam permissions.
- Perform ASL signs clearly in front of the camera, and the recognized characters will appear in your input field.
Distributed under the MIT License. See LICENSE for more information.