Three projects fused into one single-page app that runs 100% in the browser, on-device:
- Chat — an LLM (
SmolLM2) via Transformers.js. - Voice — replies are spoken with Kokoro TTS (the engine behind tts.rocks).
- Face — a three.js WebGPU Face Cap head that lip-syncs and emotes.
| Streaming | LLM tokens stream straight into Kokoro, so the face starts talking ~1 sentence in. |
| Viseme lip-sync | Phonemes → mouth shapes (ah/ee/oo/closed…) blended with live audio amplitude. |
| Emotion | A zero-cost heuristic reads each reply and sets the face's mood (joy/curious/sad/…). |
| Gaze & life | Eyes track the cursor, saccade, blink; the head turns toward you while speaking. |
| Voice input | Hold/click 🎙 to talk — Whisper (whisper-tiny.en) transcribes locally. |
| Captions | Karaoke-style captions sync to the spoken words. |
| Memory | The conversation persists across reloads. |
| Settings | Voice, speaking rate, model (135M/360M), precision, and compute device. |
| Off-main-thread | The LLM + Kokoro run in a Web Worker, so the main thread stays free and the face renders in sync with the audio. |
| WebGPU | The 3D face uses WebGPU; the LLM defaults to CPU (so the GPU is free for the face) and can opt into WebGPU in settings. |
| PWA / offline | Installable; a service worker caches everything so WEB mode keeps working with no internet. |
| WEB ⇄ LOCAL | Switch every asset source between CDN+HF and the local project folder. |
| Dark / light | Instant theme toggle. |
Serve over HTTP (ES modules, WebGPU and the service worker need a real origin):
python -m http.server 5173 # or: npx serve .Open the page, type or hit 🎙, press Enter. First run downloads the models and caches them.
The face needs WebGPU (Chrome/Edge), falling back to WebGL2. Chat + voice work regardless.
A. Cache button (WEB mode). Settings → Cache everything for offline. A service worker stores the libraries + current models in the browser; afterwards WEB mode runs with no network.
B. LOCAL mode (ship the folder). Vendor everything into the project, then flip the switch to LOCAL:
powershell -ExecutionPolicy Bypass -File tools/fetch-offline.ps1 # ~296 MB (q8). add -With360 for the 360M modelIf LOCAL is selected without the assets present, the app stays on WEB and tells you what to run.
index.html bootstrap: CSP + SRI import map + SW/PWA + theme → loads app.js
app.js orchestrator: chat → stream → speak → emote, settings, offline, controls
infer-worker.js Web Worker running BOTH the LLM and Kokoro (shared transformers instance)
infer.js main-thread client for the worker
face.js three.js WebGPU head (visemes + mood + gaze)
speech.js audio player: gapless playback, viseme timeline, captions, metering
stt.js Whisper push-to-talk
emotion.js heuristic mood persist.js conversation memory
styles.css analog-broadcast design system (2 themes)
sw.js offline service worker manifest.webmanifest icons/
vendor/stub-empty.js node built-in no-ops for kokoro-js
tools/ fetch-offline.ps1 + Playwright tests (smoke / e2e-plus / stt / offline)
vendor/<libs>, models/, node_modules/ are git-ignored (regenerated by the script / npm).
node tools/smoke.mjs http://127.0.0.1:5173/ # wiring, theme, face
node tools/e2e-plus.mjs http://127.0.0.1:5173 # streaming, captions, ON AIR, stop, settings, SW
node tools/stt-test.mjs http://127.0.0.1:5173 # Whisper transcription
node tools/offline-test.mjs http://127.0.0.1:5173 # full loop with all external network blocked