Skip to content

feat(speech): add Deepgram as an alternate STT/TTS provider (Soniox remains default)#1

Open
pramodthe wants to merge 6 commits into
contextablemark:mainfrom
pramodthe:feat/speech-providers-deepgram
Open

feat(speech): add Deepgram as an alternate STT/TTS provider (Soniox remains default)#1
pramodthe wants to merge 6 commits into
contextablemark:mainfrom
pramodthe:feat/speech-providers-deepgram

Conversation

@pramodthe

Copy link
Copy Markdown

Summary

Adds Deepgram as an optional speech provider alongside the existing Soniox integration, behind a thin provider facade. Soniox remains the default and its behavior is unchanged.

  • New facade (speech_cfg / speech_stt / speech_tts) so main, push‑to‑talk, and the AG‑UI client stay provider‑agnostic.
  • Deepgram backends: Listen v1 (STT) and Speak v1 (TTS) over WebSocket — linear16 @ 16 kHz, reusing the same ES8311 mic/speaker path as Soniox.
  • Runtime selection from the AMOLED‑setup captive portal: pick the provider, paste that vendor's API key, choose a voice.
  • Backward compatible: existing devices (legacy soniox_key, or no speech_prov set) keep using Soniox with no reconfiguration.

Motivation

The device is a first‑class AG‑UI voice client; the STT/TTS vendor shouldn't be baked in. This makes the speech layer pluggable so users can choose a provider (and so a third can be added later) without touching the rest of the app.

What changed

Area Change
speech_cfg Resolve provider + API key from NVS; caches the provider (invalidated on portal save); migrates the legacy soniox_key.
speech_stt / speech_tts Facades that dispatch to the active backend and tear down the previous backend's mic / drain task before opening the other provider.
deepgram_stt Deepgram Listen v1 WSS backend (nova-3, interim results, endpointing/utterance‑end).
deepgram_tts Deepgram Speak v1 WSS backend (Aura‑2 voices, streaming PCM → speaker, cancelable for barge‑in).
net_prov/portal Provider dropdown + per‑provider voice lists; requires a key when switching provider; resets tts_voice to the new provider's default.
soniox_client / soniox_tts_client Add *_deinit() for clean provider switching; read the key via speech_cfg. Otherwise unchanged.
main Call the facades instead of Soniox directly.
docs New docs/speech-providers.md (NVS keys, architecture, how to add a provider).

Design notes

  • Single ES8311 owner. bsp_audio_codec_microphone_init() mints a new codec‑dev handle each call, so switching providers without a reboot could double‑own the mic. The STT facade tears down the previous backend (esp_codec_dev_close + esp_codec_dev_delete) before initializing the next, keeping a single owner of the shared I²S‑RX.
  • Sequential TLS preserved. The Deepgram backends mirror the Soniox open → feed → finish → drain lifecycle; PSRAM buffers, and no new concurrent‑TLS path.
  • Single‑writer transcript. The Deepgram STT commit path runs only on the WebSocket task (matching Soniox), avoiding cross‑task races on the transcript buffer.

Compatibility / migration

  • Default provider is Soniox on fresh flash and on any device that has not set speech_prov.
  • The legacy soniox_key is still read as a fallback when the provider is Soniox.
  • Switching provider in the portal now requires entering a key for the new vendor.

NVS keys (appcfg)

Key Meaning
speech_prov deepgram | soniox (default soniox)
speech_key API key for the active provider
soniox_key Legacy Soniox‑only key (fallback when provider is Soniox)
tts_voice Provider‑specific voice / model id

Testing

  • idf.py build on ESP‑IDF 5.5.x / esp32s3 — builds clean, no new warnings (esp32_agui.bin, 31% app partition free).
  • Manual (on Waveshare ESP32‑S3‑Touch‑AMOLED‑1.8):
    • Portal → provider = Deepgram + Deepgram key + Wi‑Fi + AG‑UI URL → PTT → transcript → spoken reply
    • Portal → switch to Soniox + Soniox key → same flow (no reboot)
    • Barge‑in (new PTT during TTS) cancels playback
    • Existing device with only legacy soniox_key still boots straight into Soniox

Notes for maintainers

See docs/speech-providers.md for the facade layout and how to add a third provider. Rule of thumb: never call a provider backend from main — always go through the facade.

pramodthe and others added 6 commits July 13, 2026 02:23
Introduce a thin provider facade (speech_cfg / speech_stt / speech_tts) over the
existing Soniox clients and add Deepgram Listen v1 (STT) + Speak v1 (TTS)
WebSocket backends (linear16 @ 16 kHz, same ES8311 mic/speaker path). The
provider is chosen at runtime from the AMOLED-setup captive portal; Soniox
remains the default and is unchanged for existing devices (legacy soniox_key and
fresh flash both keep Soniox).

- speech_cfg: resolve provider + key from NVS with a cached provider value
  (invalidated on portal save) plus legacy soniox_key migration.
- speech_stt / speech_tts: facades that tear down the previous backend's mic /
  drain task before opening the other provider, so a no-reboot portal switch
  never double-owns the ES8311.
- portal: provider dropdown + per-provider voice lists; requires a key when
  switching provider and resets tts_voice to the new provider's default.
- soniox_client / soniox_tts_client: add *_deinit() for clean provider switching
  and read the key via speech_cfg; otherwise unchanged.

Docs: docs/speech-providers.md. Builds clean on ESP-IDF 5.5.x / esp32s3.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On-device companion UI, ported from the NIMO reference sketch and scaled to
the 368x448 AMOLED:

- face_engine: NIMO-style animated face rendered into a PSRAM canvas via a
  small software GFX (rounded-rect eyes, rect pupils + glint, overlay-masked
  expressions, spring physics, blink). Moods: normal/listen/think/speak/happy/
  sleepy/sad/angry/surprised/dizzy/love/suspicious. Shake -> dizzy (orbiting
  pupils + stars) -> angry (red eyes, brows, "!" marks, teeth). Talking mouth
  opens/closes during speech.
- companion_pages: Eyes / Clock / Chat with tap-to-cycle and voice-state
  auto-switch; face visibility/activity gated to the Eyes page.
- imu_qmi8658: QMI8658 6-axis IMU driver (smoothed tilt for gaze, shake
  intensity, orientation label for ambient context).
- chat_ui: drive face moods from voice/turn status; wire companion pages.

Speech + tooling refinements across the provider facade and device layer:
deepgram STT/TTS, soniox client, speech_stt/speech_tts, device_tools,
net_prov/portal, app_cfg, and main (PTT/turn wiring). Docs + project guide.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tap now toggles Eyes <-> Chat only; drops the clock overlay labels and the 500ms refresh timer. The agent answers time queries, so the on-device clock was redundant.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Was aura-2-asteria-en (female). aura-2-orion-en is already in the portal voice list; override via NVS.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…atchdog

- chat_ui: HTTPS JPEG show_image overlay + face/status hooks
- net_prov: prefer last-good SSID on boot; build-time clock fallback when
  NTP/HTTP time is blocked; net_creds_remove_prefix() API
- deepgram_stt/tts: connection + WS reassembly refinements
- bsp: longer in-flight flush wait before brightness tx_param
- main: wire the speech_* facade, PTT max-listen watchdog, gate a turn on
  wall-clock sync so TLS to Deepgram/AG-UI has a valid cert window

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…back

Addresses two review findings on PR contextablemark#1:

- bsp_display_brightness_set now returns ESP_ERR_TIMEOUT (without touching the
  shared QSPI IO) when a race-free tx_param can't be guaranteed — the lock is
  held by an active LVGL flush, or a flush is still in flight after the bounded
  drain. It previously always returned ESP_OK, so chat_ui screen_power_task's
  "retry next poll" guards were dead and the tx_param-vs-flush UI-STALL race
  could still fire on a lock timeout. Early boot (no LVGL adapter) still sends
  unlocked as before.

- net_time_http_fallback again fetches the real wall clock via the HTTPS Date
  header before falling back to the firmware build time, and the unused
  time_from_http_body helper is removed (was defined-but-unused →
  -Wunused-function). On NTP-blocked hotspots the clock is accurate whenever
  TLS can complete, with build time only as a last resort.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant