feat(speech): add Deepgram as an alternate STT/TTS provider (Soniox remains default)#1
Open
pramodthe wants to merge 6 commits into
Open
Conversation
Introduce a thin provider facade (speech_cfg / speech_stt / speech_tts) over the existing Soniox clients and add Deepgram Listen v1 (STT) + Speak v1 (TTS) WebSocket backends (linear16 @ 16 kHz, same ES8311 mic/speaker path). The provider is chosen at runtime from the AMOLED-setup captive portal; Soniox remains the default and is unchanged for existing devices (legacy soniox_key and fresh flash both keep Soniox). - speech_cfg: resolve provider + key from NVS with a cached provider value (invalidated on portal save) plus legacy soniox_key migration. - speech_stt / speech_tts: facades that tear down the previous backend's mic / drain task before opening the other provider, so a no-reboot portal switch never double-owns the ES8311. - portal: provider dropdown + per-provider voice lists; requires a key when switching provider and resets tts_voice to the new provider's default. - soniox_client / soniox_tts_client: add *_deinit() for clean provider switching and read the key via speech_cfg; otherwise unchanged. Docs: docs/speech-providers.md. Builds clean on ESP-IDF 5.5.x / esp32s3. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On-device companion UI, ported from the NIMO reference sketch and scaled to the 368x448 AMOLED: - face_engine: NIMO-style animated face rendered into a PSRAM canvas via a small software GFX (rounded-rect eyes, rect pupils + glint, overlay-masked expressions, spring physics, blink). Moods: normal/listen/think/speak/happy/ sleepy/sad/angry/surprised/dizzy/love/suspicious. Shake -> dizzy (orbiting pupils + stars) -> angry (red eyes, brows, "!" marks, teeth). Talking mouth opens/closes during speech. - companion_pages: Eyes / Clock / Chat with tap-to-cycle and voice-state auto-switch; face visibility/activity gated to the Eyes page. - imu_qmi8658: QMI8658 6-axis IMU driver (smoothed tilt for gaze, shake intensity, orientation label for ambient context). - chat_ui: drive face moods from voice/turn status; wire companion pages. Speech + tooling refinements across the provider facade and device layer: deepgram STT/TTS, soniox client, speech_stt/speech_tts, device_tools, net_prov/portal, app_cfg, and main (PTT/turn wiring). Docs + project guide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tap now toggles Eyes <-> Chat only; drops the clock overlay labels and the 500ms refresh timer. The agent answers time queries, so the on-device clock was redundant. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Was aura-2-asteria-en (female). aura-2-orion-en is already in the portal voice list; override via NVS. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…atchdog - chat_ui: HTTPS JPEG show_image overlay + face/status hooks - net_prov: prefer last-good SSID on boot; build-time clock fallback when NTP/HTTP time is blocked; net_creds_remove_prefix() API - deepgram_stt/tts: connection + WS reassembly refinements - bsp: longer in-flight flush wait before brightness tx_param - main: wire the speech_* facade, PTT max-listen watchdog, gate a turn on wall-clock sync so TLS to Deepgram/AG-UI has a valid cert window Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…back Addresses two review findings on PR contextablemark#1: - bsp_display_brightness_set now returns ESP_ERR_TIMEOUT (without touching the shared QSPI IO) when a race-free tx_param can't be guaranteed — the lock is held by an active LVGL flush, or a flush is still in flight after the bounded drain. It previously always returned ESP_OK, so chat_ui screen_power_task's "retry next poll" guards were dead and the tx_param-vs-flush UI-STALL race could still fire on a lock timeout. Early boot (no LVGL adapter) still sends unlocked as before. - net_time_http_fallback again fetches the real wall clock via the HTTPS Date header before falling back to the firmware build time, and the unused time_from_http_body helper is removed (was defined-but-unused → -Wunused-function). On NTP-blocked hotspots the clock is accurate whenever TLS can complete, with build time only as a last resort. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Deepgram as an optional speech provider alongside the existing Soniox integration, behind a thin provider facade. Soniox remains the default and its behavior is unchanged.
speech_cfg/speech_stt/speech_tts) somain, push‑to‑talk, and the AG‑UI client stay provider‑agnostic.linear16 @ 16 kHz, reusing the same ES8311 mic/speaker path as Soniox.soniox_key, or nospeech_provset) keep using Soniox with no reconfiguration.Motivation
The device is a first‑class AG‑UI voice client; the STT/TTS vendor shouldn't be baked in. This makes the speech layer pluggable so users can choose a provider (and so a third can be added later) without touching the rest of the app.
What changed
speech_cfgsoniox_key.speech_stt/speech_ttsdeepgram_sttnova-3, interim results, endpointing/utterance‑end).deepgram_ttsnet_prov/portaltts_voiceto the new provider's default.soniox_client/soniox_tts_client*_deinit()for clean provider switching; read the key viaspeech_cfg. Otherwise unchanged.maindocs/speech-providers.md(NVS keys, architecture, how to add a provider).Design notes
bsp_audio_codec_microphone_init()mints a new codec‑dev handle each call, so switching providers without a reboot could double‑own the mic. The STT facade tears down the previous backend (esp_codec_dev_close+esp_codec_dev_delete) before initializing the next, keeping a single owner of the shared I²S‑RX.Compatibility / migration
speech_prov.soniox_keyis still read as a fallback when the provider is Soniox.NVS keys (
appcfg)speech_provdeepgram|soniox(defaultsoniox)speech_keysoniox_keytts_voiceTesting
idf.py buildon ESP‑IDF 5.5.x / esp32s3 — builds clean, no new warnings (esp32_agui.bin, 31% app partition free).soniox_keystill boots straight into SonioxNotes for maintainers
See
docs/speech-providers.mdfor the facade layout and how to add a third provider. Rule of thumb: never call a provider backend frommain— always go through the facade.