feat(ambient): pluggable ASR backend — SenseVoice for multilingual capture - #41
Conversation
…pture
The passive ambient STT was a Nov-2023 English-only LibriSpeech Zipformer;
in a multilingual (Mandarin+English) household non-English speech decodes to
gibberish. Add an AMBIENT_ASR_BACKEND selector: "zipformer" (default,
unchanged) or "sense_voice" (SenseVoice-Small — zh/en/ja/ko/yue, auto
per-utterance language detect + ITN punctuation).
- config: asr_backend + sense_voice_dir (default zipformer → existing installs
unaffected; edge flips via systemd drop-in).
- pipeline: factory dispatch (from_sense_voice vs from_transducer); the
arena-off provider string reaches both. transducer-only decode knobs stay
scoped to that branch. meta.asr carries the backend label.
- _asr_feats: omit ys_log_probs when empty (CTC/SenseVoice emit none) and
strip SenseVoice's <|...|> delimiters so meta.lang is clean ("zh" not
"<|zh|>") — caught by real-model E2E, not the stubbed unit tests.
Bench (edge, int8, 4 threads): SenseVoice-2024-07-17 English 0% ER (vs
zipformer 12.5%), Mandarin/code-switch usable (vs gibberish), RTF 0.12.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1f507c3261
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if ys: | ||
| feats["ys_log_probs"] = ys |
There was a problem hiding this comment.
When an utterance has no log-probs (SenseVoice/CTC, and any empty transducer vector), this now omits ys_log_probs while _SHADOW_VER stays p0.1. That mixes old p0.1 rows where the key was always present with new p0.1 rows where it can be absent, so offline quality analysis filtered by shadow_ver can misinterpret missing values as corrupt data or bias experiments; bump the shadow version with this schema change.
Useful? React with 👍 / 👎.
1 participant
Summary
The passive ambient STT is a Nov-2023 English-only LibriSpeech Zipformer. On a multilingual install, non-English speech gets force-decoded into English gibberish. This adds a pluggable ASR backend selector so multilingual installs can run SenseVoice-Small (zh/en/ja/ko/yue, auto per-utterance language detect + ITN punctuation) while English-only installs keep the incumbent Zipformer unchanged.
Change
config.py:AMBIENT_ASR_BACKEND(zipformerdefault |sense_voice) +AMBIENT_SENSE_VOICE_DIR. Default iszipformer, so existing installs are byte-for-byte unaffected; a multilingual edge flips the backend via a systemd drop-in env line. Only the selected backend's model files need to be present.pipeline.py: factory dispatch inAmbientEngine.__init__(from_sense_voicevsfrom_transducer). The arena-off ORT provider string reaches both backends; the transducer-only decode knobs (decoding_method/max_active_paths) stay scoped to the transducer branch.meta.asrnow carries the active backend label._asr_feats: omitsys_log_probswhen empty (SenseVoice/CTC emit none, so it's no longer stored as noise on every row) and strips SenseVoice's<|…|>token delimiters someta.langis a clean value ("zh", not"<|zh|>").Why SenseVoice-2024-07-17 (int8)
Benchmarked on the edge VM (4 threads, int8) against the incumbent and the newer 2025-09-09 release, scored by token error rate vs ground-truth references:
English must not regress: the 2024 model matches/beats Zipformer on English and fixes the multilingual failure, so it wins. The 2025 WenetSpeech-Yue release traded English accuracy for expanded dialect coverage and is rejected.
Testing
_asr_featsomits empty log-probs + strips<|…|>delimiters). Full ambient suite green (154 tests); ruff clean.sense_voicebranch builds a real recognizer and decodes English / non-English / code-switch correctly with cleanlangtags, and the defaultzipformerpath still constructs. The delimiter bug above was caught by this E2E, not by the stubbed unit tests.Deploy
No-op for existing installs (default unchanged). A multilingual edge sets
AMBIENT_ASR_BACKEND=sense_voice, installs the model into~/models/sense-voice(README documents the download), and restarts the ambient bridge.🤖 Generated with Claude Code