feat(huddle): make STT model selectable + add multilingual Parakeet v3 (#2478)#2520
feat(huddle): make STT model selectable + add multilingual Parakeet v3 (#2478)#2520artile wants to merge 1 commit into
Conversation
…block#2478) Huddle transcription was hard-pinned to an English-only STT model, so non-English speech transcribed as garbage (block#2478). There was no language setting, env override, or model-swap path — the loader knew one model. Make the huddle STT model selectable behind a small registry: - `STT_MODELS` in huddle/models.rs is the single source of truth for each model: id, download URL, archive checksum, expected files, sherpa-onnx family, manifest version, license, and language coverage. - `select_stt_model` picks a model from the `BUZZ_STT_MODEL` override, then the system locale (non-English -> multilingual), then the English default. The function is pure and unit-tested. - Two models ship: the existing Parakeet TDT-CTC 110M (English, default) and Parakeet TDT 0.6B v3 (multilingual, 25 European languages). Both carry a pinned SHA-256; the v3 hash was computed from the k2-fsa release archive. - huddle/stt.rs configures the sherpa-onnx recognizer per family: NeMo CTC (single model.int8.onnx) or transducer (encoder/decoder/joiner). tokens.txt is shared. The archive checksum is now optional per model (size cap, safe extraction, and expected-files verification always apply); the English default stays pinned. Coverage note: Parakeet v3 covers European languages only. Korean/CJK — the concrete case in block#2478 — is a follow-up: the registry makes adding a CJK model (e.g. SenseVoice-Small) one data entry plus a family already modelled. Zero unsafe, no panics/unwraps on the new paths. New model selection logic is covered by unit tests in huddle::models. Refs block#2478 Signed-off-by: Taras Kornichuk <dread9ko@gmail.com>
d6c3482 to
9236cb8
Compare
LauraGPT
left a comment
LauraGPT
left a comment
There was a problem hiding this comment.
Thanks for opening the model-selection path. I tested the exact language from #2478 and found one correctness gap: the current locale fallback still routes ko-KR to Parakeet v3, although v3 does not support Korean.
I verified the official sherpa-onnx SenseVoice export end to end:
- archive:
sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17.tar.bz2 - size:
163002883bytes - SHA-256:
7d1efa2138a65b0b488df37f8b89e3d91a60676e416f515b952358d83dfd347e - required files:
model.int8.onnx,tokens.txt - coverage: zh/en/ja/ko/yue
- live CPU check with sherpa-onnx 1.13.4: the packaged 4.608 s Korean sample decoded in 0.211 s, detected
<|ko|>, and produced normal Korean text. The packaged zh/ja/en/yue samples also decoded successfully.
The smallest complete fix is to add a SenseVoice family and registry entry, then select models by declared primary-language tags: zh/ja/ko/yue -> SenseVoice, Parakeet's supported European tags -> v3, en -> the existing English default. Unknown non-English locales should not silently select a model that does not support them.
One licensing detail: SenseVoice is under the FunASR Model Open Source License 1.1, not CC-BY-4.0. The archive includes a LICENSE pointer to the FunASR license, so the generic license_text comments should not assume every model is CC-BY.
| .unwrap_or("") | ||
| .to_ascii_lowercase(); | ||
| if !lang.is_empty() && lang != "en" { | ||
| if let Some(model) = STT_MODELS.iter().find(|m| m.multilingual) { |
There was a problem hiding this comment.
This selects the first multilingual model for every non-English locale. In the reported case, ko-KR therefore selects parakeet-v3, but the registry itself documents that v3 has no Korean coverage. Please dispatch on supported primary-language tags rather than a boolean (and add a regression test for ko-KR). If SenseVoice is deferred, at minimum only auto-select v3 for its 25 declared language tags so this PR does not replace one unsupported model with another.
| // Coverage note: Parakeet v3 covers European languages only. CJK/Korean | ||
| // (the concrete case in #2478) is not covered by this model; the registry | ||
| // makes adding a CJK model (e.g. SenseVoice-Small) a follow-up — one data | ||
| // entry, no engine change beyond a family already handled here. |
There was a problem hiding this comment.
SenseVoice is not one data entry with the currently handled families: sherpa-onnx 1.13.4 exposes it through cfg.model_config.sense_voice, so this needs a SttFamily::SenseVoice arm in stt.rs. The configuration is a single model.int8.onnx path plus language = "auto" and use_itn = true; it should not be configured as NemoCtc or Transducer.
|
I implemented the complete SenseVoice follow-up and opened a collaboration PR directly against your current branch: artile#1 It is based exactly on Validation completed on the exact patch:
|
2 participants
Summary
Huddle transcription was hard-pinned to an English-only STT model, so non-English speech transcribed as garbage (#2478). There was no language setting, env override, or model-swap path — the loader knew exactly one model.
This makes the huddle STT model selectable, and ships a multilingual option alongside the English default.
Closes the "no model-swap path" root cause in #2478; see the coverage note below for what is and isn't covered by the bundled multilingual model.
What changed
huddle/models.rs— a small STT model registry.STT_MODELSis the single source of truth for each model: id, download URL, archive SHA-256, expected files, sherpa-onnx family, manifest version, license text, and language coverage.select_stt_model(override, locale)— pure, unit-tested selection. Precedence (issue Huddle transcription is English-only: non-English speech (e.g. Korean) produces garbled transcripts #2478 options 2 + 3):BUZZ_STT_MODELenv override, when it names a known model (case-insensitive),parakeet-en(default)parakeet-v3huddle/stt.rsconfigures sherpa-onnx per family: NeMo CTC (singlemodel.int8.onnx) or transducer (encoder/decoder/joiner.int8.onnx);tokens.txtis shared. Downloaded on demand via the existing flow.Option. The size cap, safe extraction, and expected-files verification always apply; the English default stays hash-pinned. Both shipped models are pinned today (the v3 hash was computed from the k2-fsa release archive, since upstream publishes none).Model choice — why Parakeet v3, and the honest coverage note
The issue suggested NVIDIA Parakeet v3, and a sherpa-onnx export exists (
sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8), so it is what this PR bundles. It adds 25 European languages with automatic language-ID.Parakeet v3 covers European languages only. The concrete case in #2478 is Korean, which v3 does not cover. This PR removes the architectural root cause (the English-only hard pin) and adds the multilingual model + per-family config path; a CJK model (e.g. SenseVoice-Small, KO/EN/JA/ZH/yue) is now a one-entry follow-up in the registry — the transducer/CTC plumbing it would reuse is already here. Happy to add it in a follow-up if you'd like Korean in this same PR instead.
Testing
cargo check -p buzz-desktop— clean.cargo clippy -p buzz-desktop— clean, no warnings.cargo fmt— clean.cargo test -p buzz-desktop huddle::models— 8 passed, 0 failed. New tests cover: English default, non-English locale → multilingual, override precedence + case-insensitivity, unknown/empty override fallback, registry invariants (unique ids, English default hash-pinned, a multilingual model exists), license-sidecar readiness, and per-family (transducer) readiness.I can't run a live non-English transcription in CI here (it needs the ~465 MB v3 model + a huddle), but the selection/config/readiness paths are unit-tested and the archive contents/subdir/files were verified against the k2-fsa release. Glad to help test Korean once a CJK model lands (ties back to #2478).
Notes
unsafe; no panics/unwraps on the new paths.Refs #2478