diff --git a/docs/research-2026-07-asr-diarization-refresh.md b/docs/research-2026-07-asr-diarization-refresh.md new file mode 100644 index 0000000..72c797a --- /dev/null +++ b/docs/research-2026-07-asr-diarization-refresh.md @@ -0,0 +1,192 @@ +# Research: ASR / diarization / alignment tech refresh (July 2026) + +Status: **research** (2026-07-30). Survey of speech-AI developments since millet's +first release (March 2026) that could improve its core pipeline, with a scoped, +license-aware upgrade plan. **No streaming/real-time work is in scope** (explicit +decision). Turkish + Farsi support is a **hard constraint**. Whisper +large-v3-turbo stays the **default** ASR backend. + +This is a planning document. It records the landscape, the constraints, and a +phased plan. It does not change any code. + +--- + +## 1. Where millet is today + +Current core pipeline (`millet/transcribe.py`): + +1. Load dual-channel WAV → mono for transcription. +2. Transcribe with WhisperX / faster-whisper — default `whisper-large-v3-turbo` + (CTranslate2). MLX path on Apple Silicon; opt-in Parakeet ONNX path. +3. Align with **wav2vec2** for word-level timestamps + (`ALIGNMENT_MODELS`, `transcribe.py:212`). +4. Diarize with **pyannote `speaker-diarization-community-1`** via WhisperX's + `DiarizationPipeline` (`transcribe.py:1876`, `2109`). +5. Merge diarization with transcription; voiceprint naming; outputs. + +Relevant observations: + +- millet is **already current** on two big pieces: it uses pyannote + community-1 (the 4.0-era OSS model) and Whisper large-v3-turbo. +- The **Parakeet backend exists but is pinned to the English-only v2** + (`nemo-parakeet-tdt-0.6b-v2`, `parakeet.py:42`). +- The **alignment stage is the oldest component**: 2020-era wav2vec2 models, + with heavyweight per-language downloads (tr/fa ≈ 1.2 GB each, + `transcribe.py:230`). + +--- + +## 2. Constraints (decided) + +- **Offline / local-first.** Every recommended model must run locally with no + mandatory API dependency. +- **License-permissive.** Prefer MIT / Apache 2.0. Treat CC-BY-4.0 (attribution + required) as second-class. +- **Keep Turkish + Farsi.** Rules out EN-only / EU-only models as *defaults*. +- **No streaming / real-time.** Batch pipeline only. +- **Whisper large-v3-turbo stays the default.** Newer models are opt-in + backends, never a forced replacement. + +--- + +## 3. The 2026 landscape + +The "Whisper monoculture" ended in 2026. Whisper large-v3 has been overtaken on +the [Hugging Face Open ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard) +by roughly ten open models — but leaderboard WER (English short-form) is a poor +proxy for millet's job (multilingual, long-form, noisy meeting audio). + +### 3.1 ASR / transcription + +| Model | License | Languages | Local | Notes | +|---|---|---|---|---| +| Whisper large-v3-turbo (**current default**) | **MIT** | 99+ | ✅ | 809M, ~6× faster than large-v3, mature runtime (whisper.cpp / faster-whisper / WhisperX). Still the best license-clean multilingual default. | +| **Qwen3-ASR 0.6B / 1.7B** | **Apache 2.0** | 52 (incl. **tr, fa**) | ✅ | Qwen3-Omni-based. Robust in low-SNR/noisy audio. Ships inference toolkit + separate forced-alignment model for timestamps (11 langs). Community reports it competitive with / beating Whisper on many inputs. Ecosystem younger than Whisper. | +| Parakeet-TDT-0.6b-v3 | CC-BY-4.0 | 25 EU (**no tr/fa**) | ✅ | 600M FastConformer-TDT, ~6.34% avg WER, ~49× Whisper throughput, native word/segment timestamps, ONNX/CoreML/ANE. Attribution burden; drops Turkish/Farsi. | +| Canary-Qwen-2.5B | CC-BY-4.0 | English | ✅ | SALM: transcribe **and** summarize in one model (~5.63% WER). EN-only, 2.5B, CC-BY. Niche. | +| IBM Granite Speech 4.1 / 3.3 | Apache 2.0 | EN-focused | ✅ | Top of leaderboard by WER, but English-focused. | +| Voxtral / Kyutai STT | Apache 2.0 / CC-BY | few | ✅ | Streaming-oriented — **out of scope**. | + +**Takeaway:** with a "license-permissive + keep tr/fa" filter, the standout +*new* backend is **Qwen3-ASR (Apache 2.0, 52 langs)** — not Parakeet v3. Parakeet +v3 is a genuine speed win but is CC-BY and EU-only, so it belongs as a +speed-focused opt-in, not a default. + +### 3.2 Diarization + +| Model | License | Notes | +|---|---|---| +| pyannote `speaker-diarization-community-1` (**current**) | CC-BY-4.0 | pyannote.audio 4.0 OSS flagship. ~50% less speaker confusion vs 3.1; VBx clustering (not AHC); new `exclusive_speaker_diarization` output for cleaner STT reconciliation. ~11–13% DER range on academic benchmarks. | +| DiariZen | (OSS) | Competitive open alternative (~13.3% DER in the 2026 benchmark paper, arXiv:2509.26177), where pyannote precision-2 (commercial) leads at 11.2%. Candidate only if millet's audio exposes pyannote weaknesses. | +| NVIDIA Sortformer (offline/streaming) | CC-BY-4.0 (NeMo) | End-to-end; streaming variant is real-time. 4-speaker cap, English-optimized, heavy NeMo dep. Streaming → **out of scope**; offline variant not compelling enough to displace pyannote for millet. | + +**Takeaway:** millet is already on the best OSS diarizer. The open question is +whether it is *fully exploiting* pyannote 4.0 (exclusive mode, VBx) through the +WhisperX wrapper, or leaving those gains on the table. + +### 3.3 Alignment (word timestamps) + +millet's wav2vec2 alignment (`ALIGNMENT_MODELS`) is the oldest component. +Newer ASR models (Parakeet v3, Qwen3-ASR) emit **native word-level timestamps**, +which can remove the separate wav2vec2 stage for those backends — fewer +downloads, less code, simpler pipeline. + +--- + +## 4. Plan (phased, license-aware, no streaming) + +### Phase 0 — Verification & benchmark harness (prerequisite, gates everything) + +1. **Confirm pyannote 4.0 feature usage.** Determine whether WhisperX's + `DiarizationPipeline` actually surfaces community-1's `exclusive_speaker_diarization` + output and VBx clustering, or whether a direct `pyannote.audio` 4.0 call is + needed to unlock the reduced-confusion benefits. (The dual-diarize + consolidation heuristics — `_consolidate_dual_diarize_speakers` — and the + backchannel guards from the cluster-bleed spike may be partly compensating + for reconciliation weaknesses that 4.0 exclusive mode fixes upstream.) +2. **Build a small labeled benchmark set** with multilingual meeting clips + (**including tr + fa**), reference transcripts, and RTTM. Compute **WER** and + **DER**. No model swap ships without a measured win on this harness. + +### Phase 1 — ASR upgrades + +**1a. Bump the existing Parakeet backend v2 → v3** (low effort — `parakeet.py:42`) +- Set `DEFAULT_PARAKEET_MODEL = "nemo-parakeet-tdt-0.6b-v3"`. +- Gains 25 EU languages + auto-language-detection for free, same ~600M size, + native word timestamps. +- Keep opt-in (`--asr-backend parakeet`). **Document the CC-BY-4.0 attribution + obligation.** Does not affect tr/fa (Parakeet has neither). + +**1b. Add a Qwen3-ASR backend (Apache 2.0, multilingual) — strategic addition** +- New module `millet/qwen_asr.py` mirroring `parakeet.py`'s isolation pattern: + same WhisperX-shaped result dict (`segments`/`language`/`text`), VAD chunking + for long recordings, lazy `millet download qwen-asr` fetch, opt-in via + `--asr-backend`. +- Covers **tr/fa permissively**, where Parakeet v3 cannot — a cleaner + multilingual story than the current wav2vec2-per-language approach. +- Uses its native timestamps → feeds Phase 2. + +**1c. Keep Whisper large-v3-turbo as the default** (MIT, 99 langs, mature). +Newer models remain opt-in backends. + +### Phase 2 — Simplify the alignment stage + +- For ASR backends that emit native word-level timestamps (Parakeet v3, + Qwen3-ASR), **bypass the wav2vec2 alignment step**, gated on backend + capability. +- Keep wav2vec2 alignment for the Whisper path where it is still required. +- Benefit: fewer heavyweight downloads (esp. tr/fa ≈ 1.2 GB each), simpler + pipeline, less code in the alignment registry. +- Validate native-timestamp quality per-language against the Phase 0 harness + before dropping alignment for that language. + +### Phase 3 — Diarization enhancements + +1. **Adopt pyannote 4.0 exclusive mode** if Phase 0 shows WhisperX isn't already + using it — cleaner ASR↔diarization reconciliation and ~50% less speaker + confusion vs 3.1. Some existing dual-diarize consolidation heuristics may then + be simplifiable. +2. **Evaluate DiariZen** only if the benchmark exposes pyannote weaknesses on + millet's real audio. Optional. + +### Out of scope (this cycle) + +- **Streaming / real-time transcription and diarization.** (Sortformer streaming, + Voxtral/Kyutai realtime, `millet live`.) Explicitly deferred. +- **Summarization LLM refresh.** Minor/incremental; not part of this ASR-focused + effort. + +--- + +## 5. Sequencing & risk + +| Phase | Effort | Value | Risk | +|---|---|---|---| +| 0 Benchmark harness | Low–Med | Critical (gates all) | Low | +| 1a Parakeet v2→v3 | **Low** | Med | Low (opt-in) | +| 3 pyannote 4.0 exclusive mode | Low–Med | Med–High | Low | +| 1b Qwen3-ASR backend | Med | **High** | Med | +| 2 Alignment simplification | Med | Med | Med (timestamp quality) | + +**Recommended order:** 0 → 1a → 3 → 1b → 2. + +--- + +## 6. Decisions locked + +1. **No streaming.** Batch pipeline only. +2. **Keep Turkish + Farsi.** → Qwen3-ASR (Apache 2.0) is the multilingual bet; + Parakeet v3 stays a speed-focused, EU-only, opt-in backend. +3. **Whisper large-v3-turbo stays the default.** No auto-selection heuristic; + new models are opt-in. + +## 7. Sources (retrieved 2026-07-30) + +- Hugging Face Open ASR Leaderboard — https://huggingface.co/spaces/hf-audio/open_asr_leaderboard +- Qwen3-ASR-0.6B — https://huggingface.co/Qwen (Apache 2.0, 52 languages) +- Parakeet-TDT-0.6b-v3 — https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3 +- Canary-1B-v2 & Parakeet-TDT-0.6B-v3 report — https://arxiv.org/abs/2509.14128 +- pyannote community-1 — https://huggingface.co/pyannote/speaker-diarization-community-1 +- pyannote 4.0 / community-1 announcement — https://www.pyannote.ai/blog/community-1 +- Benchmarking Diarization Models (2026) — https://arxiv.org/html/2509.26177v1 +- ASR license comparison (2026) — https://www.marktechpost.com/2026/07/23/best-open-speech-recognition-asr-models-in-2026-wer-languages-latency-and-license-compared/