feat: speaker-turn segmentation for the system source (streaming Sortformer)#3
Merged
Conversation
Add Transcriber.finalizeThroughLatest(), which calls SpeechAnalyzer's finalize(through: nil) to force-finalize the consumed audio while keeping the session running (unlike finalizeAndFinish, which ends it). Wire an env-gated debug timer (STENO_FINALIZE_SPIKE=<sec>, via make dev SPIKE=<sec>) that fires it periodically on both transcribers. Groundwork for speaker-turn segmentation: confirmed that finals can be force-cut mid-utterance and analysis continues without dropping audio (~100ms from call to published final). A diarizer's speaker-change signal will later replace the fixed-interval trigger so cuts land on turn boundaries instead of arbitrary time marks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add SpeakerSegmenter wrapping FluidAudio's Streaming Sortformer (CoreML) to detect speaker turn boundaries on the system source. On a turn boundary it calls the system transcriber's finalize(through:) so a long back-and-forth no longer merges into one record. The goal is segmentation, not speaker ID, so label accuracy is not required (4-speaker cap and label permutation are tolerable). Gated behind STENO_DIAR=1 (make dev DIAR=1). Diarizer runs on .cpuAndGPU to avoid contending with the two SpeechAnalyzer instances on the ANE. Measured on M4 Max: - startup ~1.3s (model cached; first run downloads once). No 67s regression. - coexists with 2x SpeechAnalyzer, no E5RT / crash. - detects turns on acoustically distinct voices; near-identical synthetic voices collapse to one slot (confirmed against FluidAudio's own CLI, so it's the audio, not the integration). - end-to-end: a merged multi-turn blob now splits into per-turn records. Boundaries fire ~0.5-1s after the true turn change (tentative + debounce). Known gap: finalize(through: nil) cuts at "now", so ~1s of the next speaker bleeds into the previous record. Tightening needs finalize(through: boundaryTime) with diarizer<->transcriber clock alignment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lding Promote speaker-turn segmentation from an env-gated spike to default behavior after a full day of real-call validation: - enable diarization by default (STENO_DIAR=0 to opt out) - drop the STENO_FINALIZE_SPIKE periodic-finalize harness; its job (proving finalize(through:) force-cuts mid-utterance and continues) is done - reword spike-era comments to production tone - update README and CLAUDE.local.md for the reversed design decision: why on-device streaming diarization is now viable here — segmentation not labeling (so label accuracy / 4-speaker cap / permutation are tolerable), ~1.3s startup (not 67s), .cpuAndGPU keeps it off the ANE the two SpeechAnalyzers use, and finalize(through:) needs no token-time alignment Full-day result: ~31% fewer >=100-char multi-speaker merges, no E5RT/crash, coexists with the production build. Known rough edges (~1s boundary bleed, back-channel over-segmentation) documented as follow-ups. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The
systemsource is a mix of whatever the call's far side outputs. When twoor more remote people talk back-and-forth,
SpeechAnalyzermerges the exchangeinto one long
final— a single record that scrambles multiple speakers andloses meaning downstream.
This adds speaker-turn segmentation to the system source: when the speaker
changes, the current utterance is cut so each turn becomes its own record. The
goal is segmentation, not speaker identification — no labels are emitted, so
imperfect speaker tracking (4-speaker cap, label permutation) is tolerable.
This reverses an earlier decision to ship without diarization. That call was
about labeling (low-accuracy
spk_Nhurts a downstream LLM) and about cost(67s startup, ANE contention). Reframing the requirement to boundary detection
removes those objections — see
CLAUDE.local.mdfor the full rationale.How
SpeakerSegmenterruns FluidAudio's Streaming Sortformer (CoreML) on thesystem tap audio, in parallel with the transcriber.
Transcriber.finalize(through:), lettingSpeechAnalyzeritself cut the record at the turn — so no token-timealignment or text re-splitting is needed, and the jsonl schema is unchanged.
.cpuAndGPUso it stays off the ANE that the twoSpeechAnalyzerinstances use (avoids the E5RT crash, and lets a second buildrun in parallel).
Validation
Ran a full work day on real calls, alongside the previous build for A/B:
Known rough edges (follow-ups, not blockers)
first ~1s can land in the previous record. Tightening needs
finalize(through: boundaryTime)with diarizer↔transcriber clock alignment.cut into many tiny records (system ≤5-char records ~13% → ~20%). A
min-cut-interval would suppress sub-~1.5s speaker flips.
Config
STENO_DIAR=0to opt out (also degrades gracefully toplain transcription if the model fails to load).
FluidInference/diar-streaming-sortformer-coremlfromHuggingFace, then caches locally. No other new network use.
Test plan
swift buildcleanmake install, production app runs with diarization enabled (no env needed)