Skip to content

feat: speaker-turn segmentation for the system source (streaming Sortformer)#3

Merged
m5d215 merged 3 commits into
mainfrom
feat/speaker-turn-segmentation
Jun 25, 2026
Merged

feat: speaker-turn segmentation for the system source (streaming Sortformer)#3
m5d215 merged 3 commits into
mainfrom
feat/speaker-turn-segmentation

Conversation

@m5d215

@m5d215 m5d215 commented Jun 25, 2026

Copy link
Copy Markdown
Owner

What & why

The system source is a mix of whatever the call's far side outputs. When two
or more remote people talk back-and-forth, SpeechAnalyzer merges the exchange
into one long final — a single record that scrambles multiple speakers and
loses meaning downstream.

This adds speaker-turn segmentation to the system source: when the speaker
changes, the current utterance is cut so each turn becomes its own record. The
goal is segmentation, not speaker identification — no labels are emitted, so
imperfect speaker tracking (4-speaker cap, label permutation) is tolerable.

This reverses an earlier decision to ship without diarization. That call was
about labeling (low-accuracy spk_N hurts a downstream LLM) and about cost
(67s startup, ANE contention). Reframing the requirement to boundary detection
removes those objections — see CLAUDE.local.md for the full rationale.

How

  • SpeakerSegmenter runs FluidAudio's Streaming Sortformer (CoreML) on the
    system tap audio, in parallel with the transcriber.
  • On a detected turn boundary it calls Transcriber.finalize(through:), letting
    SpeechAnalyzer itself cut the record at the turn — so no token-time
    alignment or text re-splitting is needed, and the jsonl schema is unchanged.
  • The diarizer loads with .cpuAndGPU so it stays off the ANE that the two
    SpeechAnalyzer instances use (avoids the E5RT crash, and lets a second build
    run in parallel).
  • Boundary detection uses the live (tentative) speaker with a small debounce.

Validation

Ran a full work day on real calls, alongside the previous build for A/B:

  • ~31% fewer ≥100-char multi-speaker merges (the target problem).
  • Startup ~1.3s (model cached; first run downloads once), no 67s regression.
  • No E5RT / crash, coexisted with the parallel build all day.

Known rough edges (follow-ups, not blockers)

  • ~1s boundary bleed — boundary detection lags ~1s, so the next speaker's
    first ~1s can land in the previous record. Tightening needs
    finalize(through: boundaryTime) with diarizer↔transcriber clock alignment.
  • Back-channel over-segmentation — rapid "はい"/acknowledgement exchanges get
    cut into many tiny records (system ≤5-char records ~13% → ~20%). A
    min-cut-interval would suppress sub-~1.5s speaker flips.

Config

  • Default on. Set STENO_DIAR=0 to opt out (also degrades gracefully to
    plain transcription if the model fails to load).
  • First run downloads FluidInference/diar-streaming-sortformer-coreml from
    HuggingFace, then caches locally. No other new network use.

Test plan

  • swift build clean
  • make install, production app runs with diarization enabled (no env needed)
  • full-day real-call run, A/B vs prior build

m5d215 and others added 3 commits June 24, 2026 06:11
Add Transcriber.finalizeThroughLatest(), which calls SpeechAnalyzer's
finalize(through: nil) to force-finalize the consumed audio while keeping
the session running (unlike finalizeAndFinish, which ends it). Wire an
env-gated debug timer (STENO_FINALIZE_SPIKE=<sec>, via make dev SPIKE=<sec>)
that fires it periodically on both transcribers.

Groundwork for speaker-turn segmentation: confirmed that finals can be
force-cut mid-utterance and analysis continues without dropping audio
(~100ms from call to published final). A diarizer's speaker-change signal
will later replace the fixed-interval trigger so cuts land on turn
boundaries instead of arbitrary time marks.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add SpeakerSegmenter wrapping FluidAudio's Streaming Sortformer (CoreML) to
detect speaker turn boundaries on the system source. On a turn boundary it
calls the system transcriber's finalize(through:) so a long back-and-forth
no longer merges into one record. The goal is segmentation, not speaker ID,
so label accuracy is not required (4-speaker cap and label permutation are
tolerable).

Gated behind STENO_DIAR=1 (make dev DIAR=1). Diarizer runs on .cpuAndGPU to
avoid contending with the two SpeechAnalyzer instances on the ANE.

Measured on M4 Max:
- startup ~1.3s (model cached; first run downloads once). No 67s regression.
- coexists with 2x SpeechAnalyzer, no E5RT / crash.
- detects turns on acoustically distinct voices; near-identical synthetic
  voices collapse to one slot (confirmed against FluidAudio's own CLI, so
  it's the audio, not the integration).
- end-to-end: a merged multi-turn blob now splits into per-turn records.
  Boundaries fire ~0.5-1s after the true turn change (tentative + debounce).

Known gap: finalize(through: nil) cuts at "now", so ~1s of the next
speaker bleeds into the previous record. Tightening needs
finalize(through: boundaryTime) with diarizer<->transcriber clock alignment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lding

Promote speaker-turn segmentation from an env-gated spike to default behavior
after a full day of real-call validation:

- enable diarization by default (STENO_DIAR=0 to opt out)
- drop the STENO_FINALIZE_SPIKE periodic-finalize harness; its job (proving
  finalize(through:) force-cuts mid-utterance and continues) is done
- reword spike-era comments to production tone
- update README and CLAUDE.local.md for the reversed design decision: why
  on-device streaming diarization is now viable here — segmentation not
  labeling (so label accuracy / 4-speaker cap / permutation are tolerable),
  ~1.3s startup (not 67s), .cpuAndGPU keeps it off the ANE the two
  SpeechAnalyzers use, and finalize(through:) needs no token-time alignment

Full-day result: ~31% fewer >=100-char multi-speaker merges, no E5RT/crash,
coexists with the production build. Known rough edges (~1s boundary bleed,
back-channel over-segmentation) documented as follow-ups.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@m5d215
m5d215 merged commit 35a3392 into main Jun 25, 2026
2 checks passed
@m5d215
m5d215 deleted the feat/speaker-turn-segmentation branch June 25, 2026 00:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant