Skip to content

Recordings lost their first ~2 seconds; Whisper couldn't install when URLSession can't reach Hugging Face - #13

Open
nicw wants to merge 34 commits into
mainfrom
fix-vad-preload-and-download-fallback
Open

Recordings lost their first ~2 seconds; Whisper couldn't install when URLSession can't reach Hugging Face#13
nicw wants to merge 34 commits into
mainfrom
fix-vad-preload-and-download-fallback

Conversation

@nicw

@nicw nicw commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Two small fixes, one branch (based on granite-shadow-transcription, so it carries that line's history — the fixes themselves are the last 3 commits, +~600/−20 across 4 files).

Problem 1 — every recording trimmed its opening words

The VAD model loaded from scratch inside start() on every session (~2s on an M2 Max), and the mic tap only exists after — so the first couple of words of every memo and call were simply never captured. Nic noticed it as "initial audio is trimmed."

Fix: load the VAD model once and reuse it — a single-flight load kicked at app launch ([ENGINE-VAD-PRELOAD] in the log), with start() falling back to loading it on demand if the preload hasn't finished or failed. The mic tap now goes up immediately on record.

Problem 2 — some networks make Whisper uninstallable in-app

On Nic's network (documented in the benchmark-results doc), Apple's URLSession cannot connect to the HF Xet CDN at any timeout while curl connects in ~30ms — so WhisperKit's downloader always fails and the model picker's auto-revert kicks in every time. Graceful, but Whisper could never actually be installed from the app.

Fix: when the SDK download throws, a curl-based fetcher (CurlModelFetcher, /usr/bin/curl via Process) takes over: lists the variant via the HF tree API, downloads into the exact HubApi layout WhisperKit loads from, tokenizer included. TOME_FORCE_CURL_MODEL_FETCH=1 forces the path for testing on a healthy network.

Live-verified end-to-end: forced-fallback run fetched the full 1.5 GB variant in 51 seconds and the model loaded and served (lastGood flipped to whisper). VAD preload confirmed firing in the launch log.


Reviewer's notes (Claude)

  • Cancellation: a cancelled provisioning cycle terminates the curl child via withTaskCancellationHandler + per-file Task.checkCancellation — no zombie multi-GB downloads. Continuation resumes exactly once (no terminationHandler; cancel's terminate() unblocks the same waitUntilExit, an isCancelled flag converts that resume to CancellationError).
  • Resume semantics: -C - was deliberately dropped — with --fail it 416s on already-complete files and kills re-runs. Replaced by size-check skip against the tree listing's byte sizes (file-granularity resume, no 416 class at all). Tokenizer files have no listed size and always redownload — intentional, they're tiny.
  • VAD reuse safety: verified against the pinned FluidAudio source — VadManager wraps an immutable CoreML model; per-run streaming state is external (makeStreamState), and two live transcribers already share one instance today. stop() never nil'd the manager.
  • No shell interpolation: curl args go as an argv array to Process — spaces in "Application Support" are a non-issue.
  • Observability: fetch outcome (COMPLETED / FAILED + reason) now hits the unified log; previously a fetch failure was visible only in the Settings failure line, which cost us a confused test cycle.
  • Tests: 161/161 (7 new: tree-JSON parsing, URL/path derivation, progress math, size-skip decision). Known minor left as-is: stdout is drained to EOF before stderr (with -sS stderr stays tiny; documented).
  • One unverified-by-human bit: the trim fix is log-verified (preload fires before any recording) but Nic hadn't yet done the say-a-word-at-record-instant ear test when this PR was opened — daily use will confirm it immediately.

🤖 Generated with Claude Code

nicw and others added 30 commits July 9, 2026 19:19
Hidden-flag shadow comparison: granite-speech-4.1-2b via llama-server
sidecar transcribes every post-processed session's diarized segments in
parallel with the primary model for ~a week; per-segment comparison
artifacts + HTML report decide whether the dual-slot accuracy-model
design is justified.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Basis for the granite shadow transcription spec: granite-speech-4.1-2b
recommended; higgs-audio-v3-8b and canary-qwen-2.5b rejected (meeting-audio
profile + no Apple Silicon path); repo scalability review of model N+1 and
the dual-slot split.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mode

Per Nic: benchmark first + short shadow. Adds a WER harness on the
leaderboard's ESB test sets (AMI/Earnings-22 + CORAAL held-out sample)
with a fidelity gate reproducing granite's published numbers; shadow
pass shortened to a few-day domain confirmation. Success criteria split
into Phase 0 gates + Phase 1 confirmation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…g deadline)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…a curl)

Verifies llama-server is present and >= b9045 (granite-speech mtmd
support), then downloads granite-speech-4.1-2b-Q8_0.gguf and
mmproj-model-f16.gguf via resumable curl (URLSession can't reach the
HF CDN on this network). Fixed a set -e footgun in the version-check
pipeline (grep returning 1 on no match silently killed the script
before it could print its own error message).

Verified for real: llama.cpp b9910 installed via brew, both GGUFs
downloaded (1.95 GB + 1.15 GB), llama-server health check returned
{"status":"ok"} ~2s after model load.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ine)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implements configuration struct for granite shadow transcription feature,
enabling job-time configuration reads of server path, model directory, and port
with tilde expansion and UserDefaults overrides.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Updated SKIP_MARKERS to match hf-audio/esb-datasets-test-only's complete
TED-LIUM ignore_segments set, sourced from datasets-test-only.py upstream
loading script. Old set {"ignore_time_segment_in_scoring", "<unk>", ""}
is now a strict subset of the new set which includes all non-speech and
scoring-gap markers: <noise>, <music>, [noise], [laughter], [silence],
[vocalized-noise], <crosstalk>, <affirmative>, <inaudible>, <laugh>.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…adow)

Behavior-preserving: moves the merge/pad/read arithmetic from
SegmentReTranscriber.run() into pure SegmentAudio functions so the
upcoming granite shadow runner reads byte-identical segment audio.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
readSegment now throws on AVAudioFile.read failure instead of swallowing
it into nil, so the caller's per-segment catch surfaces the
"[transcription failed]" placeholder exactly as the pre-refactor code
did. nil is reserved for buffer allocation failure (silent skip, as
before).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Converts arbitrary AVAudioPCMBuffer instances to 16 kHz mono PCM16 WAV bytes.
Tests verify RIFF header structure and stereo-48k to mono-16k resampling.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implements GraniteRequest to build and parse llama-server requests for
Granite speech transcription. Matches the pinned contract in
scripts/asr-bench/granite_request.md exactly: OpenAI-compatible shape with
input_audio content part, prompt "can you transcribe the speech into a
written format?", temperature 0, max_tokens 2048, stream false.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…fety)

Hardens against AVAudioConverter under-draining on large buffers during
sample-rate conversion (Apple QA1317). Implements drain loop to accumulate
output across multiple convert() calls until exhausted, plus output-length
sanity check to catch residual silent truncation. Added convertsMono8kTo16k
test to cover upsampling case. All 116 tests pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Renders *.comparison.json shadow artifacts into a single side-by-side
HTML report: aggregate stats header, word-level diff highlighting via
difflib, sorted by disagreement descending. Extends the base design
with an error badge for segments with graniteError and an INCOMPLETE
marker for sessions where the shadow run didn't finish. Stdlib only
(argparse, difflib, html, json, pathlib) so it runs with plain python3
on Nic's machine.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nable

granite-4.1-2b Q8_0 via llama-server on M2 Max: AMI 5.98 / E22 8.01 /
TED-LIUM 3.12 WER vs Parakeet v3 7.47/10.39/3.29 and Whisper turbo
13.85/10.74/3.68; RTF ~0.05; 0 request errors over 3214 utterances.
Wins the held-out set too, so not an in-domain artifact.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Enabled graniteShadowEnabled on Nic's live Tome install and ran a real
call-capture session end-to-end: silent (volume-0) system-audio tap capture,
granite shadow phase (spawn-per-job llama-server), artifact write, and
sidecar teardown all verified on the installed app rather than the bench
harness. RTF 0.047, matching Phase 0. Records the display-asleep caveat
(ScreenCaptureKit needs an awake display for system-audio capture, unrelated
to shadow transcription) and the orphan-WAV relocation done to keep an
unattended relaunch from blocking on a launch-time recovery modal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
App quit mid-shadow-transcription previously orphaned the llama-server
child process: nothing outside Transcription/ knew a sidecar existed, so
neither quit path (Quit Anyway, nor the 60s Wait-cap expiry) could stop it.

Add SidecarRegistry, a lock-protected process-wide set of live sidecar
pids with an injectable-signaler killAll() (SIGTERM all, poll kill(pid,0)
for up to a 2s grace, SIGKILL stragglers). DefaultProcessLauncher
registers after Process.run(); RealSidecarProcess unregisters in
terminate()/forceKill()/deinit. TomeApp.applicationShouldTerminate calls
killAll() on both the "Quit Anyway" branch and the wait-cap-expiry branch.
start() previously polled /health without ever checking whether something
was already listening on the port (silently adopting a foreign/orphaned
llama-server) or whether the just-launched child had already died (e.g. a
bind failure), spinning through the full readyTimeout before failing.

- Pre-launch probe: if /health already answers 200 before we've launched
  anything, refuse to adopt it (diagLog + .failed) rather than launch on
  top of or talk past a process we don't control.
- Check process.isRunning before each poll iteration; a dead child now
  fails immediately instead of waiting out the full timeout.
- Launch args gain -c 16384 (Q8 KV at 16k context is fine on 64 GB) and
  --no-webui.

SidecarHTTP.post now returns (Data, Int) instead of discarding the HTTP
status. A non-2xx llama-server response was previously indistinguishable
from a 200-with-garbage-body: both surfaced as an opaque ParseError with
no relaunch. transcribe() now treats non-2xx as a connection-class
failure (the existing single-relaunch-then-fail path), while a genuine
2xx-with-unparseable-body still propagates as ParseError untouched.

Ride-along: transcribe()'s relaunch catch now checks Task.isCancelled
before spending the one relaunch budget — a cancelled caller doesn't need
a fresh sidecar respawned on its behalf.

Tests updated for the new two-probe-per-start() health sequencing
(FakeHTTP gains a call-numbered hang so a test can target the post-launch
poll specifically) and the (Data, Int) post() signature.
ShadowArtifacts paired primary/shadow segments by startTime alone.
Overlapping-speaker diarization can produce two merged segments sharing a
startTime for different speakers; uniquingKeysWith silently dropped one
primary text on collision. Key pairing on startTime+speaker on both sides
instead — same-speaker same-startTime can't survive merge(), so this key
is unique.

AudioWAVExport's drain loop counted every iteration's frames into
totalFrames but only ever copied bytes from the LAST iteration's `out`
buffer (reset to empty at the top of each loop pass). Correct only by the
single-iteration assumption; if convert() ever returned output across
multiple .haveData chunks, earlier samples would be silently dropped
while still counted. Append each iteration's produced bytes to a running
Data immediately, before the reset, so the loop is correct regardless of
converter chunking behavior. No behavior change in the common
single-iteration case — existing tests pass unchanged.
granite-shadow-report.py now wraps each *.comparison.json parse in a
try/except: a single malformed file no longer aborts the whole report,
it's skipped with a stderr warning and counted ("N unreadable") in the
report header.

GraniteShadowPhase.shouldRun() now diagLogs the reason when the flag is
on but the phase is skipped anyway (no rebuild / no primary results) —
the spec promised logged skips. Flag-off stays silent, the common case.
(The Task.isCancelled relaunch-skip ride-along landed already, bundled
into the sidecar start()/post() commit since it touches the same
transcribe() catch block.)
…neration

SidecarRegistry.killAll() blocks the main thread during quit, but
GraniteSidecar runs on its own actor executor: an in-flight transcribe()
whose connection dropped BECAUSE killAll was terminating its server
satisfied every relaunch precondition (state .ready, Task not cancelled,
relaunch budget unspent) and spawned + registered a fresh llama-server
after killAll's victim snapshot — reproducing the orphan the registry
exists to prevent.

SidecarRegistry gains a permanent quitting gate sharing the pid set's
lock, flipped inside killAll's snapshot acquisition (single atomic step,
never reset — there is no un-quit) and exposed as isQuitting.
GraniteSidecar checks it at both spawn points via an injectable
isQuitting closure (production default reads the global; tests inject
their own so they're isolated from the permanent process-global flag):
start() refuses to launch, and transcribe()'s catch rethrows without
relaunching, same treatment as cancellation.

Also: start()'s pre-launch-probe 200 branch was missing the generation
re-check every sibling branch has — a stop() interleaving during the
probe suspension would have .failed stomped over its .idle. The probe
result is now acted on only after the generation guard passes.
nicw and others added 4 commits July 10, 2026 01:14
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fix A — VAD load no longer trims the opening of every recording.
TranscriptionEngine constructed a fresh VadManager (~1.7–2.1s on M2 Max)
on every start(), before installing the mic tap, losing the first ~2s of
audio. Now the manager loads once and is reused for the engine's lifetime
(single-flighted via a shared load Task, matching ModelProvisioner's
style), and is warmed at launch via preloadVAD() fired from ContentView's
boot task. Reuse is safe: FluidAudio.VadManager is an actor wrapping only
the immutable CoreML model + config; all per-run streaming state lives
outside it (makeStreamState/processStreamingChunk state param), and the
two live transcribers already share one instance. stop() deliberately
keeps the manager.

Fix B — Whisper model download survives networks where URLSession can't
reach the HF Xet CDN. On some networks URLSession (and urllib) can't
connect at any timeout while curl connects in ~30ms, so WhisperKit.download
always failed and Whisper could never install in-app. New CurlModelFetcher
lists files via the HF tree API and downloads them with /usr/bin/curl
(app is not sandboxed) into the exact HubApi layout. WhisperBackend.prepare
now falls back to it on any SDK download error (throwing both messages if
the fallback also fails), and honors TOME_FORCE_CURL_MODEL_FETCH=1 to
exercise the path on a healthy network.

Tests: 154 pass (26 suites). New CurlModelFetcherTests covers tree-JSON
parsing, resolve/tree URL + dest-path derivation, and progress math; no
curl is spawned.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…p replaces -C resume

Two review fixes to CurlModelFetcher:

1. Cancellation. runCurl used a bare continuation and the download loop
   never checked cancellation, so a cancelled provisioning cycle left the
   curl child downloading gigabytes with the await pending forever. Now
   runCurl is wrapped in withTaskCancellationHandler whose onCancel
   terminates the child (Process.terminate is thread-safe); the terminate
   unblocks the same waitUntilExit the normal path uses, so the
   continuation resumes exactly once, and an isCancelled flag turns that
   resume into CancellationError instead of a curl-exit error. A
   lock-guarded box closes the race where cancellation lands before the
   process is registered. The download loop also checks
   Task.checkCancellation() at the top of each file iteration.

2. 416 on retry-after-interruption. `-C -` with `--fail` makes curl exit
   22 (HTTP 416) on a Range request for an already-complete file, so a
   re-run after interruption failed the whole fetch. fileList now keeps
   each tree entry's size (RemoteFile), and before downloading, a file
   whose on-disk size equals the expected size is skipped (counted toward
   progress); otherwise any partial is removed and the file is downloaded
   fresh WITHOUT -C. This avoids the 416 entirely and gives
   resume-by-skipping at file granularity. The skip decision is a pure
   function (shouldSkipDownload) covered by fixture tests — no curl, no
   network. fetchFiles (tokenizer; no tree listing, tiny files) passes
   nil sizes and always redownloads.

Also drops the redundant self.vadManager double-assign in
TranscriptionEngine.start() — loadVADManager() already assigns it on the
success path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ings-only

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant