Recordings lost their first ~2 seconds; Whisper couldn't install when URLSession can't reach Hugging Face - #13
Open
nicw wants to merge 34 commits into
Open
Recordings lost their first ~2 seconds; Whisper couldn't install when URLSession can't reach Hugging Face#13nicw wants to merge 34 commits into
nicw wants to merge 34 commits into
Conversation
Hidden-flag shadow comparison: granite-speech-4.1-2b via llama-server sidecar transcribes every post-processed session's diarized segments in parallel with the primary model for ~a week; per-segment comparison artifacts + HTML report decide whether the dual-slot accuracy-model design is justified. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Basis for the granite shadow transcription spec: granite-speech-4.1-2b recommended; higgs-audio-v3-8b and canary-qwen-2.5b rejected (meeting-audio profile + no Apple Silicon path); repo scalability review of model N+1 and the dual-slot split. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mode Per Nic: benchmark first + short shadow. Adds a WER harness on the leaderboard's ESB test sets (AMI/Earnings-22 + CORAAL held-out sample) with a fidelity gate reproducing granite's published numbers; shadow pass shortened to a few-day domain confirmation. Success criteria split into Phase 0 gates + Phase 1 confirmation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…g deadline) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…a curl)
Verifies llama-server is present and >= b9045 (granite-speech mtmd
support), then downloads granite-speech-4.1-2b-Q8_0.gguf and
mmproj-model-f16.gguf via resumable curl (URLSession can't reach the
HF CDN on this network). Fixed a set -e footgun in the version-check
pipeline (grep returning 1 on no match silently killed the script
before it could print its own error message).
Verified for real: llama.cpp b9910 installed via brew, both GGUFs
downloaded (1.95 GB + 1.15 GB), llama-server health check returned
{"status":"ok"} ~2s after model load.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ine) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implements configuration struct for granite shadow transcription feature, enabling job-time configuration reads of server path, model directory, and port with tilde expansion and UserDefaults overrides. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Updated SKIP_MARKERS to match hf-audio/esb-datasets-test-only's complete
TED-LIUM ignore_segments set, sourced from datasets-test-only.py upstream
loading script. Old set {"ignore_time_segment_in_scoring", "<unk>", ""}
is now a strict subset of the new set which includes all non-speech and
scoring-gap markers: <noise>, <music>, [noise], [laughter], [silence],
[vocalized-noise], <crosstalk>, <affirmative>, <inaudible>, <laugh>.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…adow) Behavior-preserving: moves the merge/pad/read arithmetic from SegmentReTranscriber.run() into pure SegmentAudio functions so the upcoming granite shadow runner reads byte-identical segment audio. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
readSegment now throws on AVAudioFile.read failure instead of swallowing it into nil, so the caller's per-segment catch surfaces the "[transcription failed]" placeholder exactly as the pre-refactor code did. nil is reserved for buffer allocation failure (silent skip, as before). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Converts arbitrary AVAudioPCMBuffer instances to 16 kHz mono PCM16 WAV bytes. Tests verify RIFF header structure and stereo-48k to mono-16k resampling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implements GraniteRequest to build and parse llama-server requests for Granite speech transcription. Matches the pinned contract in scripts/asr-bench/granite_request.md exactly: OpenAI-compatible shape with input_audio content part, prompt "can you transcribe the speech into a written format?", temperature 0, max_tokens 2048, stream false. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…fety) Hardens against AVAudioConverter under-draining on large buffers during sample-rate conversion (Apple QA1317). Implements drain loop to accumulate output across multiple convert() calls until exhausted, plus output-length sanity check to catch residual silent truncation. Added convertsMono8kTo16k test to cover upsampling case. All 116 tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Renders *.comparison.json shadow artifacts into a single side-by-side HTML report: aggregate stats header, word-level diff highlighting via difflib, sorted by disagreement descending. Extends the base design with an error badge for segments with graniteError and an INCOMPLETE marker for sessions where the shadow run didn't finish. Stdlib only (argparse, difflib, html, json, pathlib) so it runs with plain python3 on Nic's machine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nable granite-4.1-2b Q8_0 via llama-server on M2 Max: AMI 5.98 / E22 8.01 / TED-LIUM 3.12 WER vs Parakeet v3 7.47/10.39/3.29 and Whisper turbo 13.85/10.74/3.68; RTF ~0.05; 0 request errors over 3214 utterances. Wins the held-out set too, so not an in-domain artifact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Enabled graniteShadowEnabled on Nic's live Tome install and ran a real call-capture session end-to-end: silent (volume-0) system-audio tap capture, granite shadow phase (spawn-per-job llama-server), artifact write, and sidecar teardown all verified on the installed app rather than the bench harness. RTF 0.047, matching Phase 0. Records the display-asleep caveat (ScreenCaptureKit needs an awake display for system-audio capture, unrelated to shadow transcription) and the orphan-WAV relocation done to keep an unattended relaunch from blocking on a launch-time recovery modal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
App quit mid-shadow-transcription previously orphaned the llama-server child process: nothing outside Transcription/ knew a sidecar existed, so neither quit path (Quit Anyway, nor the 60s Wait-cap expiry) could stop it. Add SidecarRegistry, a lock-protected process-wide set of live sidecar pids with an injectable-signaler killAll() (SIGTERM all, poll kill(pid,0) for up to a 2s grace, SIGKILL stragglers). DefaultProcessLauncher registers after Process.run(); RealSidecarProcess unregisters in terminate()/forceKill()/deinit. TomeApp.applicationShouldTerminate calls killAll() on both the "Quit Anyway" branch and the wait-cap-expiry branch.
start() previously polled /health without ever checking whether something was already listening on the port (silently adopting a foreign/orphaned llama-server) or whether the just-launched child had already died (e.g. a bind failure), spinning through the full readyTimeout before failing. - Pre-launch probe: if /health already answers 200 before we've launched anything, refuse to adopt it (diagLog + .failed) rather than launch on top of or talk past a process we don't control. - Check process.isRunning before each poll iteration; a dead child now fails immediately instead of waiting out the full timeout. - Launch args gain -c 16384 (Q8 KV at 16k context is fine on 64 GB) and --no-webui. SidecarHTTP.post now returns (Data, Int) instead of discarding the HTTP status. A non-2xx llama-server response was previously indistinguishable from a 200-with-garbage-body: both surfaced as an opaque ParseError with no relaunch. transcribe() now treats non-2xx as a connection-class failure (the existing single-relaunch-then-fail path), while a genuine 2xx-with-unparseable-body still propagates as ParseError untouched. Ride-along: transcribe()'s relaunch catch now checks Task.isCancelled before spending the one relaunch budget — a cancelled caller doesn't need a fresh sidecar respawned on its behalf. Tests updated for the new two-probe-per-start() health sequencing (FakeHTTP gains a call-numbered hang so a test can target the post-launch poll specifically) and the (Data, Int) post() signature.
ShadowArtifacts paired primary/shadow segments by startTime alone. Overlapping-speaker diarization can produce two merged segments sharing a startTime for different speakers; uniquingKeysWith silently dropped one primary text on collision. Key pairing on startTime+speaker on both sides instead — same-speaker same-startTime can't survive merge(), so this key is unique. AudioWAVExport's drain loop counted every iteration's frames into totalFrames but only ever copied bytes from the LAST iteration's `out` buffer (reset to empty at the top of each loop pass). Correct only by the single-iteration assumption; if convert() ever returned output across multiple .haveData chunks, earlier samples would be silently dropped while still counted. Append each iteration's produced bytes to a running Data immediately, before the reset, so the loop is correct regardless of converter chunking behavior. No behavior change in the common single-iteration case — existing tests pass unchanged.
granite-shadow-report.py now wraps each *.comparison.json parse in a
try/except: a single malformed file no longer aborts the whole report,
it's skipped with a stderr warning and counted ("N unreadable") in the
report header.
GraniteShadowPhase.shouldRun() now diagLogs the reason when the flag is
on but the phase is skipped anyway (no rebuild / no primary results) —
the spec promised logged skips. Flag-off stays silent, the common case.
(The Task.isCancelled relaunch-skip ride-along landed already, bundled
into the sidecar start()/post() commit since it touches the same
transcribe() catch block.)
…neration SidecarRegistry.killAll() blocks the main thread during quit, but GraniteSidecar runs on its own actor executor: an in-flight transcribe() whose connection dropped BECAUSE killAll was terminating its server satisfied every relaunch precondition (state .ready, Task not cancelled, relaunch budget unspent) and spawned + registered a fresh llama-server after killAll's victim snapshot — reproducing the orphan the registry exists to prevent. SidecarRegistry gains a permanent quitting gate sharing the pid set's lock, flipped inside killAll's snapshot acquisition (single atomic step, never reset — there is no un-quit) and exposed as isQuitting. GraniteSidecar checks it at both spawn points via an injectable isQuitting closure (production default reads the global; tests inject their own so they're isolated from the permanent process-global flag): start() refuses to launch, and transcribe()'s catch rethrows without relaunching, same treatment as cancellation. Also: start()'s pre-launch-probe 200 branch was missing the generation re-check every sibling branch has — a stop() interleaving during the probe suspension would have .failed stomped over its .idle. The probe result is now acted on only after the generation guard passes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fix A — VAD load no longer trims the opening of every recording. TranscriptionEngine constructed a fresh VadManager (~1.7–2.1s on M2 Max) on every start(), before installing the mic tap, losing the first ~2s of audio. Now the manager loads once and is reused for the engine's lifetime (single-flighted via a shared load Task, matching ModelProvisioner's style), and is warmed at launch via preloadVAD() fired from ContentView's boot task. Reuse is safe: FluidAudio.VadManager is an actor wrapping only the immutable CoreML model + config; all per-run streaming state lives outside it (makeStreamState/processStreamingChunk state param), and the two live transcribers already share one instance. stop() deliberately keeps the manager. Fix B — Whisper model download survives networks where URLSession can't reach the HF Xet CDN. On some networks URLSession (and urllib) can't connect at any timeout while curl connects in ~30ms, so WhisperKit.download always failed and Whisper could never install in-app. New CurlModelFetcher lists files via the HF tree API and downloads them with /usr/bin/curl (app is not sandboxed) into the exact HubApi layout. WhisperBackend.prepare now falls back to it on any SDK download error (throwing both messages if the fallback also fails), and honors TOME_FORCE_CURL_MODEL_FETCH=1 to exercise the path on a healthy network. Tests: 154 pass (26 suites). New CurlModelFetcherTests covers tree-JSON parsing, resolve/tree URL + dest-path derivation, and progress math; no curl is spawned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…p replaces -C resume Two review fixes to CurlModelFetcher: 1. Cancellation. runCurl used a bare continuation and the download loop never checked cancellation, so a cancelled provisioning cycle left the curl child downloading gigabytes with the await pending forever. Now runCurl is wrapped in withTaskCancellationHandler whose onCancel terminates the child (Process.terminate is thread-safe); the terminate unblocks the same waitUntilExit the normal path uses, so the continuation resumes exactly once, and an isCancelled flag turns that resume into CancellationError instead of a curl-exit error. A lock-guarded box closes the race where cancellation lands before the process is registered. The download loop also checks Task.checkCancellation() at the top of each file iteration. 2. 416 on retry-after-interruption. `-C -` with `--fail` makes curl exit 22 (HTTP 416) on a Range request for an already-complete file, so a re-run after interruption failed the whole fetch. fileList now keeps each tree entry's size (RemoteFile), and before downloading, a file whose on-disk size equals the expected size is skipped (counted toward progress); otherwise any partial is removed and the file is downloaded fresh WITHOUT -C. This avoids the 416 entirely and gives resume-by-skipping at file granularity. The skip decision is a pure function (shouldSkipDownload) covered by fixture tests — no curl, no network. fetchFiles (tokenizer; no tree listing, tiny files) passes nil sizes and always redownloads. Also drops the redundant self.vadManager double-assign in TranscriptionEngine.start() — loadVADManager() already assigns it on the success path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ings-only Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two small fixes, one branch (based on granite-shadow-transcription, so it carries that line's history — the fixes themselves are the last 3 commits, +~600/−20 across 4 files).
Problem 1 — every recording trimmed its opening words
The VAD model loaded from scratch inside
start()on every session (~2s on an M2 Max), and the mic tap only exists after — so the first couple of words of every memo and call were simply never captured. Nic noticed it as "initial audio is trimmed."Fix: load the VAD model once and reuse it — a single-flight load kicked at app launch (
[ENGINE-VAD-PRELOAD]in the log), withstart()falling back to loading it on demand if the preload hasn't finished or failed. The mic tap now goes up immediately on record.Problem 2 — some networks make Whisper uninstallable in-app
On Nic's network (documented in the benchmark-results doc), Apple's URLSession cannot connect to the HF Xet CDN at any timeout while curl connects in ~30ms — so WhisperKit's downloader always fails and the model picker's auto-revert kicks in every time. Graceful, but Whisper could never actually be installed from the app.
Fix: when the SDK download throws, a curl-based fetcher (
CurlModelFetcher,/usr/bin/curlviaProcess) takes over: lists the variant via the HF tree API, downloads into the exact HubApi layout WhisperKit loads from, tokenizer included.TOME_FORCE_CURL_MODEL_FETCH=1forces the path for testing on a healthy network.Live-verified end-to-end: forced-fallback run fetched the full 1.5 GB variant in 51 seconds and the model loaded and served (
lastGoodflipped to whisper). VAD preload confirmed firing in the launch log.Reviewer's notes (Claude)
withTaskCancellationHandler+ per-fileTask.checkCancellation— no zombie multi-GB downloads. Continuation resumes exactly once (no terminationHandler; cancel'sterminate()unblocks the samewaitUntilExit, anisCancelledflag converts that resume toCancellationError).-C -was deliberately dropped — with--failit 416s on already-complete files and kills re-runs. Replaced by size-check skip against the tree listing's byte sizes (file-granularity resume, no 416 class at all). Tokenizer files have no listed size and always redownload — intentional, they're tiny.VadManagerwraps an immutable CoreML model; per-run streaming state is external (makeStreamState), and two live transcribers already share one instance today.stop()never nil'd the manager.Process— spaces in "Application Support" are a non-issue.-sSstderr stays tiny; documented).🤖 Generated with Claude Code