transcription: dictation examples, one section per language - #23
Open
andredezzy wants to merge 13 commits into
Open
transcription: dictation examples, one section per language#23andredezzy wants to merge 13 commits into
andredezzy wants to merge 13 commits into
Conversation
andredezzy
changed the title
WhisperKit's downloadBase defaults to ~/Documents/huggingface. With iCloud Desktop & Documents sync on, the model weights are replicated and, once the disk fills, evicted to dataless placeholders (SF_DATALESS, st_blocks=0). CoreML mmap()s the weight blob during ANE compilation, and mmap of an evicted file blocks indefinitely: the daemon prints "loading <model>..." and never reaches "listening on fn hold". A partially materialised read instead fails as CoreML error 3, "Failed to read first word from AudioEncoder.mlmodelc/ coremldata.bin. It is not a valid .mlmodelc file." Observed on macOS 26.4.1 with whisper-large-v3-turbo (1.6 GB) on a 94%-full disk. Point downloadBase at Application Support, which is neither user-visible nor sync-managed. Existing installs keep their Documents copy; note it on load so the space is reclaimable rather than silently abandoned.
Same model as whisper-large-v3-turbo, quantised by argmax to 632 MB instead of 1620 MB. Measured on an 8-utterance mixed Portuguese/English corpus: identical word accuracy (89.4%, the same 7 errors), with only two cosmetic differences across the eight transcriptions. Per-utterance latency is marginally worse (0.61–0.93 s versus 0.56–0.76 s): quantisation buys memory and disk, not compute, since the Neural Engine still unpacks the weights. Worth offering to anyone short on disk, which on a machine whose models were being evicted is not hypothetical.
Adds Parakeet TDT v3 alongside WhisperKit. Measured on the same eight recordings: an order of magnitude faster (0.07-0.11 s against 0.6-2.5 s per utterance) and better on short utterances carrying English technical terms, worse on long sentences. Which side of that trade matters depends on how a person dictates, so it is a choice in the menu rather than a new default. The model is picked from a Model submenu and remembered like the input device. `--model` still wins for the launch that passes it. Switching loads the incoming model before dropping the current one, so a failed download leaves the user dictating with what they had, and the preference is written only after the load succeeds. Whatever is not in use is deleted: 792 MB reclaimed switching away from large-v3-turbo-compressed, plus 11 MB of tokenizers that live outside the weight directory. The same sweep runs at startup, so models stranded by a crash or an older version come back as free space. whisperKitID becomes engineID: the field now holds Parakeet ids too.
Parakeet throws invalidAudioData under 0.25 s where WhisperKit returns an empty string, so a tap on the hotkey logged a failure instead of doing nothing. Measured: 0.1 s throws, 0.3 s returns "". The threshold comes from the engine rather than a number of ours, so it follows whatever the library decides its minimum is.
Deleting everything but the active model made switching cost a download each way — six minutes to try the other engine and six more to come back, which is not a choice anyone makes twice. The model switched away from is now kept, so going back is instant. Measured: Parakeet -> Whisper Base kept both on disk (461 MB + 145 MB, freed 0), and switching back took 6 s with no download. Everything else still goes: a first run with nothing remembered freed 657 MB. The purge moved out of ActiveTranscriber, which knew nothing about history, and into the two places that do — startup and the menu callback.
andredezzy
force-pushed
the
feat/dictation-example
branch
from
August 1, 2026 21:46
a84b1ed to
c7151b5
Compare
Picking a model left the menu unchanged until the load finished, which for a model that is not on disk yet means minutes. The row you clicked kept the old tick, the state line kept the old name, and the only honest reading was that the click had been ignored. The row now takes a dash the moment it is clicked, the other rows go quiet until the switch settles, and the state line carries the download as a percentage. Both engines report the download and neither reports the load, so the line drops the number and says loading for the last stretch rather than sitting at 100%. WhisperKit only reports progress on its static download, so the weights are fetched first and the pipeline is pointed at the folder afterwards.
Whisper can be told what to expect before it decodes, which is the mechanism the industry uses for domain vocabulary. WhisperKit exposes it and it returned nothing: the decoder forces prompt tokens one at a time, and a prediction it makes while doing so — discarded on the next iteration — was still allowed to complete the segment. One `<|endoftext|>` guess mid-prompt ended every prompted transcription before it began. The fix is two lines in the decoder, carried on a fork until upstream takes it. Measured on eight utterances, same model, same audio: no example 85.8% word accuracy 46% technical terms with example 89.5% word accuracy 85% technical terms The file holds an *example* of speech, not a description of the speaker, and that distinction is the whole feature: "Sou desenvolvedor e falo de pull requests" recovers 62% of terms, while "Preciso revisar os pull requests antes do merge" recovers 85%. The model reads it as the sentence that came before, so it wants a sentence, not a biography. The template says so in the file. Read per dictation, inert while empty, ignored by Parakeet, which has no way to accept text at all.
Whisper can be told what to expect before it decodes. WhisperKit exposes it and it returned nothing: the decoder forces prompt tokens one at a time, and a prediction it makes while doing so — discarded on the next iteration — was still allowed to complete the segment. Two lines in the decoder, carried on a fork until upstream takes it. The file holds examples of speech, not a description of the speaker. That distinction is most of the feature: "Sou desenvolvedor e falo de pull requests" recovered 62% of technical terms where "Preciso revisar os pull requests antes do merge" recovered 85%. Sections per language, because one blob is not safe for anyone who speaks more than one. Measured on Portuguese and English audio: example pt accuracy en accuracy none --- 96.4% Portuguese 88.4% 25.0% English --- 92.9% both mixed 17.7% 92.9% by section 92.3% 92.9% A prompt drags the decoder into its own language, so English audio under a Portuguese example comes back translated. Whisper left to choose for itself does the same thing, which is why the language is now detected and pinned rather than inferred mid-decode. Detection costs a second encoder pass and is skipped when the file names a single language — the common case, and nothing to choose between.
Pinning the language broke every English-only model: `detectLangauge` throws unless the model is multilingual, and the throw propagated, so whisper-base.en — the recommended default — failed 100% of transcriptions. The registry already records each model's languages, so there is nothing to detect for those; detection is now the last resort rather than the first move, and a detection failure degrades to automatic instead of ending the transcription. Found by benchmarking every registered model over every recording on this machine rather than the one model in use. Measured across 32 recordings, 251 s of speech: model pt WER pt terms en WER median parakeet-tdt-v3 94.7% 62% 85.7% 0.15 s whisper-large-v3-turbo 91.1% 92% 96.4% 1.69 s turbo-compressed 92.1% 92% 92.9% 1.72 s whisper-small.en -3.7% 54% 78.6% 1.85 s whisper-base.en -0.5% 15% 78.6% 0.74 s
The template invited a paragraph. Measured across eleven recordings on large-v3-turbo-compressed, it should not have: examples accuracy terms median cost none 87.2% 50% 1.34 s — one sentence 90.8% 86% 1.33 s free two 88.1% 86% 1.49 s +150 ms four 89.9% 93% 1.89 s +550 ms eight 89.0% 86% 2.41 s +1070 ms The first sentence carries the whole effect and costs nothing. Four looks best on terms, but that is one occurrence out of fourteen — noise. Eight is worse on both numbers and twice as slow, because every prompt token is a decode step taken before the speech is heard. The template now shows one sentence per language and says why, so nobody pays a second of latency for a paragraph that does not help.
Packing more terminology into the example does nothing, and the reason is worth writing down. Measured on eleven recordings, one line per language: example accuracy terms two technical terms 90.8% 86% four 89.4% 86% eight 89.9% 86% those eight as a bare list 87.2% 50% no example at all 87.2% 50% Term recall does not move between two terms and eight. The list of the same eight words scores exactly what no example scores, to the decimal. So the model is not being given vocabulary, it is being given a sample of how this person builds sentences — which is also why every word-list mechanism tried against this failure returned nothing. The template now carries the plainest sentence that works, and says why enriching it does not.
Handing Whisper a prompt needs argmaxinc/argmax-oss-swift#514: before it, any transcription carrying promptTokens came back empty, because a prediction made while forcing the prompt was allowed to complete the segment. The newest tag, v1.0.0, is older than the fix, so this pins the revision instead. Move to a version requirement once a release carries it.
andredezzy
force-pushed
the
feat/dictation-example
branch
from
August 1, 2026 22:01
c7151b5 to
7728e67
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hand Whisper one example sentence per language, in the words you actually dictate.
Includes #21 as a dependency, and pins argmax-oss-swift past argmaxinc/argmax-oss-swift#514: before that fix, any transcription carrying promptTokens came back empty, because a prediction made while forcing the prompt was allowed to complete the segment. The newest tag, v1.0.0, is older than the fix, so this pins the revision rather than a version.
Measured across eleven recordings, then narrowed by what the measurements said:
Worth 39 points of technical-term recall on my own recordings, from 46% to 85%, at no cost in latency.