fix: catch clips whose audio holds far more speech than the label - #27
Conversation
The rate check only looked one way. It caught a transcript paired with too little audio, and was blind to the reverse, which a VAD that never fires produces constantly. From a real session: 56 of 79 clips are exactly max_segment_s long, because vad.threshold was 0.02 while the room's noise floor ran 0.029 to 0.047, so the gate never closed and every segment was cut by the timeout rather than by silence. Six of those clips are 20 seconds of audio labelled "Thank you.", "No?" or "I love you.", between 0.05 and 0.15 words per second. The old check caught none of them. Training on those is as damaging as the too-fast case and less obvious: a model learns to emit two words for twenty seconds of speech, which is an omission trainer. Both halves look individually fine, which is why nothing downstream caught it. speech_rate_mismatch replaces implausible_speech_rate and reports which direction failed, so the skip breakdown says what is wrong rather than just that something is. The floor is 0.5 words per second, against a measured median of 2.2 and a 25th percentile of 1.2, and only applies above 2 seconds, since a one second "Yes" is legitimately one word per second. Over the real manifest: 79 rows to 67 usable, 6 dropped for the new reason. Documented both the VAD symptom and the drop reason in troubleshooting, with how to read your noise floor off the debug output and pick a threshold above it.
Every clip in a real session ended on the max_segment_s timeout rather than on silence, 56 of 79 at exactly 20.063s, cutting sentences mid-phrase and producing labels like "Thank you." over twenty seconds of audio. The cause is not the threshold. The capture device returns a constant +0.03135 offset, and nothing removed it, so the RMS was the offset. Measured over the 84 clips on disk, the level moved by a factor of 1.18 between silence and speech. No threshold can separate those, and lowering the threshold, which is what the config had done, makes it worse rather than better. Removing the offset first: the same clips separate by 5.52x, silence at 0.0025 RMS and speech at 0.0178, with speech correlated 0.98 against the original. Implemented as a one-pole DC blocker through scipy.signal.lfilter with its state carried across chunks, so the output matches filtering the whole stream and there is no step at a chunk boundary. Subtracting each chunk's own mean would introduce one. The state is primed from the first sample, so a session does not open with a time constant of un-blocked offset. I first wrote the recursion by hand as a cumulative sum weighted by powers of the pole. It is accurate over a chunk and degrades over long inputs, because the weights underflow. lfilter is exact and already available. The debug line now reports the offset when it removes one, because a threshold chosen against the old reading will be far too high once the reading is correct. Documented, with the measurement to make and the value it implies here.
|
Found the actual cause of the VAD failure, and it was not the threshold. The capture device returns a constant +0.03135 DC offset, and nothing removed it. The RMS in the debug log was the offset: So the measured level never changed whether or not anyone was speaking. Across the 84 clips on disk, silence and speech differ by a factor of 1.18. No threshold can separate that, and lowering the threshold, which is what the config had done to catch soft word endings, makes it strictly worse. With the offset removed, the same clips separate by 5.52x: silence at 0.0025 RMS, speech at 0.0178, speech correlated 0.98 against the original. ImplementationOne-pole DC blocker through I first wrote the recursion by hand as a cumulative sum weighted by powers of the pole. It is accurate over one chunk and degrades over long inputs because the weights underflow, which a test that only asserted finiteness would not have caught. The debug line now reports the removal, because a threshold chosen against the old reading is far too high once the reading is correct: ThresholdMeasured after the fix, a working threshold sits between 0.0031 and 0.0090. Roughly: [vad]
threshold = 0.006The current config has 384 tests on 3.13, 362 and 12 skipped on 3.11 and on a dev-only install. |
Whisper-Large-v3-Turbo produced both when re-transcribing a real session. Only the Cloud variant was mapped, so the Claude one passed through.
1 participant
The rate check added in #25 only looked one way. It caught a transcript paired with too little audio and was blind to the reverse, which is exactly what a VAD that never fires produces.
What a real session looks like
56 of 79 clips are exactly
max_segment_slong, becausevad.thresholdwas0.02while the room's noise floor ran0.029to0.047. The gate never closed, so every segment was cut by the timeout rather than by silence.Six of those clips are 20 seconds of audio labelled with almost nothing:
The old check caught zero of them.
Training on these is as damaging as the too-fast case and less obvious. A model learns to emit two words for twenty seconds of speech, which is an omission trainer. Both halves look individually fine, so nothing downstream caught it.
Change
speech_rate_mismatchreplacesimplausible_speech_rateand reports which direction failed, so the skip breakdown says what is wrong rather than only that something is.The floor is 0.5 words per second, against a measured median of 2.2 and a 25th percentile of 1.2 over that session, and it only applies above 2 seconds, since a one second "Yes" is legitimately one word per second.
Against the real manifest
Docs
Two troubleshooting entries. One for the symptom itself, since "every clip is exactly
max_segment_s" is the visible sign that VAD is not working, with how to read your noise floor off the--debugRMS output and pick a threshold above it. One for the drop reason, explaining both directions.Worth stating plainly there:
max_segment_sis a backstop against Whisper's 30 second window, not a segmenter. If it fires every time, VAD is not working.371 tests on 3.13, 349 and 12 skipped on 3.11.
Not fixed here
The VAD threshold itself is configuration, not code, and a recording session is live, so I have not touched
config.toml.