Add concurrent transcription and benchmark aliases to eval#88
Merged
Conversation
Ports the straightforward wins from the tiny-audio eval framework: - Built-in benchmark aliases (its DATASET_REGISTRY): `assembly eval tedlium` now resolves the hub id plus the fiddly subset/split/ audio-column defaults each set needs; explicit flags still win. - `--concurrency N` (its --num-workers): fans transcription out across a thread pool using the transcribe-batch pattern, with the first worker error cancelling the not-yet-started items. Sequential with the per-item spinner stays the default. - DER component breakdown (missed / false alarm / confusion): der.py already computed the components; the summary line and JSON payload (`der_breakdown`) now surface them instead of discarding them. https://claude.ai/code/session_014egNbYWh1Z7pkdaUHHDJ15
alexkroman
enabled auto-merge (squash)
June 12, 2026 00:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds two major features to the
assembly evalcommand:--concurrencyflag that allows multiple items to be transcribed in parallel using a thread pooltedlium,librispeech) for common evaluation datasets, eliminating the need to remember full Hugging Face Hub IDs and their configuration defaultsAdditionally, DER (Diarization Error Rate) scoring now includes a detailed breakdown showing the contribution of each error type (missed detection, false alarm, confusion).
Key Changes
Concurrent transcription: Added
_transcripts()function that handles both sequential (default) and parallel transcription modes. With--concurrency > 1, items are transcribed viaThreadPoolExecutorwhile maintaining dataset order. On error, the pool is shut down withcancel_futures=Trueto avoid wasting API calls on queued items.Benchmark aliases: Introduced
Aliasdataclass andALIASESdict ineval_data.pywith 15 pre-configured benchmarks (librispeech, tedlium, earnings22, spgispeech, ami, gigaspeech, peoples, commonvoice, voxpopuli, switchboard, expresso, loquacious, callhome). Theload()function now resolves aliases before fetching from the Hub, allowing explicit--subset/--split/--audio-columnflags to override alias defaults.DER breakdown: Extended the payload to include
der_breakdownwith normalized scores formissed,false_alarm, andconfusion. The summary output now displays these components alongside the overall DER percentage.Updated help text and examples: Simplified documentation to reference the new aliases instead of full Hub IDs, and added examples showing the
--concurrencyflag.Implementation Details
as_completed()to process futures as they finish, but returns results in original dataset order by re-collecting from the futures list (which preserves submission order).https://claude.ai/code/session_014egNbYWh1Z7pkdaUHHDJ15