Skip to content

Add concurrent transcription and benchmark aliases to eval#88

Merged
alexkroman merged 2 commits into
mainfrom
claude/admiring-knuth-sjyfxy
Jun 12, 2026
Merged

Add concurrent transcription and benchmark aliases to eval#88
alexkroman merged 2 commits into
mainfrom
claude/admiring-knuth-sjyfxy

Conversation

@alexkroman

Copy link
Copy Markdown
Collaborator

Summary

This PR adds two major features to the assembly eval command:

  1. Concurrent transcription via a new --concurrency flag that allows multiple items to be transcribed in parallel using a thread pool
  2. Benchmark aliases that provide short memorable names (e.g., tedlium, librispeech) for common evaluation datasets, eliminating the need to remember full Hugging Face Hub IDs and their configuration defaults

Additionally, DER (Diarization Error Rate) scoring now includes a detailed breakdown showing the contribution of each error type (missed detection, false alarm, confusion).

Key Changes

  • Concurrent transcription: Added _transcripts() function that handles both sequential (default) and parallel transcription modes. With --concurrency > 1, items are transcribed via ThreadPoolExecutor while maintaining dataset order. On error, the pool is shut down with cancel_futures=True to avoid wasting API calls on queued items.

  • Benchmark aliases: Introduced Alias dataclass and ALIASES dict in eval_data.py with 15 pre-configured benchmarks (librispeech, tedlium, earnings22, spgispeech, ami, gigaspeech, peoples, commonvoice, voxpopuli, switchboard, expresso, loquacious, callhome). The load() function now resolves aliases before fetching from the Hub, allowing explicit --subset/--split/--audio-column flags to override alias defaults.

  • DER breakdown: Extended the payload to include der_breakdown with normalized scores for missed, false_alarm, and confusion. The summary output now displays these components alongside the overall DER percentage.

  • Updated help text and examples: Simplified documentation to reference the new aliases instead of full Hub IDs, and added examples showing the --concurrency flag.

Implementation Details

  • The concurrent path uses as_completed() to process futures as they finish, but returns results in original dataset order by re-collecting from the futures list (which preserves submission order).
  • Aliases are frozen dataclasses to prevent accidental mutation.
  • All new code paths are covered by comprehensive tests including concurrency synchronization tests, error handling, and alias resolution.
  • Snapshot tests updated to reflect new help text and examples.

https://claude.ai/code/session_014egNbYWh1Z7pkdaUHHDJ15

Ports the straightforward wins from the tiny-audio eval framework:

- Built-in benchmark aliases (its DATASET_REGISTRY): `assembly eval
  tedlium` now resolves the hub id plus the fiddly subset/split/
  audio-column defaults each set needs; explicit flags still win.
- `--concurrency N` (its --num-workers): fans transcription out across
  a thread pool using the transcribe-batch pattern, with the first
  worker error cancelling the not-yet-started items. Sequential with
  the per-item spinner stays the default.
- DER component breakdown (missed / false alarm / confusion): der.py
  already computed the components; the summary line and JSON payload
  (`der_breakdown`) now surface them instead of discarding them.

https://claude.ai/code/session_014egNbYWh1Z7pkdaUHHDJ15
@alexkroman
alexkroman enabled auto-merge (squash) June 12, 2026 00:40
@alexkroman
alexkroman merged commit af7d435 into main Jun 12, 2026
8 checks passed
@alexkroman
alexkroman deleted the claude/admiring-knuth-sjyfxy branch June 12, 2026 01:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants