Add eval scoring stack (jiwer/pyannote) with lazy imports#84
Merged
Conversation
…rade hints The Homebrew formula's resource list predated `assembly eval`, so a brew install was missing jiwer/pyannote (and packaging) — and wer.py's module-scope `import jiwer` sat on every command's import path (main.py -> commands/evaluate.py -> wer.py), so *every* invocation, including --help and doctor, died with ModuleNotFoundError. - wer.py: import jiwer lazily inside the scoring functions (mirrors der.py's lazy pyannote import); new test pins the CLI import chain free of the whole scoring stack (jiwer/rapidfuzz/pyannote/numpy/ scipy/pandas/sklearn) in a fresh interpreter. - Formula/assembly.rb: regenerate resources from uv.lock (19 missing: jiwer, rapidfuzz, the pyannote stack, numpy/scipy/pandas/scikit-learn, packaging, ...) and extend the brew test beyond --version: --help imports the full command tree, plus a direct probe of the lazily imported scoring deps. - update_check.py: the pipx/uv upgrade hints used the console command (assembly), but pipx/uv track the distribution name — "pipx upgrade assembly" fails with "not installed". Now aai-cli. - doctor/login: "rejected (HTTP 401)" now says 401/403 (is_auth_failure treats both as rejected; proxies often answer 403), and the ffmpeg/PortAudio fix hints are no longer Debian-only. - README: signup link in Authentication, Python 3.12+ stated before the install commands (with a --python recovery hint), brew trust marked conditional, ffmpeg/PortAudio decoupled from plain transcribe, a warning that PyPI's assemblyai-cli is not this project, and `assembly stream --sample` as the no-mic streaming starter. https://claude.ai/code/session_01BRpDimSLtwpYAgpHyVtJL1
alexkroman
enabled auto-merge (squash)
June 11, 2026 23:57
alexkroman
added a commit
that referenced
this pull request
Jun 12, 2026
* Replace minor release bump script with patch bump Rename scripts/bump_minor.sh to scripts/bump_patch.sh and change the version increment from X.(Y+1).0 to X.Y.(Z+1), updating the shellcheck list in check.sh and the release docs in AGENTS.md. Also trim the --help sed range so it no longer prints the trailing 'set -eu' line. https://claude.ai/code/session_01VzK2siFmrUzLziSivJqDfV * Add libyaml dependency for pyyaml formula resource The pyyaml resource added in #84 builds a native extension against libyaml, and brew's FormulaAudit/ResourceRequiresDependencies check fails without the matching depends_on line. https://claude.ai/code/session_01VzK2siFmrUzLziSivJqDfV --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for the
assembly evalcommand's WER/DER scoring by including jiwer and pyannote dependencies in the Homebrew formula, while refactoring the scoring modules to use lazy imports. This ensures the CLI remains lightweight for users who don't use the eval command, and allows installations missing the scoring stack to still run all other commands.Key Changes
Formula updates (Formula/assembly.rb)
--helpworks (imports full command tree) andassembly evalscoring stack is availableLazy import refactoring (aai_cli/wer.py)
jiwerimport from module scope to function scope via new_transform()helpernormalize_words()andscore()to call_transform()at runtime instead of using module-level_TRANSFORMImport hygiene test (tests/test_import_hygiene.py) — new file
aai_cli.maindoes not eagerly load jiwer, rapidfuzz, pyannote, numpy, scipy, pandas, or scikit-learnDocumentation & UX improvements
transcribe(no extras needed) vsstream/agent(need ffmpeg/PortAudio)doctorcommand to mention both Debian/Ubuntu and Fedora package managers for ffmpeg and PortAudioaai-cli) instead of console command names for pipx/uvTest updates
test_doctor.pyto verify error messages mention both HTTP 401 and 403 (corporate proxies may return 403)test_login.pyto check for 401/403 in rejection messagetest_update_check.pycomments to clarify pipx/uv upgrade by distribution nameImplementation Details
The lazy import pattern mirrors the existing
der.pymodule's approach with pyannote. By deferringimport jiweruntil_transform()is called (only bynormalize_words()andscore()), the module stays import-cheap for the command layer. This allows the CLI to function fully on installations that lack the eval scoring stack — a critical requirement for Homebrew users who may not have all optional dependencies.The import hygiene test runs in a fresh subprocess to catch any module-scope imports that would have broken the entire CLI on incomplete installs (as happened previously with a module-level
import jiwer).https://claude.ai/code/session_01BRpDimSLtwpYAgpHyVtJL1