Refactor test modules and improve CLI error handling#91
Merged
Conversation
…tion
- client.py (_sdk_errors): compact httpx 'Request: <...>' reprs out of generic
API error messages, strip the SDK's redundant 'failed to retrieve
transcript(s):' preamble, and attach a network suggestion (parity with the
llm gateway path).
- llm.py: only suggest a paid plan on gateway 401/403 when the response
actually mentions the entitlement (plan/upgrade/billing/no access);
otherwise point at the key/network so a proxy 403 doesn't send users
to billing.
- llm command: validate --transcript-id locally before auth/network (same
check as transcripts get), add min=1 to --max-tokens, warn on stderr when
piped stdin is ignored because --transcript-id takes priority (quiet
suppresses; --json gets a structured {"warning": ...} line), and reject
-o/--output in --list-models mode (it was silently ignored).
- transcripts get: run the cheap local id validation before auth so a
malformed id errors immediately instead of after login.
- sessions list: --status is now a closed choice set (created|completed|
error) and --limit gets min=1, matching transcripts list.
- transcripts subcommands reordered so 'list' precedes 'get' in --help,
matching sessions. (Bare 'assembly transcripts'/'assembly sessions'
already rendered help via no_args_is_help.)
- Regenerated help snapshots; added mutation-killing tests for both branches
of every new conditional.
https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
- AMS session commands (balance/usage/limits/keys/audit/sessions) now carry a
session-specific suggestion: a browser login is the only fix, so the dead-end
"set ASSEMBLYAI_API_KEY" advice is dropped from resolve_session's error.
- resolve_api_key strips env/keyring/flag values and treats whitespace-only
keys as missing, so ASSEMBLYAI_API_KEY=' ' hits the clean exit-4
not-signed-in path instead of leaking a raw httpx illegal-header error.
- assembly login warns "Could not open a browser" when webbrowser.open returns
False (headless Linux), not only when it raises — no more silent 120s wait.
- assembly login (both browser and --api-key paths) preflights
config.keyring_usable() and fails fast (exit 2, keyring_unusable) with
doctor's ASSEMBLYAI_API_KEY guidance before opening any browser.
- assembly whoami still renders the local profile/env/masked-key/session table
when key validation fails on a network error; status reads "unreachable
(network error)" (reachable: null in JSON), distinct from "key rejected",
and exits 1 with a suggestion instead of swallowing the table.
- assembly usage validates --end >= --start and --window in {day, week, month}
client-side (exit 2, before session/network); assembly audit --limit gets
min=1 so zero/negative values are a click usage error.
- assembly logout stays idempotent (exit 0) but reports truthfully: "No stored
credentials for '<profile>' — nothing to clear" on a fresh machine, and the
JSON output now includes "cleared": true/false.
- --profile names are validated at resolution time (AppState.resolve_profile
via the new public config.validate_profile), so a typo'd profile is a fast
exit-2 in the root callback before any network round-trip.
- assembly keys create rejects empty/whitespace --name before auth/network.
- assembly login --json keeps stderr machine-readable: the flow's progress
notes (opening browser / waiting / multi-org pick) ship as {"hint": ..,
"url": ..} objects instead of prose, plumbed via run_login_flow(json_mode=).
Skipped: a `keys disable` subcommand — aai_cli/auth/ams.py has no AMS endpoint
wrapper for disabling tokens (only create/rename), so there is nothing to wire
a CLI command to. `keys` already had no_args_is_help=True; a test now pins it.
https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
…essages
assembly eval:
- Per-row error handling: a failed transcription no longer aborts the run and
discards completed (paid) rows. Failed rows are recorded (JSON "error" field,
human ERROR column), the summary pools only the scored rows, and the command
exits 1 with an eval_failed error. NotAuthenticated still aborts immediately
(exit 4) so a rejected key doesn't bill through the dataset.
- Resolve the API key before downloading the dataset, so a signed-out user
fails fast instead of pulling the whole dataset first.
- --collar without --speaker-labels is now a usage error (it was silently
inert); the 1.0 default is applied internally.
eval_data:
- HF 401/403 errors surface the response body, and the HF_TOKEN hint only
shows when the body reads like gating/auth — a proxy 403 ("Host not in
allowlist") is no longer misattributed to a gated dataset.
- Manifest suffixes outside .csv/.jsonl are rejected with "Manifests must be
.csv or .jsonl" instead of a confusing JSONL parse error.
- Grammar: "an audio column" / "a text column" (article matches the noun).
assembly transcribe:
- --json with a conflicting -o field (e.g. -o text) is now a usage error
instead of -o silently winning over --json.
- A single local source with a non-audio extension (outside AUDIO_EXTENSIONS)
gets a stderr warning (structured under --json, suppressed by --quiet).
- --audio-end now rejects negatives (min=0); --temperature is bounded to the
SDK-documented 0..1 range.
- --out's parent directory is validated (exists + writable) before the billed
transcription runs, exiting 2 up front instead of dying as internal_error
afterwards; the '..' traversal check moved up front with it.
- Rejoined the docstring paragraph that rendered with a hard mid-sentence
line break in --help.
youtube (yt-dlp wrapper):
- Download errors trim yt-dlp's report-a-bug boilerplate ("please report this
issue on ... yt-dlp -U") and the redundant "ERROR:" prefix, keeping only the
meaningful message.
https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
…g hygiene
- speak: every --help example now puts the root --sandbox flag before the
subcommand (the documented form errored with "No such option"), and the
unsupported_environment error spells out the corrected invocation
("Re-run as: assembly --sandbox speak …").
- New aai_cli/streaming/diagnostics.py, shared by stream/agent/speak:
silence_streaming_logging() extends the existing websockets silencing with
the assemblyai.streaming SDK logger, so neither a reader-thread traceback
("unexpected internal error") nor the SDK's own "Connection failed: …"
ERROR line duplicates the CLI's normalized error or pollutes --json stderr.
stream_audio now calls it (agent/tts already silenced websockets via ws.py).
- WS handshake 401/403 now carry an actionable suggestion (whoami / key-env
match / network-proxy access to the env's streaming host) in
client.stream_audio, agent/session, and tts/session; 401 is
NotAuthenticated (exit 4, rejected_key), 403 stays an APIError (exit 1).
- speak: --voice SPEAKER=VOICE mappings on non-speaker-labeled input now warn
("Ignoring --voice SPEAKER=VOICE mappings; input has no speaker labels.")
instead of being dropped silently, mirroring the bare-voice warning; --voice
help names example voices for discovery.
- agent/speak help docstrings rewrapped so Rich renders no widowed words.
- --sample-rate gains min=1 on stream and speak (the API requires a positive
rate); stream's help now says it also declares the rate of raw PCM on stdin.
- agent --list-voices: the English/Multilingual grouping moved from comments
into data (Voice dataclass with a language tag); the human list renders
grouped and the --json array carries {name, language}.
- stream <downloadable URL>: regression test pinning that credentials resolve
before yt-dlp runs (ordering was already correct at HEAD).
https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
…s, telemetry disclosure
- Click parse errors (missing arg, bad option value) now honor --json: when the
invocation opted into JSON, the error formatter emits the uniform
{"error": {"type": "usage_error", ...}} envelope on stderr (exit code still 2)
instead of the human Rich panel.
- Misplaced-flag guidance: unknown --json/-j at root level points at the
per-command placement ("assembly <command> --json") instead of the misleading
"(Possible options: --version)" guess; unknown -q/--quiet/--sandbox/--env/
--profile/-p on a subcommand explains they are global flags that go before
the subcommand.
- --sandbox with a disagreeing --env now exits 2 with a clear conflict error
(human + JSON) instead of silently resolving to --env; the agreeing combo
(--env sandbox000) still works.
- `assembly version` suggests `assembly --version` instead of "Did you mean
'sessions'?"; dropped the stale "the version subcommand stays for parity"
claim from the --version callback docstring.
- First-run telemetry disclosure: when the anonymous device_id is first minted,
one stderr line discloses collection and the opt-outs; suppressed under
--quiet/--json, printed at most once ever, and wrapped so it can never break
a command. Adds the tiny additive config.has_device_id() probe.
- `telemetry status` now reports *why* (source: env:AAI_TELEMETRY_DISABLED /
env:DO_NOT_TRACK / config / default) in both human and JSON output, and the
hint flips to "Re-enable with 'assembly telemetry enable'." when disabled.
- doctor no longer claims "(HTTP 401)" for any rejected-key signal — the
validate_key boundary can't distinguish 401 from 403/proxy blocks.
- Bare `assembly telemetry` shows help instead of "Missing command."
(no_args_is_help; setup already had it).
https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
…ard)
assembly init:
- An existing FILE target is now a clean usage error (exit 2,
"<path> exists and is not a directory."), not an internal-bug traceback.
- --force preserves a configured (non-placeholder) .env key instead of
silently resetting it to the placeholder, prints a stderr notice that
existing files are being overwritten, and the --force help text notes
the overlay (extra files are kept).
- The version banner prints only after validation passes, so pure error
runs stay undecorated like the sibling commands.
- The report's `key` row is emitted symmetrically when a key IS found
("written — from environment/keyring"), including under --json.
- The --no-install sign-off hint (and the no-key launch-skipped row)
carry the chosen --port into the `assembly dev` command.
assembly dev / assembly share:
- A busy --port substituting a neighbor is announced on stderr
("Port 5000 is in use; using 5001."), structured under --json and
suppressed by --quiet (port 0 means "any", so no notice).
- share's tunnel-timeout error now names the cloudflared log file kept
for debugging, and the temp log is deleted on clean exits.
- A failed webbrowser.open (headless box) prints a stderr hint with the
URL instead of silence (covers init's launch path too).
assembly deploy:
- Gains the standard --json/-j flag: errors use the {"error": ...}
envelope and the declined-confirmation path emits
{"status": "aborted", "target": ...}.
assembly onboard:
- The wizard tracks per-section failures: any failed section ends the
run with "Set up with N issue(s) (<sections> failed)." and exit 1
(the auth stop keeps its exit 4) instead of a cheery exit 0.
- The environment section computes its summary from the actual checks:
warnings-only runs say "Ready — 1 warning (only affects
streaming/agent)." instead of "Everything looks good."
- --json now emits a machine-readable section summary on stdout (and
forces the non-interactive prompter so prose can't corrupt it); a
failing run also raises the standard JSON error envelope.
https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
- Split test_init_command/test_login/test_onboard_sections to stay under the 500-line gate (new test_init_force_and_report, test_login_guards, test_onboard_environment modules; behavior unchanged). - generated_code_compile_gate.py: capture stdout only (click CliRunner, mix_stderr=False) and disable telemetry, so stderr chrome like the new first-run telemetry notice can't corrupt the compiled fixtures; declare click as a dev dependency (DEP004-ignored like coverage). - Reword two comment lines that tripped the net-new-Any counter. https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
![aikido-pr-checks[bot]](https://avatars.githubusercontent.com/in/898896?s=60&v=4){kind=small}
alexkroman
enabled auto-merge (squash)
June 12, 2026 01:23
…fix branch Reconciles main's eval --concurrency thread pool with this branch's per-row failure resilience: _transcribe_one returns per-item Transcript-or-CLIError outcomes in both modes; NotAuthenticated and non-CLIError bugs still abort and cancel queued items. DER breakdown folded into _pooled_metrics; doctor keeps the no-status-code wording. Post-merge fallout: eval_data's HF datasets-server client moved to eval_hf_api.py and the eval concurrency/failure tests to test_eval_failures.py (500-line gate); _transcripts split per mode (xenon); init's key-source row no longer misreports a blank ASSEMBLYAI_API_KEY as 'environment' (review finding). https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
…into the QA-fix branch Main's #90 fixed two of the same QA findings differently: adopt its --sandbox/--env conflict warning (--env wins, surfaced as {"warning": …} in JSON mode) over this branch's hard usage error, and keep this branch's fuller speak sandbox-hint suggestion which subsumes main's. https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
Five tests asserted plain substrings on Click usage errors; under GITHUB_ACTIONS Rich forces color on and interleaves style codes mid-message, so they failed only in CI. Assert on the color-free render instead (the existing test_eval_command.py pattern). https://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR refactors test organization to keep modules under the 500-line gate, improves CLI error handling with better validation and suggestions, and enhances the onboarding wizard with JSON mode support and better error reporting.
Key Changes
Test Organization
test_init_command.pyintotest_init_force_and_report.pyfor--forcesemantics and port hintstest_login.pyintotest_login_guards.pyfor keyring preflight and offline whoami teststest_onboard_sections.pyintotest_onboard_environment.pyfor environment section teststest_streaming_diagnostics.pyfor WebSocket hygiene helpersCLI Error Handling & Validation
UsageErrorfor file-exists conflicts (exit 2 instead of internal error);--forcenow warns when overwriting existing files--temperature(0-1 range) and negative--audio-endvalues_parse_day()helper and better error messages_failed_result()for graceful error handling in batch operationsOnboarding Wizard Improvements
json_ctxfixture_capture_stderr()helper_patch_sections()for testingVoice Agent & Streaming
aai_cli/agent/voices.pyto useVoicedataclass with language groupingVOICE_NAMESconstant for validation andENGLISH/MULTILINGUALlanguage groupsaai_cli/streaming/diagnostics.pyfor shared WebSocket error handling and logger silencingHelp Text & Output
--sandboxflag placement hints in examples (must come before subcommand){"error": …}envelopes on stderrConfiguration & Utilities
validate_profile()for profile name validationvalidate_out_path()for early--outvalidation before transcriptionconsent_source()to telemetry for tracking consent decision layerconfig.resolve()Dev Server & Deployment
--porthandling withnotify_port_change()warningAuth Flow
_open_browser()signature to accept**kwargsfor flexibility_note()helper for structured JSON/human progress notes during loginNotable Implementation Details
_capture_stderr(),_patch_sections(), and_login_result()now acceptjson_modeparameter for consistent testinghttps://claude.ai/code/session_01LkB3yQbPDG7Ex4mNL3Wtb6