[Hackathon] create-batch --run without --wait always exits 1 even when every trigger succeeds

[kshitij-heizen]

Discord Username / User ID

GitHub: @kshitij-heizen

Discord: spade1819

What does this improvement do?

Fixes test create-batch --run (without --wait) always exiting 1 even when every trigger succeeds — which breaks the documented exit-code contract and fails CI pipelines on fully successful dispatches.

The no-wait path returns each result with the trigger response's status, which is queued by design (src/commands/test.ts:2743-2756; the field is documented as "Terminal status if --wait; queued if no --wait" at src/commands/test.ts:1697). But the exit-code logic only recognizes terminal statuses (src/commands/test.ts:2910-2949):

const allPassed = batchRunResults.every(r => r.status === 'passed'); // queued → false
if (allPassed) return;
...
throw new CLIError(
  `Batch run finished: ${...} of ${...} run(s) did not pass.`,
  1,
);

Every no-wait result is queued, so allPassed is always false, no error exit codes exist, and the command falls through to exit 1 with a misleading "N of N run(s) did not pass".

Reproduction

testsprite test create-batch --plans ./plans.jsonl --run --output json
# stdout: results[] with every entry status:"queued", no errors
# stderr: "Batch run finished: 2 of 2 run(s) did not pass."
# exit code: 1  ← CI marks a fully successful dispatch as FAILED

This is the exact shape of the documented example (DOCUMENTATION.md create-batch section uses --run --max-concurrency 4 --output json, no --wait). It's also internally inconsistent: single test run <id> without --wait exits 0 on a successful queued dispatch.

Details / implementation notes

In the no-wait case, "success" means every trigger was dispatched without error — statuses are non-terminal by definition. Minimal fix in the exit-code block:

const allOk = opts.wait
  ? batchRunResults.every(r => r.status === 'passed')
  : batchRunResults.every(r => r.error === undefined);
if (allOk) return; // exit 0

Trigger errors in no-wait mode keep today's aggregation (uniform specific code, else exit 1), and the misleading "did not pass" message becomes "N of N trigger(s) failed" when --wait is absent. The stderr summary is likewise corrected for no-wait mode (it currently prints "0/N passed" for a fully successful dispatch).

Covered by unit tests in src/commands/test.create-batch-run.spec.ts (all-queued → resolves without error; partial trigger failure → still exits non-zero; --wait semantics unchanged). Existing no-wait specs only exercise error scenarios and never assert the success exit code, which is how this slipped through.

Not covered by existing issues: #153/#154/#156/#158 are TimeoutError/RequestTimeoutError stdout/classification asymmetries in run/rerun/wait; merged #130/#128 touched batch deadline sharing and idempotency-key output. Nothing claims the no-wait exit-code contract for create-batch --run.

PR to follow shortly.

Confirmations

• I have searched existing issues and this is not a duplicate.
• I have provided my Discord identity above for reward coordination.

[kshitij-heizen]

/assign

[testsprite-hob]

Assigned to @kshitij-heizen. Thanks for taking this on. (1 more slot available.)

Your open assigned issues (2/3):

• [Hackathon] test failure/artifact get --out <existing-dir> silently deletes unrelated pre-existing files (data loss) #159 — [Hackathon] test failure/artifact get --out silently deletes unrelated pre-existing files (data loss)

[zeshi-du]

Status check: re-verified against current main — the bug is real and still present. runBatchRun's exit-code logic only recognizes passed/timeout outcomes; without --wait every result is queued (no .error field), so allPassed is always false and the command falls through to exit 1 even when every trigger succeeded. That breaks the CI contract for fire-and-forget batch creates. @kshitij-heizen you claimed this on 07-03 — are you still on it? If we don't hear back within a few days we'll open the slot to other contributors (or fix it maintainer-side); it's now also tracked internally.

[kshitij-heizen]

yeah still working here

[zeshi-du]

Hi @kshitij-heizen — following up on the Discord Username / User ID field in this issue's template, which still reads "will add my Discord ID by edit shortly" here and on #159 / #160.

All three of your reports landed as merged fixes (#162, #163, #267), so you're included in the Season 3 CLI Improvement Bonus review. We reach contributors using the Discord username from that template field, and yours is the only one we haven't been able to resolve.

Could you drop your Discord username in a comment here (or edit it into the issue body)? No rush on the format — username or user ID both work.

Thanks for the reports, by the way — the --out stale-file sweep and the create-batch --run exit-code inversion were both genuine contract breaks, and the fixes are in.

[kshitij-heizen]

@zeshi-du added my discord username

[zeshi-du]

Thanks @kshitij-heizen — got it, spade1819 is recorded.

One last step: please join the TestSprite Discord and reach out to Kairui | TestSprite there — they handle the CLI Improvement Bonus coordination from here.

https://discord.com/invite/GXWFjCe4an

[zeshi-du]

@kshitij-heizen — one more note on timing: please reach out to Kairui | TestSprite on Discord by 2026-07-29 23:59 UTC. Slots not claimed by then may be reallocated.

https://discord.com/invite/GXWFjCe4an

[kshitij-heizen]

@zeshi-du thanks for the reminder. I have direct messaged Kairui on discord.