feat(workflows): tournament soft-scored runner rewrite (V5, #2488)

[flora131]

Summary

Slice V5 of the LLM-as-a-Verifier adoption program (specs/2026-08-17-tournament-soft-selection.md §5.2): the tournament builtin rewritten on the soft-scored pivot-pairing schedule.

• New inputs: n_evaluations (default 2), pivots (default 1), seed (default 0), criteria (V1 shapes, default 3 judge criteria), models (optional ordered list, round-robin per attempt slot, assignment recorded)
• Schedule from plan_comparisons (V4); judge stages score one criterion, both slots (score_a/score_b 1–20, evidence), swapped reps presented reversed with scores recorded in candidate order; prompts via build_scoring_prompt + warm_first_fan_out (V3)
• Parse failure → one re-ask → {invalid:true} excluded; a fully-invalid pair contributes 0.5/0.5 and is flagged
• soft_win/accumulate → select_pivots → pivot rounds → rank_candidates
• comparisons.json replaces bracket.json (per-job orientation/rep/criterion/scores/validity, totals, ranking, seed, budget); outputs keep result/winner/winner_artifact_path/result_path/attempt_artifact_paths/artifact_dir, rename bracket_path → comparisons_path, add ranking + seed
• Rider: coding-agent's bundled slash-command metadata for tournament synced to the new input set (stale user-facing metadata otherwise)

Base: verifier/selection-math (V4, #2512). Stack: V1 → V3 → V4 → V5 → V6.

Size note

828 changed source lines — over the program's 500 cap. The spec pre-authorized exactly this overage as a two-part split (V5a runner core / V5b prompts+ledger+contract), and the commit history is grouped that way (implement soft tournament runner, then add tournament prompts and output contract, then tests/repairs). Kept as one PR to avoid post-hoc history surgery mid-stack; review commit-by-commit if the whole diff is too much at once.

Evidence

Produced by an implement→review→repair goal run (2 turns; completion/evidence/risk reviewers all complete; remaining_work: none):

• npm run check — green (includes the coding-agent tsgo erasableSyntaxOnly pass)
• npx vitest --run --project unit -t "tournament" — green: stubbed judges → complete comparisons.json; invalid exclusion with fully-invalid pair flagged 0.5/0.5; ranking recomputable from the ledger alone; fixed seed twice → identical ledgers; retained output names; swapped-job candidate-order recording
• CHANGELOG: packages/workflows/CHANGELOG.md — ### Breaking Changes (bracket_path rename, schedule change) + ### Added

Spec contract: specs/2026-08-17-tournament-soft-selection.md §5.2 (Q1–Q3 resolved; models round-robin ships here).

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

Greptile Summary

This change expands the tournament workflow with criterion-specific judging, repeated comparisons, ranking output, and persisted comparison records.

A judge response can currently claim a different criterion from the one it was asked to score. Its scores are still accepted into the tournament result, while the judge artifact records the returned criterion and the comparison ledger records the scheduled criterion. This leaves ranking evidence internally inconsistent and skips the existing re-ask behavior.

T-Rex validation blocked

The Greptile artifact-upload tool was unavailable. A focused mocked workflow execution reproduced the inconsistent criterion records and absence of a re-ask, but the executable harness and captured output could not be uploaded as required review evidence.

Confidence Score: 4/5

Not safe to merge until judge reports are required to identify the criterion scheduled for their scoring job.

There is one independent, non-security defect: a mismatched but structurally valid criterion identifier is accepted and creates contradictory persisted scoring records.

Files Needing Attention: packages/workflows/builtin/tournament-runner.ts needs to compare the returned criterion identifier with the scheduled job before treating a judge report as valid.

T-Rex Logs

What T-Rex did

• T-Rex attempted a general contract validation, but the Greptile artifact-upload tool was unavailable, blocking emission of a complete anchored proof.
• A mocked runTournament was executed with a schema-valid response where the returned criterion_id differed from the scheduled correctness job; scores were persisted as valid, the identifier was written into the judge artifact, and the correctness was recorded in the comparison ledger, with no re-ask.
• The required harness and command output could not be uploaded, so this execution cannot be emitted as a complete anchored proof.
• T-Rex produced proof for a posted P1 finding; see the corresponding review comment for finding details.
• A precise blocker summary noted that the available namespaces only include read, grep, glob, todowrite, apply_patch, Exa search, and greptile-review_multibash, with none providing artifact upload, creation, attachment, or URL issuance, and that an upload-capable tool is not exposed, so redispatch remains local-path only.

_{Ran code and verified through T-Rex}

Comments Outside Diff (1)

1. General comment

   Tournament accepts a judge report for the wrong criterion and records contradictory artifacts

   • Bug
     • A schema-valid judge response with a criterion ID different from its scheduled scoring job is accepted, written unchanged to the judge artifact, and scored in a comparison row under the scheduled criterion ID. It does not invoke the invalid-report re-ask.
   • Cause
     • judgeReport validates only that criterion_id is a string, while runPhase writes report but independently records stage.job.criterionId without requiring equality.
   • Fix
     • Reject report.criterion_id !== stage.job.criterionId through the existing invalid-report re-ask path.

   _{Ran code and verified through T-Rex}

Prompt To Fix All With AI

### Issue 1
packages/workflows/builtin/tournament-runner.ts:50-62
**Criterion identity is not validated**

A schema-valid judge response with a `criterion_id` different from the scheduled job is accepted as valid. The saved judge artifact retains the returned ID, while `comparisons.json` records `stage.job.criterionId` and includes the scores in ranking, producing contradictory audit records without invoking the invalid-report re-ask. Reject reports where `report.criterion_id !== stage.job.criterionId` through the existing retry path.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "merge origin/main" | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

[greptile-apps]

Criterion identity is not validated

A schema-valid judge response with a criterion_id different from the scheduled job is accepted as valid. The saved judge artifact retains the returned ID, while comparisons.json records stage.job.criterionId and includes the scores in ranking, producing contradictory audit records without invoking the invalid-report re-ask. Reject reports where report.criterion_id !== stage.job.criterionId through the existing retry path.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/workflows/builtin/tournament-runner.ts
Line: 50-62

Comment:
**Criterion identity is not validated**

A schema-valid judge response with a `criterion_id` different from the scheduled job is accepted as valid. The saved judge artifact retains the returned ID, while `comparisons.json` records `stage.job.criterionId` and includes the scores in ranking, producing contradictory audit records without invoking the invalid-report re-ask. Reject reports where `report.criterion_id !== stage.job.criterionId` through the existing retry path.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.