Complete the human-grounded promotion evidence loop

README.md

@@ -121,12 +121,25 @@ python3 skills/skill-eval-loop/scripts/skill_eval_loop.py calibrate \
 `calibrate` exits `0` when agreements meet the locked threshold, `1` when the
 runner is valid but the judge disagrees, and `2` when a judgment is invalid.
 
+## Complete a promotion review
+
+A business-ready promotion result requires more than `run --promotion`. Use the
+[promotion evidence workflow](skills/skill-eval-loop/references/promotion-workflow.md)
+to create a blinded packet, collect two independent human label files and the
+custodian's holdout
+attestation, then run `finalize-review`. The final report measures human and
+automated-judge agreement, outcomes across trials, regressions, usage, and
+recorded cost while leaving the promotion decision with the accountable human
+owner.
+
 ## Boundaries
 
 The minimum runner supports Codex, deterministic graders, a provisional
-same-provider rubric judge, blinded pairwise comparison, and human-labeled
-calibration fixtures. It does not provide independent judging, pricing,
-parallel execution, provider discovery, or adapters for other harnesses.
+same-provider rubric judge, blinded pairwise comparison, human-labeled
+calibration fixtures, and hash-bound two-reviewer promotion evidence. It records
+operator-supplied cost; it does not discover pricing, provide an independent
+automated judge, authenticate human identities, run in parallel, discover
+providers, or adapt other harnesses.
 
 Live evaluation is a trusted local-operator workflow. The configured harness
 and Codex executable can read the run-local Codex credentials and therefore

docs/minimum-eval-contract.md

@@ -321,9 +321,21 @@ human labels for the rubric or preference decisions, and measured agreement
 between those labels and any automated judge. A visible development suite must
 not be relabeled as a holdout after it has guided changes.
 
-The remaining real-promotion gate is external to this runner: independently
-control the holdout, obtain human labels, repeat trials, and review the retained
-transcripts before making a promotion claim.
+After a quality-complete promotion run, `prepare-review` creates a hash-bound
+packet containing only the blinded A/B prompts and empty templates. The
+custodian attests task-hash-bound independence, development secrecy, and
+coverage of positive, negative, ambiguous, near-tie, and adversarial cases. Two
+distinct reviewers label every transcript and dimension independently with
+rationale. `finalize-review` verifies those inputs against the retained run,
+measures human and automated-judge agreement, restores condition outcomes,
+reports variance by task, regressions, usage, and operator-recorded cost, and
+retains limitations plus hash-listed copies of the manifest, attestation, and
+complete reviewer inputs. See the packaged
+[promotion workflow](../skills/skill-eval-loop/references/promotion-workflow.md).
+
+The machine cannot authenticate human identity or custody claims. A complete
+review package is evidence for the accountable owner; it is not an automatic
+promotion verdict.
 
 A one-task pilot can establish runner acceptance. It cannot establish that a
 skill is generally effective. Capability suites should contain enough

skills/skill-eval-loop/SKILL.md

@@ -126,8 +126,11 @@ calibration, and repeated trials:
 `--promotion` rejects target-owned tasks, fewer than three trials, and rubric
 runs without accepted calibration. It records the promotion role; it does not
 prove task independence, representativeness, human labeling, or judge
-agreement. Retain that evidence separately and keep the holdout unavailable to
-the hill-climbing agent.
+agreement. Keep the holdout unavailable to the hill-climbing agent, then follow
+[`references/promotion-workflow.md`](references/promotion-workflow.md): run
+`prepare-review`, collect the custodian attestation and two independent label
+files, and run
+`finalize-review` before making a promotion claim.
 
 ## Calibrate the pairwise judge
 
@@ -235,3 +238,10 @@ response before making a quality claim.
 Do not claim broad skill quality from one pilot or from same-provider judging.
 Use realistic unsaturated tasks, repeated trials, deterministic outcomes,
 blinded comparison, human calibration, and human transcript review.
+
+`prepare-review` and `finalize-review` are the final evidence loop. The first
+copies only blinded A/B prompts into a hash-bound packet. The second fails closed
+unless the packet, retained run, holdout attestation, two distinct reviewers,
+every transcript, every dimension, rationales, and recorded cost are complete.
+Its `complete_human_review` status means the evidence is ready for an accountable
+human decision; it is not an automated promotion verdict.

skills/skill-eval-loop/references/promotion-workflow.md

@@ -0,0 +1,106 @@
+# Promotion evidence workflow
+
+This is the complete business workflow for turning a valid paired run into an
+auditable, human-grounded promotion decision package. The evaluator prepares
+and verifies evidence; the accountable human owner makes the decision.
+
+## Roles
+
+- **Holdout custodian:** controls the task file, keeps it out of skill authoring
+  and development runs, and attests that it covers positive, negative,
+  ambiguous, near-tie, and adversarial cases.
+- **Evaluator operator:** calibrates the judge, runs the locked promotion
+  evaluation, prepares the blinded packet, and records actual cost.
+- **Two reviewers:** independently inspect every blinded A/B transcript and
+  label the overall winner plus every rubric dimension with rationale.
+- **Decision owner:** reviews disagreements, regressions, variance, usage, cost,
+  and limitations before accepting or rejecting promotion.
+
+One person may operate the evaluator and own the decision. The two label files
+must still use distinct reviewer identities and be completed independently.
+
+## 1. Lock and run the holdout
+
+The custodian supplies an absolute task path outside the target skill. Run the
+accepted calibration and inspect the dry-run plan before authorizing live calls.
+
+```bash
+"$EVALUATOR" run \
+  --skill /absolute/path/to/target-skill \
+  --tasks /absolute/custodian/path/holdout.jsonl \
+  --output /absolute/path/to/fresh-promotion-run \
+  --harness codex \
+  --harness-bin /absolute/path/to/codex \
+  --model exact-runner-model \
+  --judge-model exact-judge-model \
+  --calibration /absolute/path/to/calibration.json \
+  --trials 3 \
+  --timeout-seconds 300 \
+  --promotion \
+  --dry-run
+```
+
+Run the identical command without `--dry-run` only after the hashes, models,
+invocation count, and cost authority are accepted.
+
+## 2. Prepare a blinded review packet
+
+```bash
+"$EVALUATOR" prepare-review \
+  --run-dir /absolute/path/to/fresh-promotion-run \
+  --output /absolute/path/to/fresh-review-packet
+```
+
+Give reviewers only the review packet, not the retained run. The packet contains
+the exact A/B prompts previously shown to the automated judge, a hash-bound
+manifest, a label template, and a holdout-attestation template. It contains no
+control/treatment mapping.
+
+The custodian completes `holdout-attestation-template.json`. Each reviewer makes
+a private copy of `labels-template.json`, sets a distinct `reviewer_id`, labels
+every overall comparison and dimension with `A`, `B`, or `tie`, provides a
+rationale, and sets `transcript_reviewed` to `true` only after reading the item.
+Reviewers must not inspect each other's labels before both files are final.
+
+## 3. Finalize the evidence
+
+```bash
+"$EVALUATOR" finalize-review \
+  --run-dir /absolute/path/to/fresh-promotion-run \
+  --manifest /absolute/path/to/fresh-review-packet/manifest.json \
+  --holdout-attestation /absolute/custodian/path/holdout-attestation.json \
+  --labels /absolute/reviewer-a/labels.json \
+  --labels /absolute/reviewer-b/labels.json \
+  --cost-usd 12.34 \
+  --cost-note "Provider invoice or subscription allocation." \
+  --output /absolute/path/to/fresh-promotion-review
+```
+
+Finalization fails if the run is not a quality-complete promotion run, hashes
+drift, holdout coverage or custody is not attested, reviewer identities are not
+distinct, a transcript or dimension is unlabeled, a rationale is missing, or
+the recorded cost is invalid.
+
+The output reports:
+
+- human agreement overall and by dimension;
+- automated-judge agreement with each reviewer and human consensus;
+- disagreements with both rationales;
+- unblinded control/treatment/tie outcomes by task and across trials;
+- treatment improvements and control-winning regressions;
+- usage, recorded cost, and explicit limitations.
+
+It also retains hash-listed copies of the manifest, holdout attestation, and
+both complete reviewer files so the final handoff does not depend on scattered
+inputs.
+
+`complete_human_review` means the evidence package is complete. It does not mean
+the skill should be promoted. The decision owner must inspect disagreements and
+regressions and record the business decision separately.
+
+## What the machine cannot prove
+
+The evaluator binds files and enforces completeness. It cannot authenticate a
+reviewer's real-world identity, prove that the custodian kept the holdout secret,
+or decide whether the observed tradeoff is acceptable for the client. Those are
+explicit human-accountability boundaries, not hidden automation claims.