Harden evaluator evidence lifecycle and promotion workflow

README.md

@@ -1,9 +1,10 @@
 # skill-eval-loop
 
 `skill-eval-loop` is a self-contained Python 3 Agent Skill that measures
-whether access to one local skill changes task outcomes. It runs the same task
-under a no-skill control and an exact-hash treatment, then retains the raw
-evidence and a comparison report.
+whether explicitly applying one local skill changes task outcomes. The control
+receives the original task. The treatment receives the exact hashed skill's
+`SKILL.md` instructions in its prompt, with the installed payload available for
+referenced files. The runner retains the raw evidence and a comparison report.
 
 ## Install
 
@@ -49,30 +50,59 @@ Run a side-effect-free plan before a live invocation:
 Verify the printed hashes and invocation counts, obtain authorization for the
 live calls, then run the same command without `--dry-run`.
 
-For rubric tasks, also pass `--judge-model` with a different exact model
-identifier. The runner judges each condition only after deterministic gates
-pass. A valid same-provider judgment is `provisional_non_independent`; a
-timeout, failed gate, malformed response, or identity mismatch is `unknown`.
-A missing trace-reported model is unattested, not a quality unknown.
+### Public reference benchmark
+
+The checked-in development benchmark evaluates Vercel's
+`vercel-react-best-practices` skill against the no-skill control:
+
+- repository: `https://github.com/vercel-labs/agent-skills.git`
+- revision: `b8caa260a420a73042e35521de4b5c8baf6446cc`
+- skill path: `skills/react-best-practices`
+- tasks: `tasks/react-best-practices-v1.jsonl`
+- expected evaluator payload SHA-256:
+  `5cbdbd8d9acc6913b8f4e0c7151830e88417872421a5975b86fa4b3eba5c36d3`
+- expected task SHA-256:
+  `621a609cfcdb82756ebe6870a0fad16c6ef12f6186f6c75abb213195b4333c92`
+
+Fetch that exact revision into a controlled local directory and pass the
+absolute skill subpath plus the checked-in task file to `run --dry-run`. Reject
+the plan if the revision or payload hash differs. The public task file is
+development evidence, not a secret client holdout.
 
-The runner invokes Codex sequentially in read-only mode. Odd trials run
-control first; even trials run treatment first. It retains `run.json`, the
+For rubric tasks, also pass `--judge-model` with a different exact model
+identifier and `--calibration /absolute/path/to/calibration.json` from an
+accepted calibrate run. The runner judges each condition only after
+deterministic gates pass. A valid same-provider judgment is
+`provisional_non_independent`; a timeout, failed gate, malformed response, or
+identity mismatch is `unknown`. A missing trace-reported model is unattested,
+not a quality unknown. Omitting `--calibration` is allowed, but a rubric run
+then remains quality-incomplete and cannot exit `0`.
+
+The runner invokes Codex sequentially in read-only mode, emitting invocation
+progress to stderr. Odd trials run control first; even trials run treatment
+first. The evaluator injects the exact `SKILL.md` text itself, so treatment
+exposure does not depend on model-side discovery. Target, judge, and calibration
+invocations share one lifecycle that uses cleaned OS-temporary workspaces outside
+the evaluator repository. It retains `run.json`, the
 planned configuration, tasks, condition responses, traces, stderr, and a
 JSON/Markdown report for every pair.
 
-`runner_valid` means the runner held its declared variables and isolation
-checks. It is not a general quality claim. Read both transcripts before
+`runner_valid` means the runner held its declared variables, isolation checks,
+and treatment activation. It is not a general quality claim. Read both transcripts before
 interpreting `treatment_only`, `both_pass`, `control_only`, or `both_fail`.
 
-JSON and Markdown reports also expose activation (currently unknown),
-calibration (`not_run`), every judged dimension, `quality_status`, and
+JSON and Markdown reports expose evaluator-recorded instruction delivery plus
+optional trace telemetry when Codex also reads the installed skill, rolled-up timing and token usage,
+calibration (`not_run`, or `accepted` plus `fixtures_sha256` when a bound
+calibration is supplied), every judged dimension, `quality_status`, and
 `quality_outcome`. Deterministic-only reports say semantic quality was not
 judged. An overall pairwise winner is not a quality pass when any dimension is
-unknown or disagrees with that winner.
+unknown or favors the opposing condition. A tied dimension is compatible with
+an otherwise coherent winner.
 
-Live exit status is `0` when quality evidence is complete, `1` when the runner
-is valid but quality is unknown or was not judged, and `2` when the runner is
-invalid.
+Live exit status is `0` when quality evidence is complete, which for rubric
+runs requires a bound accepted calibration, `1` when the runner is valid but
+quality is unknown or was not judged, and `2` when the runner is invalid.
 
 Calibrate the pairwise judge against versioned human-labeled
 `known-better`, `known-worse`, and `tie` cases before a live quality pilot:
@@ -98,8 +128,18 @@ same-provider rubric judge, blinded pairwise comparison, and human-labeled
 calibration fixtures. It does not provide independent judging, pricing,
 parallel execution, provider discovery, or adapters for other harnesses.
 
+Live evaluation is a trusted local-operator workflow. The configured harness
+and Codex executable can read the run-local Codex credentials and therefore
+must be trusted. This project does not sandbox hostile executables. Keep raw
+run directories local and inspect them before sharing any evidence.
+
 ## Development
 
+Pull-request and push CI verifies evaluator mechanics with deterministic tests
+and fake harnesses. It makes no live model calls, receives no model credentials,
+and uploads no evaluation evidence. Authorized operators run live evaluations
+locally; humans inspect the retained evidence and own promotion decisions.
+
 Run the Python test suite and package healthcheck:
 
 ```bash

docs/minimum-eval-contract.md

@@ -48,7 +48,7 @@ Each non-empty JSONL line is one task:
 
 Required fields are:
 
-- `id`: a unique, non-empty string;
+- `id`: a unique, non-empty, path-safe string;
 - `prompt`: a non-empty string;
 - `graders`: a non-empty list of supported graders.
 
@@ -92,52 +92,87 @@ This is a suite-bootstrap mechanism, not proof that tasks represent real use.
 Use independently sourced task data, blinded judging, and human calibration for
 skill-quality claims.
 
+## Development versus promotion
+
+Repository pull-request and push CI verifies evaluator mechanics with
+deterministic tests and fake harnesses only. It makes no live model calls,
+receives no model credentials, and publishes no raw evaluation evidence.
+Authorized operators run live development and promotion evaluations locally.
+Humans inspect the retained evidence and own the promotion decision.
+
+The default `run` role is `development`. Development suites may be visible to
+the skill author and optimization loop. They are useful for debugging and
+regression detection, but repeated hill-climbing turns them into training data.
+
+`run --promotion` is a stricter execution guardrail. It requires:
+
+- an explicit `--tasks` path controlled outside the target skill;
+- at least three trials;
+- accepted calibration when the task set contains a rubric.
+
+The retained configuration records `evaluation_role` as `development` or
+`promotion`. The flag cannot prove that a task set was independently authored,
+kept hidden, representative of real use, or labeled by humans. Those remain
+operator evidence requirements. A second model or provider does not replace a
+human-labeled holdout.
+
 ## Paired execution
 
 Every task trial runs twice:
 
-- `control`: the target skill is unavailable;
-- `treatment`: the exact hashed skill payload is available.
+- `control`: the target skill is unavailable and Codex receives the original
+  task prompt;
+- `treatment`: the exact hashed skill payload is available and the evaluator
+  injects its exact `SKILL.md` text before the original task prompt.
 
-Prompt, harness, model, timeout, fixture, and tool posture remain fixed. Runs
-are sequential. Condition order alternates by trial to reduce a fixed-order
-confound. Trials never retry silently.
+The original task, harness, model, timeout, fixture, and tool posture remain
+fixed. The instruction injection is part of the treatment. Runs are sequential.
+Condition order alternates by trial to reduce a fixed-order confound. Trials
+never retry silently. The CLI emits invocation progress to stderr and stops
+before repeating a detected network or transport failure.
 
-Each condition starts in an empty read-only workspace. The minimum runner does
-not seed a repository or fixture tree. Consequently, repository-editing tasks
+Each condition starts in an empty OS-temporary read-only workspace outside the
+evaluator repository, preventing ancestor project instructions from entering
+the trial. The minimum runner does not seed a repository or fixture tree. Consequently, repository-editing tasks
 and claims about executed project tests are not reproducible under this
 contract; use self-contained response tasks until a separately justified
 workspace-fixture capability exists.
 
-The intervention is availability of the exact hashed skill payload, not a
-required execution path. Trace evidence about skill access is diagnostic when
-available. Its absence does not invalidate the paired outcome comparison and
-must not be scored as output quality. Results may claim only that access to the
-skill changed measured outcomes under the retained configuration, not that the
-model definitely read or followed the skill.
+The intervention is evaluator-owned injection of the exact hashed skill's main
+instructions. This guarantees treatment exposure without depending on the
+model to discover or open `SKILL.md`; the installed payload remains available
+for referenced files. Delivery does not prove faithful compliance, so human
+transcript review remains required.
 
 The first Codex implementation is deliberately direct. A shared harness
 abstraction is not justified until a second real harness demonstrates common
 behavior.
 
 ## Codex home isolation
 
-The experiment Codex home is not the user's `~/.codex`. A live run creates
+The experiment Codex home is not the user's `~/.codex`. A live run temporarily creates
 `$output/codex-home` and sets `CODEX_HOME` to that directory for control,
 treatment, and judge. Place it under the output directory, not the OS temp
 directory: some Codex builds refuse a temp-dir home.
 
 If `~/.codex/auth.json` exists, copy only that file into the run-local home.
 Do not copy skills, sessions, or `config.toml`. Copied credentials are
-runtime-only. They are not retained evidence and must not appear in reports.
-Dry-run and fake-harness runs must not require an authenticated host Codex
-home.
+runtime-only and the runner removes the entire run-local home when it exits. The runner
+does not intentionally serialize credentials into evidence. Because the
+configured executable can read the copied file, the local operator must trust
+the harness and inspect raw artifacts before sharing them. Dry-run and
+fake-harness runs must not require an authenticated host Codex home.
 
 The treatment skill remains a workspace payload at `.agents/skills/<name>`.
 Host `CODEX_HOME/skills` is not the intervention and is not consulted. A
 same-name skill in the user's Codex home is not a runner gate once the
 experiment uses a run-local home.
 
+This isolation protects the experiment from ambient Codex configuration; it is
+not a security sandbox for hostile executables. Strong isolation of untrusted
+harnesses requires a separate OS or broker boundary and is outside this
+project.
+
 ## Dry-run accounting
 
 Dry-run validates consumed inputs and prints the complete plan without creating
@@ -186,7 +221,9 @@ Rubric judging begins only after both condition runs satisfy runner isolation
 and execution checks and every deterministic grader passes. A failed gate
 produces quality status `unknown` and makes no judge call.
 
-Each qualifying condition is judged separately in a fresh read-only workspace.
+Each qualifying condition is judged separately in a fresh OS-temporary read-only
+workspace outside the evaluator repository. Target, judge, and calibration
+roles share the same workspace, environment, process, trace, and cleanup lifecycle.
 The prompt presents the task, untrusted candidate response, and locked rubric,
 but no control or treatment label. For every dimension, the judge must return
 concrete response evidence and exactly one declared level. The runner retains
@@ -237,20 +274,24 @@ content is unchanged.
 
 ## Reports and exit status
 
-Pair reports separate runner validity, activation, deterministic comparison,
+Pair reports separate runner validity, evaluator-recorded activation, deterministic comparison,
 per-output rubric status, pairwise status, quality completeness, quality
 outcome, and calibration. `run.json` repeats the rolled-up runner validity and
-quality status. Activation is `unknown` with reason `telemetry_unavailable`
-until a later telemetry source exists. Calibration is `accepted` only when a
+quality status and rolled-up timing/token usage. Activation is `observed` when
+the evaluator injects the hashed treatment instructions. `trace_skill_read`
+separately records whether Codex opened the installed main file; it is telemetry,
+not a validity gate.
+Calibration is `accepted` only when a
 validated binding is supplied. Without `--calibration`, a rubric run records
 `not_run`, quality remains `unknown`, and the runner cannot exit `0`.
 
 `quality_status` is `not_required` when no rubric is present, `unknown` when
 any required judgment is unknown, and `provisional_non_independent` when every
 required judgment succeeded. `quality_outcome` lists every dimension through
 `dimension_results` and is never a restored winner when a pairwise dimension
-disagrees with the overall winner (`inconsistent`) or when quality is unknown
-or not judged. Deterministic-only Markdown reports state that semantic quality
+favors the condition opposing the overall winner (`inconsistent`) or when
+quality is unknown or not judged. A tied dimension is compatible with an
+otherwise coherent winner. Deterministic-only Markdown reports state that semantic quality
 was not judged.
 
 Process exit status distinguishes those cases:
@@ -275,6 +316,15 @@ distribution, repeated trials, fair graders, and human review of transcripts.
 Improvement, regression, and no difference are all legitimate outcomes of a
 valid run.
 
+A promotion claim additionally requires an independently controlled holdout,
+human labels for the rubric or preference decisions, and measured agreement
+between those labels and any automated judge. A visible development suite must
+not be relabeled as a holdout after it has guided changes.
+
+The remaining real-promotion gate is external to this runner: independently
+control the holdout, obtain human labels, repeat trials, and review the retained
+transcripts before making a promotion claim.
+
 A one-task pilot can establish runner acceptance. It cannot establish that a
 skill is generally effective. Capability suites should contain enough
 realistic, unsaturated tasks to reveal meaningful differences; regression