Eval Factory

An evidence-backed improvement lifecycle for AI agent skills. When an agent proposes a change to its own tooling, this harness makes the change earn its way in through a real state machine, hash-verified promotion receipts, and a required human gate, so nothing ships on an agent's say-so.

Why this exists

Agents that edit their own skills, plugins, and instructions are convenient right up until one of them quietly rewrites the thing that decides whether its own work is correct. Once agents can change their tooling, three questions stop being optional:

• What exactly changed, and can I get the old bytes back?
• Did the change actually pass an evaluation, or did something just claim it did?
• Did a human sign off, against evidence, before it went live?

Eval Factory answers those questions with files on disk you can audit, not trust. Every improvement is a record with a lifecycle. Every promotion carries a receipt that re-verifies itself from hashes. The gates fail closed: if a receipt is missing, tampered, or inconsistent, the transition raises and the queue is restored to its prior bytes from a snapshot taken under the queue lock.

What it does

• A real lifecycle state machine. Each improvement moves through captured -> triaged -> validating -> accepted -> promoted, with explicit rejected, superseded, and stale branches. Illegal transitions are refused; if a queue operation raises partway through, the on-disk queue and its generated index are restored from an in-memory snapshot taken under the queue lock (see Limitations for what this rollback does and does not cover).
• Fingerprint deduplication. A normalized fingerprint over (target_type, target, signal_type, dedupe_key) collapses repeat signals into one record and increments an occurrence count, instead of piling up duplicate cards. Concurrent writers are serialized with a file lock.
• Inline-secret scanning. Records are rejected before they land if they contain secret-shaped content (bearer tokens, api_key=/password= pairs, PEM private keys, sk- keys) or raw session/HAR/private-body references that are not marked sanitized. Evidence is stored as references, not payloads.
• Hash-verified promotion receipts. Promoting a change requires a verifiable eval run with a frozen control manifest, before and after SHA-256 hashes of the exact target file, a blind-evaluation gate, and a recorded human ratification (a receipt naming the reviewer and their approval) that is itself SHA-256 hashed into the promotion receipt. The receipt re-verifies from those hashes, so a promoted record that is tampered with after the fact fails validation. These are hash-verified receipts, not cryptographically signed ones: there are no signer keys or identities, only content hashes that must match on re-verification.
• Symlink-escape defenses. Control files, eval artifacts, ratification receipts, and target/baseline paths are all rejected if they are symlinks or resolve outside their expected directory, so a receipt cannot point the verifier at bytes it was not supposed to read.

The lifecycle

stateDiagram-v2
    [*] --> captured
    captured --> triaged
    triaged --> validating
    validating --> accepted
    accepted --> promoted
    promoted --> [*]

    captured --> rejected
    triaged --> rejected
    validating --> rejected
    accepted --> rejected
    triaged --> stale
    validating --> stale
    accepted --> stale
    promoted --> stale
    rejected --> captured

Loading

The happy path runs left to right. accepted requires a reviewer and a verified eval run. promoted requires the full receipt described above. promoted -> stale is a rollback, and it requires proof: the restored bytes on disk must hash to the promoted baseline before the demotion is accepted.

Install

Requires Python 3.11 or newer on a POSIX system (macOS or Linux). The only runtime dependency is PyYAML.

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

Quickstart: the demo walkthrough

The repo ships with a small, fully synthetic queue so the state machine is demoable end to end without any real data. Three invented signals are seeded, and one of them (a my-formatter-skill bug) is walked from captured through triaged to validating.

Look at the seeded queue:

python scripts/skill_foundry.py validate-improvements --root improvements

That prints "ok": true and validates every record, the state-directory placement, the supersession links, and the generated index.json against the canonical YAML records.

Regenerate the demo from scratch (deterministic, byte-identical):

python scripts/seed_demo_queue.py --root improvements --reset

Capture a fresh signal and watch fingerprint dedup collapse a repeat:

# First capture creates a record.
python scripts/skill_foundry.py capture-improvement --root improvements \
  --target-type skill --target my-formatter-skill --signal-type bug \
  --severity P2 --summary "Batch formatting drops the trailing newline on every file" \
  --expected "Each formatted file ends with exactly one trailing newline." \
  --observed "A multi-file batch lost every trailing newline." \
  --source demo:cli --surface claude \
  --evidence "demo://formatter/batch-run-002" \
  --acceptance "A regression case covers a multi-file batch." \
  --dedupe-key my-formatter-skill-trailing-newline

# Running the exact same capture again does not create a second card;
# it increments occurrence_count on the existing record.

Run the evidence harness on the smoke case:

python scripts/skill_foundry.py run-case cases/smoke/

The runner executes the case's validator command, captures stdout and stderr, hashes every artifact, writes a JSONL trace, produces a scorecard, and emits a hold_for_review decision under cases/smoke/runs/<run-id>/. It never auto-promotes: a passing case holds for human review, and a failing case is rejected with the failure tags recorded.

How promotion is gated

transition-improvement ... --to promoted will only succeed when every one of these checks passes, and it verifies them itself rather than trusting the caller:

1. The eval run completed, all validators passed, the scorecard passed with no hard-fail hits, and the decision was hold_for_review requiring a human.
2. The frozen control manifest (evaluator, thresholds, discovery, holdout) still hashes to what the run recorded, and no control file changed after evaluation.
3. The target file on disk hashes to after_sha256, the baseline snapshot hashes to before_sha256, and the two differ.
4. A human-ratification receipt exists, names the same reviewer, run, target, and candidate hash, approves the change, and references a blind evaluation whose promotion_gate_verdict is pass. The ratification bytes are hashed into the promotion receipt.

The adversarial test suite drives each of these failure modes: fabricated eval runs, incomplete artifact sets, failed or inconsistent receipts, equal before/after hashes, tampered ratifications, hash-valid blind failures, and post-promotion tampering. In every case the transition is refused and the queue is preserved.

Repo map

• src/eval_factory/improvements.py - lifecycle, validation, dedup, receipts
• src/eval_factory/runner.py - deterministic case runner and artifact capture
• src/eval_factory/io.py - atomic writes and SHA-256 hashing
• src/eval_factory/cli.py - command-line interface
• scripts/skill_foundry.py - CLI entry point without an install
• scripts/seed_demo_queue.py - regenerates the synthetic demo queue
• schemas/ - JSON Schemas for the record, case, and evidence contracts
• cases/smoke/ - a minimal executable case
• improvements/ - the seeded synthetic demo queue and its generated index
• docs/ARCHITECTURE.md - the design and the deliberate V0 boundary
• tests/ - runner, contract, and adversarial promotion tests

Tests

pytest

pytest runs 79 tests covering the runner, the record contract, concurrent capture, snapshot rollback on raised errors, and the adversarial promotion gates. They run with no network and no services.

Limitations

• POSIX only. The queue uses fcntl advisory locking, so it runs on macOS and Linux, not Windows.
• Single host. The queue is a directory of files guarded by one lock. It is built for one machine, not a distributed writer set.
• Rollback covers raised errors, not crashes. The queue lock and in-memory snapshot undo a partially applied write when an operation raises an exception. They do not provide crash or power-loss atomicity: a process killed with SIGKILL, or a host that loses power mid-write, can leave the queue in a partial state because the restore handler never runs.
• V0 scope by design. There is no candidate generation, no automated blind evaluator, no scheduler, and no auto-promotion. The blind-evaluation gate verifies receipts you supply; it does not run the blind model for you. A human reviewer is always required to promote.
• The runner executes case commands. Cases are trusted inputs. Run only cases you wrote or have reviewed.

License

MIT. See LICENSE.