Catch evaluation drift before it ships.

EvalCanary shows exactly which fixed evaluation results change when a verifier, scorer, rubric implementation, or grading rule changes.

It treats an evaluator update as a migration:

• replay the same output corpus against both versions;
• classify every verdict transition;
• estimate paired uncertainty;
• inspect subgroup effects;
• attach source and execution provenance;
• enforce explicit CI policy gates.

EvalCanary does not determine automatically which evaluator is correct. It produces the change packet that a domain reviewer needs.

An aggregate benchmark score can remain nearly unchanged while many individual cases reverse in opposite directions. In reinforcement learning with verifiable rewards, an evaluator defect can become a training signal rather than merely a reporting error.

EvalCanary holds the model-output corpus fixed and changes only the evaluator. That isolates evaluator sensitivity from model sampling, prompt changes, and new generations.

Requires Python 3.11 or later; v0.1.1 is tested on Python 3.11–3.14. The runtime has no third-party dependencies.

git clone https://github.com/lmdixon23/evalcanary.git
cd evalcanary
python -m venv .venv
.venv/bin/python scripts/bootstrap_local.py
.venv/bin/evalcanary demo --out evalcanary-demo

Windows PowerShell:

git clone https://github.com/lmdixon23/evalcanary.git
Set-Location evalcanary
py --version
py -m venv .venv
& .\.venv\Scripts\python.exe .\scripts\bootstrap_local.py
& .\.venv\Scripts\evalcanary.cmd demo --out .\evalcanary-demo

Open evalcanary-demo/report/report.html.

The caller must set up Python 3.11 or later. The exact release tag is preferred for reproducibility.

name: Evaluator migration

on:
  pull_request:

permissions:
  contents: read

jobs:
  evaluator-diff:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.11"
      - name: Compare evaluator versions
        id: evalcanary
        uses: lmdixon23/evalcanary@v0.1.1
        with:
          data: outputs.jsonl
          before: verifier_before.py
          after: verifier_after.py
          policy: evalcanary.toml
          slice: |
            metadata.domain
            metadata.language
          output: evalcanary-report
      - name: Upload review packet
        if: always()
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: evalcanary-report
          path: evalcanary-report
          if-no-files-found: error

Action outputs:

• report_directory;
• report_json;
• report_markdown;
• report_html;
• run_id;
• changed_cases;
• policy_passed (true, false, or not-configured).

The action executes both verifier files with the runner account's permissions. It is process isolation, not a security sandbox. Use only trusted verifier code.

Every JSONL object requires a unique id. The same fixed object is passed to both verifiers.

{"id":"math-001","expected":"4","output":"4 ","metadata":{"domain":"math"}}

Each trusted Python verifier defines verify(case) and returns either a boolean or a dictionary containing boolean passed.

def verify(case: dict) -> dict:
    passed = case["output"].strip() == case["expected"]
    return {
        "passed": passed,
        "score": 1.0 if passed else 0.0,
        "reason": "normalized exact match",
    }

Run:

evalcanary diff \
  --data outputs.jsonl \
  --before verifier_before.py \
  --after verifier_after.py \
  --policy evalcanary.toml \
  --slice metadata.domain \
  --out evalcanary-report

Outputs:

• report.json for automation;
• report.md for pull requests and research records;
• report.html for accessible local review.

Source code is not embedded in reports by default. Add --include-source-diff only when the verifier files are safe to disclose to the report audience. Original case payloads likewise require the separate --include-content flag. Absolute and parent paths are reduced to basenames in report provenance and recorded commands; content hashes provide artifact identity.

[policy]
min_cases = 100
max_error_cases = 0
max_abs_score_delta = 0.005
max_pass_to_fail = 10
max_fail_to_pass = 10
max_changed_cases = 15
require_statistical_review_below_p = 0.05

Exit codes:

• 0: comparison completed and the configured policy passed;
• 2: comparison completed and the configured policy failed;
• 3: input, execution, or configuration error.

Version 0.1 supports trusted deterministic Python verifiers and pass/fail migration analysis. It does not yet provide:

• an untrusted-code sandbox;
• repeated LLM-judge sampling;
• semantic perturbation generation;
• hosted storage or dashboards;
• automatic causal attribution;
• a general benchmark runner.

• Project design
• Verifier API
• Policy reference
• Report schema
• Product assurance
• Release procedure
• Roadmap
• Name-clearance record

A changed verdict demonstrates evaluator sensitivity. It does not demonstrate that the previous evaluator was wrong, that the candidate is better, or that a benchmark conclusion is invalid. Reports preserve that distinction explicitly.

See CONTRIBUTING.md for the development and review gates. Report security issues through the private process described in SECURITY.md, not through a public issue.

MIT