Skip to content

Calibration: run web-dev-html-001 eval against Claude and record first scores #1

Description

@Lingz450

Objective

Run the calibration script against all 5 eval samples and compare AI scores to expected ranges.

Steps

  1. Add ANTHROPIC_API_KEY to backend/.env
  2. Start the server: uvicorn app.main:app --reload --port 8000
  3. Run: python evals/run_calibration.py --eval evals/web-dev-html-001.jsonl
  4. Record which samples PASS and which FAIL

Definition of done

  • All 5 samples return a score within their expected_score_range
  • Results saved to evals/calibration_results/
  • If any sample fails, rubric scoring_guide is adjusted and re-run until pass rate >= 80%

Files involved

  • evals/web-dev-html-001.jsonl — the 5 calibration samples
  • evals/run_calibration.py — the runner
  • rubrics/web-dev-html-001.json — the rubric to tune if needed

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions