Objective
Run the calibration script against all 5 eval samples and compare AI scores to expected ranges.
Steps
- Add
ANTHROPIC_API_KEY to backend/.env
- Start the server:
uvicorn app.main:app --reload --port 8000
- Run:
python evals/run_calibration.py --eval evals/web-dev-html-001.jsonl
- Record which samples PASS and which FAIL
Definition of done
- All 5 samples return a score within their
expected_score_range
- Results saved to
evals/calibration_results/
- If any sample fails, rubric
scoring_guide is adjusted and re-run until pass rate >= 80%
Files involved
evals/web-dev-html-001.jsonl — the 5 calibration samples
evals/run_calibration.py — the runner
rubrics/web-dev-html-001.json — the rubric to tune if needed
Objective
Run the calibration script against all 5 eval samples and compare AI scores to expected ranges.
Steps
ANTHROPIC_API_KEYtobackend/.envuvicorn app.main:app --reload --port 8000python evals/run_calibration.py --eval evals/web-dev-html-001.jsonlDefinition of done
expected_score_rangeevals/calibration_results/scoring_guideis adjusted and re-run until pass rate >= 80%Files involved
evals/web-dev-html-001.jsonl— the 5 calibration samplesevals/run_calibration.py— the runnerrubrics/web-dev-html-001.json— the rubric to tune if needed