diff --git a/CITATION.cff b/CITATION.cff new file mode 100644 index 0000000..ae6d69b --- /dev/null +++ b/CITATION.cff @@ -0,0 +1,28 @@ +cff-version: 1.2.0 +message: "If you use GRADE in your research, please cite it as below." +type: software +title: "GRADE: Grounded Reasoning & Analysis for Data in Education" +abstract: > + GRADE is an open benchmark that measures how accurately and usefully AI + systems analyze education program data. It evaluates models across five + tracks — grounded retrieval, trend interpretation, operational coaching, + equity and subgroup analysis, and research reasoning — using synthetic + fixture packs with deterministic ground truth. Scoring combines + deterministic fact-checking with cross-family LLM judging to eliminate + house-style bias. +authors: + - name: "Pearl" + website: "https://tutorwithpearl.com" +repository-code: "https://github.com/PearlEng/grade" +url: "https://tutorwithpearl.com" +license: Apache-2.0 +version: "0.1.0" +date-released: "2026-06-12" +keywords: + - benchmark + - education + - llm-evaluation + - ai-evaluation + - nlp + - grounding + - education-analytics diff --git a/README.md b/README.md index 763380f..9d36a82 100644 --- a/README.md +++ b/README.md @@ -20,25 +20,82 @@ General-purpose benchmarks don't measure this. A model can excel at broad reason ## Leaderboard -Results on the operations pack (11 tasks, 5 runs per task). Full 26-task results in progress. - -| Rank | Model | Composite | Grounding | Insight | Evidence | Calibration | Consistency | -|------|-------|:---------:|:---------:|:-------:|:--------:|:-----------:|:-----------:| -| 1 | `openai/gpt-5.5@high` | **68.3%** | 98.0% | 35.2% | 85.8% | 19.6% | 64.5% | -| 2 | `openai/gpt-5.5@medium` | **64.4%** | 87.5% | 41.7% | 75.3% | 18.0% | 66.6% | -| 3 | `openai/gpt-5.5@low` | **52.4%** | 62.4% | 35.7% | 59.3% | 23.2% | 64.2% | -| 4 | `anthropic/claude-opus-4.8` | **41.9%** | 45.5% | 30.1% | 31.2% | 27.3% | 75.5% | -| 5 | `anthropic/claude-haiku-4.5` | **37.9%** | 47.9% | 22.0% | 20.6% | 20.5% | 70.7% | -| 6 | `anthropic/claude-sonnet-4.6` | **37.5%** | 43.3% | 27.8% | 23.9% | 17.5% | 74.5% | -| 7 | `openai/gpt-oss-120b` | **25.2%** | 22.2% | 7.7% | 18.7% | 25.6% | 74.2% | -| 8 | `google/gemini-3.1-pro-preview` | **23.9%** | 23.9% | 7.9% | 17.6% | 27.4% | 60.3% | -| 9 | `nvidia/nemotron-3-ultra-550b-a55b` | **21.7%** | 22.4% | 5.4% | 19.6% | 13.4% | 70.6% | -| 10 | `google/gemini-3.5-flash` | **17.1%** | 11.1% | 6.2% | 8.9% | 27.8% | 60.3% | - -> Composite = 35% Grounding + 20% Insight + 15% Evidence + 15% Calibration + 10% Consistency + 5% Structure. -> Scores are computed on ground-truth-validated synthetic data. See [docs/methodology.md](docs/methodology.md) for caveats on partial runs and cross-family judging. - -**Key finding:** The hardest dimensions across every model are calibration (acknowledging data limitations and avoiding unsupported causal claims) and insight quality (interpreting signals rather than reciting numbers). Grounding accuracy varies most widely — the gap between GPT-5.5@high (98%) and Gemini 3.5 Flash (11%) on the same tasks reflects a real difference in whether models correctly read numbers from structured CSVs. +Official run, June 10–12 2026. Three packs: **operations** (11 tasks × 5 runs), +**outcomes** (5 tasks × 1 run), **equity research** (10 tasks × 1 run). All rows +ran with a 65,536-token completion budget so reasoning never crowds out the +answer; judging is cross-family (no judge scores its own model family). Full +caveats in [docs/methodology.md](docs/methodology.md); run notes in +[RUN_FINDINGS.md](RUN_FINDINGS.md). + +### Operations (program-data analysis, 5 repetitions) + +| Rank | Model | Composite | Grounding | Insight | Evidence | Calibration | Consistency | Coverage | +|------|-------|:---------:|:---------:|:-------:|:--------:|:-----------:|:-----------:|:--------:| +| 1 | `openai/gpt-5.5@xhigh` | **66.9%** | 91.1% | 45.0% | 77.5% | 22.4% | 63.0% | full | +| 2 | `openai/gpt-5.5@high` | **65.4%** | 87.7% | 46.3% | 76.9% | 18.5% | 62.9% | full | +| 3 | `openai/gpt-5.5@medium` | **64.9%** | 88.6% | 44.2% | 73.4% | 17.8% | 66.2% | full | +| 4 | `google/gemini-3.1-pro-preview` | **63.5%** | 79.2% | 45.0% | 76.9% | 31.2% | 61.3% | 8/11 [1] | +| 5 | `google/gemini-3.5-flash` | **61.0%** | 79.5% | 44.7% | 69.4% | 23.6% | 61.2% | 10/11 [1] | +| 6 | `openai/gpt-5.5@low` | **52.4%** | 62.4% | 35.7% | 59.3% | 23.2% | 64.2% | full | +| 7 | `anthropic/claude-opus-4.8` | **41.9%** | 45.5% | 30.1% | 31.2% | 27.3% | 75.5% | full | +| 8 | `anthropic/claude-sonnet-4.6` | **38.3%** | 42.6% | 29.2% | 27.7% | 18.3% | 76.7% | full | +| 9 | `anthropic/claude-haiku-4.5` | **37.9%** | 47.9% | 22.0% | 20.6% | 20.5% | 70.7% | full | +| 10 | `openai/gpt-oss-120b` | **25.2%** | 22.2% | 7.7% | 18.7% | 25.6% | 74.2% | 3/11 DNF [2] | +| 11 | `nvidia/nemotron-3-ultra-550b-a55b` | **25.1%** | 30.2% | 7.3% | 26.9% | 8.3% | 67.6% | full | + +> [1] Remaining tasks in flight at publication; composite covers listed tasks only. +> [2] `gpt-oss-120b`'s 131k context cannot fit the ~190k-token data payloads on 8 of 11 tasks — reported as a finding, not ranked competitively. + +### Outcomes (trend analysis, single run) + +| Rank | Model | Composite | Grounding | Calibration | Model Cost | +|------|-------|:---------:|:---------:|:-----------:|:----------:| +| 1 | `openai/gpt-5.5@low` | **90.1%** | 100.0% | 59.3% | $0.17 | +| 2 | `openai/gpt-5.5@medium` | **90.0%** | 100.0% | 55.7% | $0.21 | +| 3 | `nvidia/nemotron-3-ultra-550b-a55b` | **88.3%** | 100.0% | 45.8% | $0.04 | +| 4 | `openai/gpt-5.5@xhigh` | **86.9%** | 100.0% | 45.0% | $0.67 | +| 5 | `openai/gpt-5.5@high` | **86.6%** | 100.0% | 39.2% | $0.51 | +| 6 | `anthropic/claude-opus-4.8` | **86.2%** | 100.0% | 46.0% | $0.30 | +| 7 | `openai/gpt-oss-120b` | **84.0%** | 100.0% | 39.9% | $0.00 | +| 8 | `anthropic/claude-sonnet-4.6` | **83.0%** | 100.0% | 43.6% | $0.14 | +| 9 | `google/gemini-3.5-flash` | **81.2%** | 100.0% | 29.9% | $0.17 | +| 10 | `google/gemini-3.1-pro-preview` | **80.5%** | 100.0% | 46.1% | $0.17 | +| 11 | `anthropic/claude-haiku-4.5` | **76.7%** | 90.0% | 43.8% | $0.05 | + +### Equity research (subgroup & research reasoning, single run) + +| Rank | Model | Composite | Grounding | Calibration | Model Cost | +|------|-------|:---------:|:---------:|:-----------:|:----------:| +| 1 | `openai/gpt-5.5@medium` | **83.0%** | 79.8% | 58.9% | $1.10 | +| 2 | `openai/gpt-5.5@xhigh` | **82.8%** | 81.5% | 53.1% | $2.45 | +| 3 | `nvidia/nemotron-3-ultra-550b-a55b` | **81.8%** | 78.5% | 52.8% | $0.10 | +| 4 | `openai/gpt-5.5@high` | **81.1%** | 77.7% | 52.8% | $1.97 | +| 5 | `openai/gpt-5.5@low` | **80.6%** | 76.0% | 58.3% | $0.64 | +| 6 | `anthropic/claude-opus-4.8` | **75.4%** | 69.8% | 49.7% | $0.95 | +| 7 | `google/gemini-3.1-pro-preview` | **73.8%** | 67.8% | 48.3% | $0.59 | +| 8 | `google/gemini-3.5-flash` | **73.0%** | 67.8% | 39.2% | $0.53 | +| 9 | `openai/gpt-oss-120b` | **70.0%** | 62.3% | 32.8% | $0.01 | +| 10 | `anthropic/claude-sonnet-4.6` | **68.7%** | 55.8% | 42.8% | $0.47 | +| 11 | `anthropic/claude-haiku-4.5` | **66.9%** | 66.0% | 35.8% | $0.17 | + +> Composite = 35% Grounding + 20% Insight + 15% Evidence + 15% Calibration + 10% Consistency + 5% Structure (consistency excluded and weights renormalized on single-run packs). + +**Key findings.** (1) *Reasoning effort pays only where tasks are hard:* GPT-5.5's +effort curve rises steeply from low→medium on heavy data analysis +(52%→65%) then saturates, and **inverts** on light trend summaries (low beats +xhigh). (2) *Scores are envelope-relative:* under a 4k token budget the +default-thinking models (Gemini, Nemotron) score near the floor because +reasoning consumes the budget before any answer appears — the same models are +mid-pack or better at 64k. Nemotron still exhausts even 64k on the largest +tasks (it reasons by exhaustive enumeration), so its operations score reflects +budget economics, not analytical ability — it places top-three on both smaller +packs. (3) *Accuracy and stability trade off:* GPT-5.5 leads grounding accuracy +while Claude models lead run-to-run consistency; Claude Haiku 4.5 delivers +~90% of Opus's operations composite at ~18% of its cost. (4) *Calibration is +universally weak* (18–31%): no model reliably acknowledges data limitations +without prompting — the clearest open problem this benchmark surfaces. + +--- --- @@ -223,6 +280,40 @@ If you're a researcher, AI developer, or education technologist and want to disc --- +## Citing GRADE + +If you use GRADE in your research, please cite the repository: + +```bibtex +@software{pearl2026grade, + author = {Pearl}, + title = {{GRADE}: Grounded Reasoning \& Analysis for Data in Education}, + year = {2026}, + publisher = {GitHub}, + version = {0.1.0}, + url = {https://github.com/PearlEng/grade}, + note = {Open benchmark for AI evaluation in education analytics} +} +``` + +A Zenodo DOI for stable versioned citation will be added at the v0.1.0 release. Once available, prefer the DOI-based entry: + +```bibtex +@software{pearl2026grade, + author = {Pearl}, + title = {{GRADE}: Grounded Reasoning \& Analysis for Data in Education}, + year = {2026}, + publisher = {Zenodo}, + version = {0.1.0}, + doi = {10.5281/zenodo.XXXXXXX}, + url = {https://doi.org/10.5281/zenodo.XXXXXXX} +} +``` + +GitHub also provides a formatted citation via the **Cite this repository** button in the sidebar (powered by [`CITATION.cff`](CITATION.cff)). + +--- + ## License Apache 2.0. See [LICENSE](LICENSE).