Skip to content

Releases: PearlEng/grade

Release list

V0.1

Choose a tag to compare

@jonathanbbechtel jonathanbbechtel released this 12 Jun 06:10
9732350
Feat/backfill missing tasks (#46)

* fix(adapter): surface OpenRouter error body instead of KeyError('choices')

OpenRouter returns error objects with HTTP 200 for some failures (context
length exceeded, provider moderation). The adapter crashed with an opaque
KeyError('choices'); now it raises RuntimeError with the provider's actual
error message. Found live when gpt-oss-120b (131k context) hit the ~184k-token
operations tasks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(scripts): backfill_tasks — re-run only missing/failed tasks and merge into existing row

Reconstructs TaskRunResults from result.json + raw_outputs.jsonl, re-runs
only the named task IDs (fixtures resolve via explicit pack_id), re-aggregates
with summed judge cost, and rewrites the row in place with .bak backups.
Clears stale failures.json when the row becomes complete. Validated end-to-end
with the stub adapter (overwrite-merge and fresh-row paths).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(adapter): retry transient transport errors and 429/5xx with backoff

Mid-stream connection drops ('peer closed connection') killed whole tasks —
three leaderboard rows were damaged this way. Both HTTP call sites (candidate
run() and the shared post_chat_completion used by the judge) now retry up to
GRADE_OPENROUTER_RETRIES attempts (default 3) with exponential backoff on
httpx.TransportError and 429/500/502/503/504 responses. Timeouts still
propagate unretried (each attempt already waits the full configured timeout).
run()'s hardcoded 120s timeout now honors GRADE_OPENROUTER_TIMEOUT too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* updated readme

* docs: add citation infrastructure for research community (#45)

- Add CITATION.cff so GitHub surfaces a "Cite this repository" button
- Add ## Citing GRADE section to README with BibTeX entries (GitHub URL
  now, Zenodo DOI placeholder for v0.1.0 release)
- Set repo topics: benchmark, llm-evaluation, education, ai-evaluation,
  nlp, education-analytics, grounding, open-benchmark

Co-authored-by: jonathan bechtel <jonathanbechtel@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: jonathan bechtel <jonathanbechtel@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>