Releases: PearlEng/grade
Releases · PearlEng/grade
Release list
V0.1
Feat/backfill missing tasks (#46)
* fix(adapter): surface OpenRouter error body instead of KeyError('choices')
OpenRouter returns error objects with HTTP 200 for some failures (context
length exceeded, provider moderation). The adapter crashed with an opaque
KeyError('choices'); now it raises RuntimeError with the provider's actual
error message. Found live when gpt-oss-120b (131k context) hit the ~184k-token
operations tasks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(scripts): backfill_tasks — re-run only missing/failed tasks and merge into existing row
Reconstructs TaskRunResults from result.json + raw_outputs.jsonl, re-runs
only the named task IDs (fixtures resolve via explicit pack_id), re-aggregates
with summed judge cost, and rewrites the row in place with .bak backups.
Clears stale failures.json when the row becomes complete. Validated end-to-end
with the stub adapter (overwrite-merge and fresh-row paths).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(adapter): retry transient transport errors and 429/5xx with backoff
Mid-stream connection drops ('peer closed connection') killed whole tasks —
three leaderboard rows were damaged this way. Both HTTP call sites (candidate
run() and the shared post_chat_completion used by the judge) now retry up to
GRADE_OPENROUTER_RETRIES attempts (default 3) with exponential backoff on
httpx.TransportError and 429/500/502/503/504 responses. Timeouts still
propagate unretried (each attempt already waits the full configured timeout).
run()'s hardcoded 120s timeout now honors GRADE_OPENROUTER_TIMEOUT too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* updated readme
* docs: add citation infrastructure for research community (#45)
- Add CITATION.cff so GitHub surfaces a "Cite this repository" button
- Add ## Citing GRADE section to README with BibTeX entries (GitHub URL
now, Zenodo DOI placeholder for v0.1.0 release)
- Set repo topics: benchmark, llm-evaluation, education, ai-evaluation,
nlp, education-analytics, grounding, open-benchmark
Co-authored-by: jonathan bechtel <jonathanbechtel@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: jonathan bechtel <jonathanbechtel@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>