The gitchameleon/ subdir scaffolds an external evidence run over GitChameleon 2.0 (arXiv:2507.12367; dataset cabbage972/GitChameleon-2.0, MIT; upstream code Apache-2.0): does grounding an agent in the recorded version-pin decision make its code target the pinned API? Arms per the DG-ADR-0001 single-variable pattern (no_grounding / rac / naive_rag); the corpus builder turns each problem into a RAC version-pin decision among deterministic distractors; the dry-run bundle assembly is built and tested offline.
Remaining (funded):
- Pin the answering model and the
naive_rag embedder (the scaffold refuses to run naive_rag until pinned).
- Feed the dry-run bundles to the answering model; collect solutions per (example, arm).
- Score all arms with the upstream harness (
evaluate --solution-path …) — executable tests, deterministic; we add no scorer.
- Publish per-arm pass rates with the dataset revision pin, model pin, and reproduction commands. Results are an arm comparison under our protocol (prompts do not restate the pinned version — GCB-ADR-0001), NOT comparable to the upstream leaderboard, and an unfavourable delta is published plainly.
Evidence run only — upstream scoring never enters a CI merge gate (rac-core ADR-066/ADR-097; roadmap external-benchmark-evidence).
The
gitchameleon/subdir scaffolds an external evidence run over GitChameleon 2.0 (arXiv:2507.12367; datasetcabbage972/GitChameleon-2.0, MIT; upstream code Apache-2.0): does grounding an agent in the recorded version-pin decision make its code target the pinned API? Arms per the DG-ADR-0001 single-variable pattern (no_grounding/rac/naive_rag); the corpus builder turns each problem into a RAC version-pin decision among deterministic distractors; the dry-run bundle assembly is built and tested offline.Remaining (funded):
naive_ragembedder (the scaffold refuses to run naive_rag until pinned).evaluate --solution-path …) — executable tests, deterministic; we add no scorer.Evidence run only — upstream scoring never enters a CI merge gate (rac-core ADR-066/ADR-097; roadmap
external-benchmark-evidence).