Skip to content

GitChameleon evidence run: rac-grounding arm (funded run over the scaffold) #10

Description

@tcballard

The gitchameleon/ subdir scaffolds an external evidence run over GitChameleon 2.0 (arXiv:2507.12367; dataset cabbage972/GitChameleon-2.0, MIT; upstream code Apache-2.0): does grounding an agent in the recorded version-pin decision make its code target the pinned API? Arms per the DG-ADR-0001 single-variable pattern (no_grounding / rac / naive_rag); the corpus builder turns each problem into a RAC version-pin decision among deterministic distractors; the dry-run bundle assembly is built and tested offline.

Remaining (funded):

  1. Pin the answering model and the naive_rag embedder (the scaffold refuses to run naive_rag until pinned).
  2. Feed the dry-run bundles to the answering model; collect solutions per (example, arm).
  3. Score all arms with the upstream harness (evaluate --solution-path …) — executable tests, deterministic; we add no scorer.
  4. Publish per-arm pass rates with the dataset revision pin, model pin, and reproduction commands. Results are an arm comparison under our protocol (prompts do not restate the pinned version — GCB-ADR-0001), NOT comparable to the upstream leaderboard, and an unfavourable delta is published plainly.

Evidence run only — upstream scoring never enters a CI merge gate (rac-core ADR-066/ADR-097; roadmap external-benchmark-evidence).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions