Context
borg's harness is meant to be language-agnostic, but the eval corpus (internal/eval/corpus.go) is all Go fixtures today. A non-Go fixture would guard against the harness quietly overfitting to Go, scored by the same objective-oracle approach (does it compile? do the tests pass?).
The task
Add one Python or TypeScript fix-task fixture to the corpus: a small broken project + a prompt + an oracle that checks the fix objectively (e.g. the interpreter/compiler is happy, a test passes). Mirror the structure of the existing Go tasks.
Notes / constraints
- The oracle must be objective, not LLM-as-judge (that's the corpus's whole principle).
- Keep it hermetic and cheap — it runs in CI (deterministically) and in the live eval.
Acceptance
- The new task is proven both solvable and discriminating (a no-op agent fails it).
- Runs in the existing
RunSuite flow. Coverage ≥90%.
A great way to learn the eval harness — see CONTRIBUTING.md on the general-harness principle.
Context
borg's harness is meant to be language-agnostic, but the eval corpus (
internal/eval/corpus.go) is all Go fixtures today. A non-Go fixture would guard against the harness quietly overfitting to Go, scored by the same objective-oracle approach (does it compile? do the tests pass?).The task
Add one Python or TypeScript fix-task fixture to the corpus: a small broken project + a prompt + an oracle that checks the fix objectively (e.g. the interpreter/compiler is happy, a test passes). Mirror the structure of the existing Go tasks.
Notes / constraints
Acceptance
RunSuiteflow. Coverage ≥90%.A great way to learn the eval harness — see
CONTRIBUTING.mdon the general-harness principle.