RepoGym

A tiny, local benchmark harness for coding agents. Verified code, not vibes.

RepoGym runs the same narrow coding tasks against multiple agents in fresh workspaces, executes deterministic verification commands, measures time/tokens/cost when available, and emits a static HTML report plus a shareable SVG scorecard.

It is intentionally small:

• one obvious entrypoint: repogym.py
• Python 3.11+ standard library only
• no server, account, database, Docker, or framework
• clean snapshots made with file copies or git archive
• agent commands configured as ordinary subprocess argument arrays
• tests/build/lint commands decide whether work passes
• all artifacts remain local

RepoGym is inspired by the public engineering style of nanoGPT, nanochat, llm.c, llama2.c, micrograd, and autoresearch: sharp scope, readable code, visible metrics, fixed evaluation loops, and minimal machinery. It is not affiliated with Andrej Karpathy or those projects.

See it work offline

No API key or coding agent is required for the included demo.

python repogym.py demo

Expected shape:

[careful × fix-add]
  PASS · 1/1 checks
[careful × normalize-name]
  PASS · 1/1 checks
[cheap × fix-add]
  PASS · 1/1 checks
[cheap × normalize-name]
  FAIL · 0/1 checks

Agent    Verified  Score   Tokens  Cost
-------  --------  ------  ------  -------
careful  2/2       100.0%  2,000   $0.0200
cheap    1/2       50.0%   750     $0.0008

Open the generated demo report.

The command writes:

.repogym/demo-runs/<run-id>/
├── results.json
├── report.html
├── scorecard.svg
└── cases/
    └── <agent>/<task>/
        ├── prompt.md
        ├── agent.log
        ├── verify-1.log
        ├── result.json
        └── workspace/

Open report.html directly in a browser. It contains no JavaScript and makes no network requests.

Quick start on your repository

Requirements:

• Python 3.11 or newer
• Git on PATH for historical-commit tasks
• at least one coding-agent CLI for real runs

Create a starter configuration:

python repogym.py init /path/to/your/repo
cd /path/to/your/repo
python /path/to/repogym/repogym.py doctor

Edit repogym.toml, then run:

python /path/to/repogym/repogym.py run --agents codex --tasks fix-example

Run every configured agent against every configured task:

python /path/to/repogym/repogym.py run

Minimal configuration

[project]
name = "calculator"
repo = "."
output_dir = ".repogym/runs"
timeout_seconds = 900
verify = [
  ["{python}", "-m", "pytest", "-q"],
]

[agents.codex]
command = [
  "codex", "exec",
  "--ephemeral",
  "--sandbox", "workspace-write",
  "--cd", "{workspace}",
  "-",
]
stdin_prompt = true
timeout_seconds = 900

[agents.opencode]
command = ["opencode", "run", "{prompt}"]
timeout_seconds = 900

[[tasks]]
id = "fix-rounding"
title = "Repair invoice rounding"
prompt = """
Invoice totals currently round each line before summing. Sum exact line values first,
then round the final total to two decimal places. Preserve the existing public API.
"""
verify = [
  ["{python}", "-m", "pytest", "tests/test_invoice.py", "-q"],
]
weight = 1.0

Commands are arrays rather than shell strings. This avoids most quoting problems and works on Windows, macOS, and Linux.

Agent adapters are just commands

RepoGym does not import agent SDKs. It launches whatever CLI you configure inside the isolated workspace.

Available placeholders:

Placeholder	Value
{workspace}	isolated working directory
{prompt_file}	generated Markdown prompt path
{prompt}	full generated prompt text
{task_id}	task identifier
{task_title}	task title
{python}	current Python executable
{repogym_root}	directory containing repogym.py
{config_dir}	directory containing repogym.toml

Examples for more CLIs are in docs/agents.md. CLI interfaces change; run each tool's --help and adjust the command array when necessary.

What counts as success?

For each agent × task case, RepoGym:

1. creates a fresh workspace;
2. writes the exact prompt to prompt.md;
3. executes the agent once;
4. runs every configured verification command;
5. records changed files, exit codes, time, output, tokens, and estimated cost;
6. marks the case successful only when the agent exits successfully and every verification command passes.

The primary score is weighted verification pass rate:

score = sum(task_weight × task_pass_rate) / sum(task_weight)

A task with two checks and one passing check receives a 0.5 pass rate. A fully verified task receives 1.0.

When token totals are available, RepoGym also reports:

verified tasks per million tokens

See docs/scoring.md for caveats and interpretation.

Historical-commit tasks

Generate reviewable task stubs from recent single-parent commits:

python repogym.py mine --repo . --limit 5 --output mined-tasks.toml

RepoGym records the parent as base_rev and the commit as target_rev. During a run, it exports the base revision with git archive, so the workspace contains no .git history and the agent cannot simply inspect the target commit. Changed test paths can be overlaid from the target revision.

Generated tasks are stubs, not magic benchmark data. Review the prompt, target tests, required fixtures, and verification commands before using them. More detail is in docs/historical-tasks.md.

Commands

python repogym.py init [PATH] [--force]
python repogym.py doctor [--config FILE]
python repogym.py run [--config FILE] [--agents a,b] [--tasks x,y]
python repogym.py report RUN_DIR
python repogym.py mine [--repo PATH] [--limit N] [--output FILE]
python repogym.py demo [--output-dir PATH]

Design constraints

RepoGym deliberately does not:

• judge code with another LLM;
• upload repositories or logs;
• manage API keys;
• normalize token accounting across providers;
• provision containers or virtual machines;
• pretend tests prove semantic correctness when the tests are weak;
• hide the subprocess commands it runs.

Those constraints keep the harness inspectable. Add infrastructure around it only when your experiment requires it.

Documentation

• docs/architecture.md — execution pipeline and file layout
• docs/configuration.md — complete TOML reference
• docs/agents.md — Codex, OpenCode, and custom CLI examples
• docs/tasks.md — designing narrow, discriminating tasks
• docs/historical-tasks.md — commit-derived benchmarks
• docs/scoring.md — metrics and limitations
• docs/windows.md — Windows 11 and PowerShell notes
• docs/autoresearch.md — a safe future optimization loop
• docs/faq.md — common questions
• docs/roadmap.md — intentionally small roadmap
• docs/publishing.md — upload, release, and package steps

Test the harness

python -m unittest discover -s tests -v
python repogym.py demo

The GitHub Actions workflow runs both commands on Ubuntu and Windows.

Security

Agent CLIs execute code and may modify every file in their isolated workspace. Verification commands also execute repository code. Run RepoGym only against repositories and agents you trust. File isolation is not an operating-system sandbox.

Read SECURITY.md before benchmarking untrusted code.

License

MIT. See LICENSE.