diff --git a/docs/terminal-bench.md b/docs/terminal-bench.md new file mode 100644 index 00000000..16dd78c3 --- /dev/null +++ b/docs/terminal-bench.md @@ -0,0 +1,80 @@ +# Running hi on Terminal-Bench 2.0 + +[Terminal-Bench](https://github.com/harbor-framework/terminal-bench) is the +standard benchmark for terminal coding agents; [Harbor](https://github.com/harbor-framework/harbor) +is its official harness. Each task runs in its own Docker environment; Harbor +installs the agent into the container, hands it the task instruction, and +scores the resulting container state with the task's own tests — outcome-only, +so intermediate behavior earns no credit. This is the setting studied in +"Failure as a Process" (arXiv:2607.09510), whose 21 model–scaffold baselines +span 19–45% pass rate. + +## One-time setup + +1. **Harbor** (needs Python ≥3.12 and Docker running): + + ```bash + uv tool install harbor + ``` + +2. **A Linux build of `hi`** matching the task-image architecture. The + published Terminal-Bench 2.0 task images are **linux/amd64 only** — even on + Apple Silicon (Docker runs them under Rosetta), the binary must be x86_64. + The workspace's GPU crates don't build in the container, but the binary + crate builds alone; the reproducible path is a containerized build: + + ```bash + docker run --rm --platform linux/amd64 -v "$PWD":/src:ro \ + -v "$PWD/target-linux-amd64":/out \ + rust:1-bookworm cargo build --release -p hi \ + --manifest-path /src/Cargo.toml --target-dir /out + export HI_AGENT_BINARY="$PWD/target-linux-amd64/release/hi" + ``` + + (An emulated build on Apple Silicon takes several minutes; it only needs + to happen once per hi revision.) + +## Running + +```bash +export PIPENETWORK_API_KEY=... # or the provider your model needs +scripts/terminal_bench.sh sample # ~10 tasks, plumbing + cost check +scripts/terminal_bench.sh full # full dataset — mind the spend +scripts/terminal_bench.sh task 'git*' # tasks matching a glob +``` + +Model selection: `HI_TB_MODEL=/` (default +`pipenetwork/ipop/coder-balanced`). Known providers: `pipenetwork`, +`anthropic`, `openrouter`, `openai` — the adapter maps them to `hi --provider` +and forwards the matching API key from your environment into the container. + +Results land under `jobs/terminal-bench//` — per-trial directories +with the agent transcript (`hi.txt`), hi's own `--report` telemetry +(`hi-report.json`, including `repeated_verify_failures` and the trajectory +timeline), and Harbor's verifier verdicts. Harbor prints the aggregate pass +rate at the end of the job. + +## How the adapter works + +`integrations/terminal_bench/hi_agent.py` implements Harbor's +`BaseInstalledAgent`: + +- `install()` uploads `$HI_AGENT_BINARY` into the container at `/opt/hi/hi` + and symlinks it onto `PATH`. +- `run()` executes one `hi --plain --no-save --allow-unverified` turn with the + task instruction, `--report` pointed at Harbor's log directory, and + `XDG_CONFIG_HOME` isolated so a config file baked into a task image can + never shadow the model routing. The command never fails the trial on hi's + exit code — scoring is the verifier's job. +- `populate_context_post_run()` reads the downloaded report and fills Harbor's + token accounting plus a `hi_telemetry` metadata block. + +## Caveats + +- **Architecture**: the uploaded binary must match the task images. Harbor on + Apple Silicon uses aarch64 images by default; build accordingly. +- **Timeouts**: Terminal-Bench tasks carry their own generous agent timeouts; + hi's dynamic step caps apply within them. Use `--timeout-multiplier` on + `harbor run` for slow models. +- **Cost**: the full dataset is hundreds of agent turns. Always run `sample` + first and extrapolate spend before `full`. diff --git a/integrations/__init__.py b/integrations/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/integrations/__pycache__/__init__.cpython-312.pyc b/integrations/__pycache__/__init__.cpython-312.pyc new file mode 100644 index 00000000..79140352 Binary files /dev/null and b/integrations/__pycache__/__init__.cpython-312.pyc differ diff --git a/integrations/terminal_bench/__init__.py b/integrations/terminal_bench/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/integrations/terminal_bench/__pycache__/__init__.cpython-312.pyc b/integrations/terminal_bench/__pycache__/__init__.cpython-312.pyc new file mode 100644 index 00000000..e418d29b Binary files /dev/null and b/integrations/terminal_bench/__pycache__/__init__.cpython-312.pyc differ diff --git a/integrations/terminal_bench/__pycache__/hi_agent.cpython-312.pyc b/integrations/terminal_bench/__pycache__/hi_agent.cpython-312.pyc new file mode 100644 index 00000000..bb9d1c9b Binary files /dev/null and b/integrations/terminal_bench/__pycache__/hi_agent.cpython-312.pyc differ diff --git a/integrations/terminal_bench/hi_agent.py b/integrations/terminal_bench/hi_agent.py new file mode 100644 index 00000000..2b696420 --- /dev/null +++ b/integrations/terminal_bench/hi_agent.py @@ -0,0 +1,132 @@ +"""Harbor (Terminal-Bench 2.0) installed-agent adapter for `hi`. + +Harbor spins up each task's Docker environment, calls ``install()`` to place +the agent inside it, runs the task instruction through ``run()``, then scores +the resulting container state with the task's own tests (outcome-only, like +the Terminal-Bench paper protocol). This adapter uploads a Linux build of +``hi`` (see docs/terminal-bench.md for producing one), runs it one-shot with +``--report``, and surfaces hi's token usage and trajectory telemetry back to +Harbor. + +Usage:: + + export HI_AGENT_BINARY=/path/to/linux/hi + export PIPENETWORK_API_KEY=... + harbor run -d terminal-bench@2.0 \ + --agent integrations.terminal_bench.hi_agent:HiAgent \ + -m pipenetwork/ipop/coder-balanced -n 4 +""" + +import json +import os +import shlex +from pathlib import Path +from typing import override + +from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template +from harbor.environments.base import BaseEnvironment +from harbor.models.agent.context import AgentContext + +_REPORT = "/logs/agent/hi-report.json" +_OUTPUT = "/logs/agent/hi.txt" + +# Harbor model prefix (-m "/") → (hi --provider, key sources). +_PROVIDERS = { + "pipenetwork": ("pipenetwork", ["PIPENETWORK_API_KEY", "HI_API_KEY"]), + "anthropic": ("anthropic", ["ANTHROPIC_API_KEY", "HI_API_KEY"]), + "openrouter": ("openai", ["OPENROUTER_API_KEY", "HI_API_KEY"]), + "openai": ("openai", ["OPENAI_API_KEY", "HI_API_KEY"]), +} + + +class HiAgent(BaseInstalledAgent): + """Container-installed `hi` driven one-shot per task instruction.""" + + @staticmethod + @override + def name() -> str: + return "hi" + + @override + def get_version_command(self) -> str | None: + return "hi --version" + + @override + async def install(self, environment: BaseEnvironment) -> None: + binary = os.environ.get("HI_AGENT_BINARY", "") + if not binary or not Path(binary).is_file(): + raise ValueError( + "HI_AGENT_BINARY must point at a Linux build of `hi` matching " + "the task image architecture — see docs/terminal-bench.md" + ) + await self.exec_as_root(environment, command="mkdir -p /opt/hi") + await environment.upload_file(binary, "/opt/hi/hi") + await self.exec_as_root( + environment, + command=( + "chmod 755 /opt/hi/hi && ln -sf /opt/hi/hi /usr/local/bin/hi " + "&& hi --version" + ), + ) + + @with_prompt_template + async def run( + self, + instruction: str, + environment: BaseEnvironment, + context: AgentContext, + ) -> None: + if not self.model_name or "/" not in self.model_name: + raise ValueError( + "pass -m /, e.g. pipenetwork/ipop/coder-balanced" + ) + provider, model = self.model_name.split("/", 1) + if provider not in _PROVIDERS: + raise ValueError( + f"unknown provider {provider!r}; known: {sorted(_PROVIDERS)}" + ) + hi_provider, key_sources = _PROVIDERS[provider] + + env: dict[str, str] = {} + for key in key_sources: + value = os.environ.get(key) + if value: + env["HI_API_KEY"] = value + break + # Model routing comes entirely from flags/env — never let a config + # file baked into the task image shadow it. + env["XDG_CONFIG_HOME"] = "/tmp/hi-xdg" + + # `|| true`: Terminal-Bench scoring is outcome-only. An unverified or + # incomplete hi turn exits nonzero, but the task tests — not the exit + # code — decide the trial, so the verifier must always get to run. + command = ( + "{ hi --plain --no-save --allow-unverified " + f"--provider {hi_provider} -m {shlex.quote(model)} " + f"--report {_REPORT} " + f"{shlex.quote(instruction)} || true; }} " + f"2>&1 None: + report_path = self.logs_dir / "hi-report.json" + if not report_path.exists(): + return + try: + report = json.loads(report_path.read_text()) + except (OSError, json.JSONDecodeError): + return + # Schema v2 nests usage as {"session": {...}, "turn": {...}}; older + # reports carried flat top-level token fields. + usage = report.get("usage") or {} + session = usage.get("session") or usage + context.n_input_tokens = session.get("input_tokens") or report.get("input_tokens") + context.n_cache_tokens = session.get("cache_read_tokens") + context.n_output_tokens = session.get("output_tokens") + context.metadata = { + "hi_outcome": report.get("outcome"), + "hi_telemetry": report.get("telemetry"), + "hi_verification": report.get("verification"), + } diff --git a/scripts/terminal_bench.sh b/scripts/terminal_bench.sh new file mode 100755 index 00000000..5f05687b --- /dev/null +++ b/scripts/terminal_bench.sh @@ -0,0 +1,58 @@ +#!/bin/bash +# Run hi on Terminal-Bench 2.0 via Harbor. +# +# Usage: +# scripts/terminal_bench.sh sample # 10-task health-check sample +# scripts/terminal_bench.sh full # the full dataset (mind cost!) +# scripts/terminal_bench.sh task # tasks matching a name glob +# +# Requires: docker running, `uv tool install harbor`, a Linux build of hi at +# $HI_AGENT_BINARY (see docs/terminal-bench.md), and PIPENETWORK_API_KEY (or +# override HI_TB_MODEL, e.g. anthropic/claude-sonnet-5 + ANTHROPIC_API_KEY). +set -euo pipefail +cd "$(dirname "$0")/.." +# harbor runs in its own uv tool environment; the adapter module lives in this +# repo, so put the repo root on PYTHONPATH for the import-path --agent. +export PYTHONPATH="${PYTHONPATH:+$PYTHONPATH:}$PWD" + +MODE="${1:-sample}" +MODEL="${HI_TB_MODEL:-pipenetwork/ipop/coder-balanced}" +DATASET="${HI_TB_DATASET:-terminal-bench@2.0}" +JOBS_DIR="${HI_TB_JOBS_DIR:-jobs/terminal-bench}" +CONCURRENT="${HI_TB_CONCURRENT:-4}" + +: "${HI_AGENT_BINARY:?set HI_AGENT_BINARY to a Linux build of hi (docs/terminal-bench.md)}" +[ -f "$HI_AGENT_BINARY" ] || { echo "HI_AGENT_BINARY not found: $HI_AGENT_BINARY"; exit 2; } +docker ps >/dev/null || { echo "docker daemon not running"; exit 2; } + +ARGS=( + run + --dataset "$DATASET" + --agent integrations.terminal_bench.hi_agent:HiAgent + --model "$MODEL" + --n-concurrent "$CONCURRENT" + --jobs-dir "$JOBS_DIR" + --agent-include-logs 'hi-report.json' + --agent-include-logs 'hi.txt' +) + +case "$MODE" in + sample) + # A fixed, arbitrary-but-stable spread of ten real TB-2.0 tasks for + # plumbing + cost checks before a full run. + for t in adaptive-rejection-sampler build-cython-ext configure-git-webserver \ + db-wal-recovery chess-best-move cobol-modernization \ + count-dataset-tokens circuit-fibsqrt cancel-async-tasks \ + break-filter-js-from-html; do + ARGS+=(--include-task-name "$t") + done + ;; + full) ;; + task) + ARGS+=(--include-task-name "${2:?usage: terminal_bench.sh task }") + ;; + *) + echo "unknown mode: $MODE (sample|full|task )"; exit 2 ;; +esac + +exec harbor "${ARGS[@]}"