An open-source, reproducible benchmark harness for comparing how different models perform browser tasks under a controlled agent and browser-tool setup.

Model comparison is the default protocol: vary the model while holding the provider route, browser runtime and skill, task set, judge, limits, and concurrency fixed. Other experimental dimensions are supported, but should be changed independently.

简体中文

Warning

Public beta. Interfaces, configuration, dataset snapshots, and report schemas may change before 1.0. Results are intended for controlled comparisons, not as a universal leaderboard.

Evaluation contract

Layer	Included surface
Variable under test	Model selected with --model or config.yaml
Agent providers	pi and codex
Browser tool	ego-browser by default; additional tools can register through the browser-tool protocol
Task input	A bundled dataset, repeated --task arguments, or an Odysseys-shaped JSON file
Datasets	real-world-bench, odysseys, bu-bench, stealth-bench, online-mind2web
Judge	Pi-based, image-capable agent or single-turn judge
Interfaces	Three independent CLI stages and a local WebUI

For a model comparison, keep the provider route, browser runtime and skill, dataset snapshot, judge configuration, limits, and concurrency unchanged. Engine-level observation and teardown are expressed through CDP semantics so every browser-tool adapter presents the same control surface to the harness.

Pipeline

dataset or ad-hoc tasks
          │
          ▼
run_eval.py   · Stage A · agent execution + browser evidence
          │
          ▼
run_judge.py  · Stage B · verdicts and rubric results
          │
          ▼
run_report.py · Stage C · aggregate report + session HTML

The stages share a run directory but remain independently callable:

• Stage A writes task records, run metadata, and deduplicated screenshots.
• Stage B reads the agent session and screenshots, then writes judge artifacts and a summary. A negative verdict does not trigger another Stage A attempt.
• Stage C renders existing artifacts without changing the evaluation result.

The agent's session JSONL is the source for duration, turns, tokens, cost, tool usage, and final output. tasks.jsonl stores task state and artifact references rather than copying those derived metrics.

Requirements

• Python 3.13 or newer and uv
• Node.js 22.13 or newer, pnpm, and npm for the WebUI and local benchmark sites
• pi on PATH when Stage A or the Judge uses the Pi provider: pi-coding-agent
• fd for Pi's find tool
• For the default ego-browser adapter: macOS, the ego-lite app, and the ego-browser CLI

Quick start

1. Clone and configure an agent

git clone https://github.com/citrolabs/ego-browser-benchmark-framework.git
cd ego-browser-benchmark-framework

# Pi is the default Stage A provider and the built-in Judge provider.
cp agent-home/.pi/agent/auth.json.example agent-home/.pi/agent/auth.json

# Add this when Codex runs Stage A.
cp agent-home/.codex/auth.json.example agent-home/.codex/auth.json

Set the provider and model in config.yaml, or override them per run with --provider and --model.

2. Start the WebUI

Install and finish onboarding in ego lite first. It prepares the ego-browser CLI, Skill, and latest release SDK. Then start the console:

make serve

Before a run, use ego-browser version in New run to inspect or switch the active release. Use scripts/bootstrap.sh only for a local SDK build or to pin a release from the command line before opening the WebUI.

3. Run a smoke task

uv run python run_eval.py \
  --task "Open https://example.com and report the page title."

The command prints the generated run ID and writes artifacts under runs/<run_id>/. Plain --task inputs are not judged. To judge an ad-hoc batch, use --tasks-json with non-empty rubrics and enable --auto-judge.

CLI workflows

Run tasks

# Entire dataset
uv run python run_eval.py --benchmark real-world-bench

# One dataset task
uv run python run_eval.py \
  --benchmark real-world-bench \
  --task-ids <task-id>

# Multiple ad-hoc tasks
uv run python run_eval.py \
  --task "task one" \
  --task "task two" \
  --concurrency 2

# Odysseys-shaped ad-hoc batch with rubrics
uv run python run_eval.py \
  --tasks-json path/to/tasks.json \
  --auto-judge

Judge an existing run

# Judge tasks that do not yet have a result
uv run python run_judge.py --run-id <run_id>

# Retry only judge infrastructure errors
uv run python run_judge.py --run-id <run_id> --rejudge-errors

Stage B also supports whole-run, selected-task, single-task, and rubric-level rejudging. Run uv run python run_judge.py --help for the selectors.

Generate reports

# Aggregate report; zh is the default locale
uv run python run_report.py --single runs/<run_id>
uv run python run_report.py --single runs/<run_id> --locale en

# Session HTML only
uv run python run_report.py --export-sessions runs/<run_id>

Each entry point exposes its full option set through --help.

WebUI

make serve    # WebUI :8365 and benchmark sites :4374
make dev      # WebUI :8365, Vite :5373, benchmark sites :4374

The WebUI can:

• configure dataset and ad-hoc runs;
• queue, monitor, stop, resume, and selectively rerun tasks;
• inspect Judge progress and rejudge a run, task, or rubric subset;
• export aggregate reports, agent sessions, judge sessions, and screenshot galleries.

CLI- and WebUI-started runs use the same runs/ layout and remain available in both interfaces. Local-site tasks require the benchmark-site service started by make serve or make dev.

Configuration

Configuration precedence is:

CLI arguments > config.yaml > code defaults

config.yaml groups the run, provider, sampling, limit, judge, screenshot, WebUI, and browser-tool settings. The WebUI creates per-run overrides and records the resolved configuration in run_metadata.json; it does not rewrite config.yaml.

Artifacts and result validity

A completed run commonly contains:

runs/<run_id>/
├── run_metadata.json
├── tasks.jsonl
├── summary.json
├── screenshots/<task_id>/frames/
├── judges/
├── sessions_html/
└── judge_sessions_html/

Use the raw session JSONL, task records, judge JSON, screenshots, and accessibility trees when auditing a result. The HTML files are presentation artifacts.

Most bundled tasks target live websites. Site changes and anti-bot challenges can prevent a task from measuring the model at all. Sign the browser into required sites before a comparison run, and separate intercepted or stale-site tasks from genuine agent failures.

Add a browser tool

The built-in implementation is ego-browser, but the harness does not require callers to branch on browser-tool names.

1. Implement the BrowserTool protocol: AgentSurface describes how the agent invokes the CLI, and EngineControl provides screenshot and teardown operations.
2. Register the class with register_browser_tool before a run starts.
3. Add browser_tools.<name> to config.yaml with bin_env, bin_default, and skill_name.
4. If the CLI has task-space management subcommands, register them with register_task_management_subcommands.

The WebUI reads the registered options from the backend, so an adapter does not require a frontend-specific tool list.

Datasets and task resources

Dataset adapters live in ego_bench/datasets/, and their snapshots live in data/. See data/README.md for schemas, task counts, resource fields, and redistribution notes.

Tasks can reference named attachments under data/attachments/ or deterministic local sites under benchmark_sites/app/. See benchmark_sites/README.md before adding a local site.

Project map

Path	Purpose
ego_bench/	Core package: providers, datasets, browser tools, judging, reports, and WebUI backend
run_eval.py	Stage A entry point
run_judge.py	Stage B entry point
run_report.py	Stage C entry point
config.yaml	Repository defaults and local overrides
agent-home/	Pi and Codex runtime templates
frontend/	React WebUI
benchmark_sites/	Deterministic local benchmark sites
tests/	Python tests and repository checks

Development

uv sync
uv run ruff check .
uv run pytest -q
make build

License

The framework code is available under the MIT License. Some bundled task texts come from external benchmark datasets and retain their upstream terms; review data/README.md before redistributing them.