agent-bench

A lightweight benchmarking toolkit for LLM agents.

Think of it as pytest for agent evaluation: define tasks, run agents against them, and get structured metrics — correctness, tool usage, latency, cost, and retries.

╭─────────────────────────────────────────────────────────────────────────────╮
│                          agent-bench results                                │
├──────────────────────────┬────────┬───────┬──────────┬────────┬────────────┤
│ Task                     │ Result │ Score │  Latency │ Tokens │ Cost (USD) │
├──────────────────────────┼────────┼───────┼──────────┼────────┼────────────┤
│ arithmetic/add           │  PASS  │  1.00 │   412 ms │    82  │  $0.00002  │
│ arithmetic/subtract      │  PASS  │  1.00 │   389 ms │    81  │  $0.00002  │
│ tool_use/calculator      │  PASS  │  1.00 │   731 ms │   198  │  $0.00007  │
│ tool_use/weather_lookup  │  PASS  │  1.00 │   843 ms │   241  │  $0.00008  │
╰──────────────────────────┴────────┴───────┴──────────┴────────┴────────────╯
╭─── Summary ─────────────────────────────────────────────────────────────────╮
│ Tasks: 4   Passed: 4/4   Accuracy: 100.0%                                   │
│ Avg score: 1.000   Avg latency: 594 ms   Total tokens: 602                  │
│ Total cost: $0.00019   Tool calls: 2   Elapsed: 2.37 s                      │
╰─────────────────────────────────────────────────────────────────────────────╯

---

Install

git clone https://github.com/filippopellizzari/agent-bench
cd agent-bench

# Core library only (no LLM provider)
uv sync

# With OpenAI support
uv sync --extra openai

# With Anthropic support
uv sync --extra anthropic

# Everything (all providers + dev tools)
uv sync --extra all --extra dev

---

Quickstart

1. Define a task

# tasks/my_tasks.py
from agent_bench import Task, Score

class CapitalOfFrance(Task):
    name = "geography/capital_of_france"
    description = "Agent must name the capital of France"

    def prompt(self) -> str:
        return "What is the capital of France? Reply with only the city name."

    def grader(self, output: str) -> Score:
        correct = "paris" in output.lower()
        return Score(correct=correct, value=1.0 if correct else 0.0)

2. Run from the CLI

# Standard OpenAI
export OPENAI_API_KEY=sk-...
agent-bench run tasks/my_tasks.py --model gpt-4o-mini

# Azure OpenAI
export AZURE_OPENAI_API_KEY=...
export AZURE_OPENAI_ENDPOINT=https://<resource>.openai.azure.com/
agent-bench run tasks/my_tasks.py --model my-gpt4o-deployment

# Save results to JSON
agent-bench run tasks/my_tasks.py --model gpt-4o-mini --output results.json

3. Or run from Python (OpenAI)

import openai
from agent_bench import OpenAIAgent, BenchmarkRunner
from agent_bench.reporters.rich_reporter import print_results
from tasks.my_tasks import CapitalOfFrance

# Standard OpenAI
client = openai.OpenAI()

# Azure OpenAI
# client = openai.AzureOpenAI(
#     api_key=os.environ["AZURE_OPENAI_API_KEY"],
#     azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
#     api_version="2024-02-01",
# )

agent = OpenAIAgent(model="gpt-4o-mini", client=client)
runner = BenchmarkRunner(agent=agent, tasks=[CapitalOfFrance()])
result = runner.run()
print_results(result)

4. Or run from Python (Anthropic)

from agent_bench import AnthropicAgent, BenchmarkRunner
from agent_bench.reporters.rich_reporter import print_results
from tasks.my_tasks import CapitalOfFrance

# Reads ANTHROPIC_API_KEY from environment
agent = AnthropicAgent(model="claude-haiku-4-5-20251001")
runner = BenchmarkRunner(agent=agent, tasks=[CapitalOfFrance()])
result = runner.run()
print_results(result)

---

CLI reference

agent-bench run <files/dirs>  [OPTIONS]
  --model             -m   Deployment/model name       (default: gpt-4o-mini)
  --workers           -w   Parallel workers            (default: 1)
  --output            -o   Write results to JSON file
  --azure-endpoint         Azure endpoint URL          (or AZURE_OPENAI_ENDPOINT)
  --azure-api-key          Azure API key               (or AZURE_OPENAI_API_KEY)
  --azure-api-version      Azure API version           (or AZURE_OPENAI_API_VERSION, default: 2024-02-01)
  --api-key                Standard OpenAI API key     (or OPENAI_API_KEY, used when no --azure-endpoint)
  --temperature            Sampling temperature        (default: 0.0)
  --max-tokens             Max output tokens           (default: 1024)

agent-bench list <files/dirs>
  Lists discovered Task subclasses and their metadata.

---

Task with tool use

Tasks can expose OpenAI tool schemas and optionally implement execute_tool():

from agent_bench import Task, Score

_CALC_TOOL = {
    "type": "function",
    "function": {
        "name": "calculator",
        "description": "Evaluate a math expression.",
        "parameters": {
            "type": "object",
            "properties": {"expression": {"type": "string"}},
            "required": ["expression"],
        },
    },
}

class CalculatorTask(Task):
    name = "tool_use/calculator"
    tools = [_CALC_TOOL]

    def prompt(self) -> str:
        return "Use the calculator to compute 2 ** 10."

    def execute_tool(self, name: str, arguments: dict) -> str:
        if name == "calculator":
            expr = arguments["expression"]
            return str(eval(expr, {"__builtins__": {}}, {}))
        return "unknown tool"

    def grader(self, output: str) -> Score:
        return Score(correct="1024" in output)

---

Custom agent

Implement the Agent protocol to plug in any backend:

from agent_bench.core.agent import Agent
from agent_bench.core.metrics import AgentResponse
from agent_bench.core.task import Task

class MyAgent:
    def run(self, task: Task) -> AgentResponse:
        # call your LLM here
        return AgentResponse(
            content="...",
            input_tokens=0,
            output_tokens=0,
            latency_ms=0,
            model="my-model",
        )

---

Metrics collected

Metric	Description
correct	Boolean grader result
value	Normalised score 0.0–1.0
latency_ms	End-to-end wall time including tool call loop
input_tokens / output_tokens	Usage from the API
cost_usd	Estimated cost (built-in pricing table for OpenAI models)
tool_calls	List of ToolCall(name, arguments) structs
retries	Number of task-level retries attempted

---

Running tests

uv run pytest tests/ -v

No API key required — tests use a mock agent.

---

Project layout

src/agent_bench/
├── core/
│   ├── task.py          # Task ABC + Score
│   ├── metrics.py       # AgentResponse, RunResult, BenchmarkResult
│   ├── agent.py         # Agent Protocol + OpenAIAgent
│   └── runner.py        # BenchmarkRunner (sync + async)
├── reporters/
│   └── rich_reporter.py # Rich terminal table
└── cli.py               # Typer CLI
examples/
├── tasks/
│   ├── math_tasks.py
│   └── tool_use_tasks.py
└── run_demo.py
tests/
├── test_task.py
├── test_metrics.py
└── test_runner.py