Quickstart · Install · CLI reference · Task catalog · FAQ · vs. other benchmarks · Limitations · Contribute · Cite
Benchmark framework for evaluating AI agent systems in High-Performance Computing (HPC) environments.
AOBench measures how well AI agents complete HPC operational tasks — job scheduling, telemetry interpretation, energy reasoning, policy enforcement — using the right tools, the right roles, and the right permissions. Instead of running on live clusters, every task is evaluated against a deterministic environment snapshot with mock HPC tools (SLURM, telemetry, RBAC, docs, facility), so results are reproducible, portable, and safe to publish.
Every number in that recording was produced offline on a laptop — no cluster, no API key. Reproduce it with the two commands shown.
In one line: AOBench is an AI agent benchmark for HPC — an HPC agent evaluation framework that is role-aware, permission-enforced, tool-using, trace-based, and reproducible.
What / who / why (quotable summary):
- AOBench (Agent Operations Benchmark) is an open-source Python benchmark framework for evaluating AI agents that operate High-Performance Computing (HPC) systems.
- AOBench helps researchers and engineers measure whether an AI agent completes HPC operational tasks — job scheduling, telemetry interpretation, energy reasoning, and policy enforcement — with the right tools, roles, and permissions.
- Use AOBench when you need reproducible, role-aware, permission-enforced evaluation of HPC agents against deterministic environment snapshots instead of live clusters.
- AOBench differs from general-purpose LLM-agent benchmarks because it is domain-specific to HPC operations, enforces role-based access control (RBAC), and scores the full execution trace rather than only the final answer.
- AOBench is not intended for measuring general-purpose reasoning, software-engineering, or web-browsing agents, and it does not execute against real production clusters.
Name collision:
aobenchis also a long-standing ambient-occlusion ray-tracing microbenchmark by Syoyo Fujita (syoyo/aobench), unrelated to this project. This AOBench is the Agent Operations Benchmark. If you came here looking for the renderer, that is the link.
- Python ≥ 3.10 (3.12 is used in the Docker image and CI, and is recommended).
- Optional:
openai,anthropic, ormcpPython clients to drive the corresponding adapters.
See docs/getting-started/installation.md for the full installation guide — the Python package, the Docker CLI image, and the Docker Compose service stack.
| Principle | Meaning |
|---|---|
| Role-aware | The same question yields different answers and tool access depending on the requester role. |
| Tool-using | Agents are evaluated as systems that call HPC-native tools (SLURM, telemetry, docs, RBAC, facility). |
| Permission-aware | Success requires respecting RBAC and refusing out-of-scope requests. Permission violations hard-fail the task. |
| Trace-based | Evaluation considers the full execution trace — tool selection, arguments, sequence, and grounding — not just the final answer. |
| Reproducible | Runs target deterministic snapshot bundles, never live infrastructure. |
AOBench/
├── src/aobench/ # Python package (installed by `pip install -e .`)
│ ├── cli/ # `aobench` typer app — 15 sub-commands
│ ├── schemas/ # Pydantic data models (task, trace, snapshot, …)
│ ├── loaders/, tasks/ # Task discovery, loading, dataset splits, RAG context
│ ├── environment/ # Snapshot validator, snapshot loader factory
│ ├── tools/ # Mock SLURM, telemetry, docs, RBAC, facility tools
│ ├── adapters/ # direct_qa, openai, anthropic, mcp
│ ├── runners/ # BenchmarkRunner, TraceWriter, ExecutionContext
│ ├── scorers/ # 12 scorers across 7 weighted dimensions
│ ├── reports/ # JSON, HTML, slice, CLEAR scorecard reports
│ ├── exporters/ # Langfuse exporter (optional)
│ ├── leaderboard/ # FastAPI leaderboard service
│ ├── reproducibility/ # Artifact locking + paper-table targets
│ └── taxonomy/ # 24-leaf TRAIL-adapted HPC error taxonomy
│
├── benchmark/ # Static benchmark data (versioned in git)
│ ├── tasks/specs/ # 88 JSON task specs (80 synthetic + 8 M100 ExaData)
│ ├── tasks/task_set_v1.json # 36 HPC v1 tasks (Souza 2025 schema)
│ ├── tasks/task_set_v3.json # v3 task index (88 tasks)
│ ├── tasks/dataset_splits.py # 67 dev / 21 test (synthetic core: 59 dev / 21 test)
│ ├── tasks/lite_manifest_v1.json # AOBench-Lite curated subset
│ ├── environments/ # 29 snapshot bundles (23 synthetic + 6 M100)
│ ├── configs/ # scoring_profiles.yaml, hpc_tool_catalog.yaml,
│ │ # error_taxonomy.yaml
│ └── qa/ # AOBench-QA (~95 HPC operational queries)
│
├── data/ # Generated artifacts
│ ├── runs/ # Per-run traces & results (gitignored)
│ ├── reports/ # Validity gate reports
│ ├── robustness/ # pass^k results
│ └── rubric_validation/ # Annotator profiles, response set, guides
│
├── prompts/judge/ # LLM-judge rubric + error taxonomy templates
├── docs/ # Documentation (see Documentation section)
├── scripts/ # Bundle generation, validity gates, rubric tooling
└── tests/ # 83 test files, ~1510 tests (unit + integration)
Two commands to your first scored HPC agent task. No cluster, no API key, no
configuration — the direct_qa baseline runs offline against a frozen snapshot.
git clone https://github.com/MSKazemi/aobench.git && cd aobench && make install
aobench quickstartaobench quickstart takes no arguments: it locates the benchmark corpus, picks a
representative task, runs it, and explains every number it prints.
Aggregate score: 0.3340 (0 = worst, 1 = best)
Per dimension:
outcome 0.2400 did the answer match the gold answer
tool_use 0.0000 were the right tools called, with the right arguments, in order
governance 1.0000 did the agent stay inside its RBAC role
grounding 0.0000 was the answer supported by the snapshot evidence
efficiency 1.0000 how much work was spent getting there
That 0.334 is the tool-free floor a real agent has to beat. From there:
aobench doctor # is my install healthy?
aobench list tasks --qcat JOB # what else can I run? (also: list envs / roles / adapters)
aobench validate benchmark # do all 88 tasks and 29 environments load?
aobench run task --task JOB_USR_001 --env env_01 --adapter direct_qaEvaluate a real model and produce a CLEAR scorecard:
export OPENAI_API_KEY=sk-…
aobench run all --adapter openai:gpt-4o --split dev
aobench clear run data/runs/<run_id>Full walkthrough: docs/getting-started/quickstart.md. Other install paths (Docker, Compose, extras): docs/getting-started/installation.md.
flowchart LR
T["📋 Task spec<br/>role · question · gold trace"]
E["📦 Environment snapshot<br/>frozen SLURM · telemetry · docs"]
R["⚙️ BenchmarkRunner"]
A["🤖 Your agent<br/>direct_qa · openai · anthropic · mcp"]
TOOLS["🔧 Mock HPC tools<br/>slurm · telemetry · docs · rbac · facility"]
TR["🧾 Trace<br/>every call, argument, and answer"]
S["📊 12 scorers<br/>7 weighted dimensions"]
C["🏆 CLEAR scorecard<br/>Efficacy · Assurance · Reliability · Cost · Latency"]
T --> R
E --> R
R --> A
A -- "tool calls" --> TOOLS
TOOLS -- "role-filtered results" --> A
A --> TR
TR --> S
S --> C
classDef input fill:#e8eaf6,stroke:#3949ab,stroke-width:2px,color:#1a237e
classDef core fill:#1a237e,stroke:#0d1452,stroke-width:2px,color:#ffffff
classDef output fill:#fff3e0,stroke:#ff8f00,stroke-width:2px,color:#e65100
class T,E input
class R,A,TOOLS core
class TR,S,C output
An RBAC violation hard-fails the task — an agent that produces the right answer by overstepping its role scores zero, no matter how good the answer is.
Beyond the CLI, AOBench exposes the benchmark engine over four machine
surfaces so agents and pipelines can run and score tasks directly. Every surface
calls the same BenchmarkService façade — transports carry no scoring logic, so
CLEAR scores are identical across all of them.
| Surface | Start it with | What it exposes |
|---|---|---|
| REST / FastAPI | aobench serve rest (extra: rest) |
/v1/* HTTP endpoints — run, score, report, trace, compare, robustness, datasets, async jobs, SSE progress |
| MCP / FastMCP | aobench serve mcp (extra: mcp) |
MCP tools (run_task, score_trace, validate_benchmark, robustness) + aobench:// resources |
| A2A (Agent2Agent) | evaluation scorers + adapter core | Agent-Card conformance, delegation / comms-cost / attribution / lifecycle / card-poisoning scorers |
| CLI / terminal track | evaluation scorers + adapter core | Mock Slurm shims, destructive-command guard, end-state verification |
uv sync --extra rest --extra mcp # install both server extras (list together)
aobench serve rest --host 0.0.0.0 --port 8000
aobench serve mcp # stdio, for an MCP client to spawnSee the programmatic-access guide and the serving tutorial for end-to-end walkthroughs, and ROADMAP.md for surface status and what's next.
| Item | Count | Location |
|---|---|---|
| Tasks | 88 — 80 synthetic core (10 QCATs × 5 roles) + 8 grounded in real Marconi100 ExaData | benchmark/tasks/specs/ |
| Environments | 29 deterministic snapshot bundles — 23 synthetic + 6 built from real Marconi100 ExaData | benchmark/environments/ |
| Roles (scored) | 5 — scientific_user, sysadmin, facility_admin, researcher, system_designer |
src/aobench/schemas/task.py |
| QCATs (scored) | 10 — JOB, MON, ENERGY, PERF, DATA, SEC, FAC, ARCH, AIOPS, DOCS |
benchmark/tasks/specs/ |
| Adapters | 4 — direct_qa, openai, anthropic, mcp |
src/aobench/adapters/ |
| Mock tool families | 5 — slurm, docs, rbac, telemetry, facility | src/aobench/tools/ |
| Scorers | 12 across 6 dimensions | src/aobench/scorers/ |
| Scoring profiles | alpha0_minimal, alpha1_grounding, default_hpc_v01 |
benchmark/configs/scoring_profiles.yaml |
| Tests | ~1470 passing | tests/ |
The 7 evaluation dimensions and their default_hpc_v01 weights — verified against
benchmark/configs/scoring_profiles.yaml in CI, and printable with
aobench list profiles:
| Dimension | Weight | Scorer |
|---|---|---|
| Outcome correctness | 0.30 | OutcomeScorer (or HybridScorer) |
| Governance / RBAC | 0.20 | GovernanceScorer |
| Tool-use correctness | 0.15 | ToolUseScorer (BFCL-decomposed) |
| Grounding | 0.10 | GroundingScorer |
| Robustness (pass^k) | 0.10 | RobustnessScorer |
| Workflow (WorfEval) | 0.10 | WorfEvalScorer |
| Efficiency | 0.05 | EfficiencyScorer |
Row form for machine comparison: 0.30 | 0.15 | 0.10 | 0.20 | 0.10 | 0.05 | 0.10
(outcome, tool_use, grounding, governance, robustness, efficiency, workflow).
The CLEAR scorecard (aobench clear run) aggregates Efficacy, Assurance,
Reliability, Cost, and Latency into a single comparable score per model.
- Compare models as HPC agents. Run the same task suite across
openai,anthropic, andmcpadapters and rank them with a single CLEAR scorecard. - Test tool-use and grounding. Check whether an agent selects the right HPC-native tool (SLURM, telemetry, docs, RBAC, facility) with correct arguments and grounds its answer in the environment snapshot.
- Verify permission safety. Confirm an agent respects role-based access control and refuses out-of-scope requests — permission violations hard-fail the task.
- Reproduce and publish results. Evaluate against deterministic snapshot bundles so runs are portable and safe to publish without live-cluster access.
- Author new tasks and environments. Extend the 88-task / 29-environment corpus using the versioned JSON specs and snapshot format.
AOBench is a domain-specific benchmark for HPC operations. It complements, rather than replaces, general-purpose agent benchmarks:
| Benchmark | Primary domain | How AOBench differs |
|---|---|---|
| General LLM-agent benchmarks (e.g. AgentBench, GAIA) | Broad assistant / reasoning tasks | AOBench targets HPC operational tasks with role-aware RBAC and deterministic HPC snapshots. |
| Tool-use / function-calling benchmarks (e.g. τ-bench, BFCL) | General tool and function calling | AOBench scores tool use within HPC scenarios (SLURM, telemetry, facility) and combines it with governance, grounding, robustness, and efficiency. AOBench's ToolUseScorer is BFCL-decomposed. |
| Software-engineering agent benchmarks (e.g. SWE-bench) | Code repair / repositories | AOBench evaluates HPC operations, not software patches. |
Choose AOBench when the question is specifically "can this agent operate an HPC system correctly, safely, and within its role?" For general reasoning, web-browsing, or code-repair agents, use the corresponding general-purpose benchmark above.
- Not a live-cluster test. AOBench runs against mock tools and deterministic snapshots by design; it does not execute against real production HPC infrastructure and does not measure real-world side effects.
- HPC-scoped. The corpus covers 5 roles, 10 QCATs, 88 tasks, and 29 environments. It is not a general-purpose reasoning, web, or coding benchmark.
- API keys required for hosted models. The
openaiandanthropicadapters need the corresponding API keys and incur provider cost; thedirect_qabaseline runs without tools for reference. - Early-stage (v0.x). Scope, schemas, and scoring profiles are still evolving between minor versions.
What is AOBench? AOBench (Agent Operations Benchmark) is an open-source Python framework for evaluating AI agents that operate HPC systems. It scores agents on HPC operational tasks against deterministic environment snapshots with mock HPC tools.
How does AOBench evaluate HPC agents? Each task runs an agent (via an adapter) against a deterministic snapshot with mock SLURM, telemetry, docs, RBAC, and facility tools. AOBench records the full execution trace and scores it across six dimensions — outcome correctness, tool-use correctness, governance/RBAC, grounding, robustness (pass^k), and efficiency — then aggregates results into a CLEAR scorecard (Efficacy, Assurance, Reliability, Cost, Latency).
How is AOBench different from general LLM-agent benchmarks? General benchmarks measure broad assistant, reasoning, tool-use, or software-engineering ability. AOBench is domain-specific to HPC operations, enforces role-based access control (permission violations hard-fail), and scores the whole trace rather than only the final answer.
Do I need a live HPC cluster to run AOBench? No. AOBench runs entirely against deterministic snapshot bundles and mock tools, so it is portable and safe to publish.
Which agents and models can I evaluate?
Any model reachable through the openai, anthropic, or mcp adapters, plus a
tool-free direct_qa baseline for reference.
The full documentation site — searchable, versioned, and always built from
main. Everything below is also on it, rendered better.
Start here
| Page | What it gives you | |
|---|---|---|
| 🚀 | Quickstart | Clean machine → first scored task in five minutes |
| ⚙️ | Installation | Package, Docker image, Compose stack, extras |
| 🤖 | Evaluate your own agent | Plug your system in behind an adapter |
| 💻 | CLI reference | Every command and flag |
Understand the benchmark
| Page | What it gives you | |
|---|---|---|
| 🧭 | Framework overview | Principles and scope |
| 🏗️ | System architecture | Components, data flow, scoring pipeline |
| 📐 | Scoring dimensions | What each of the 7 weighted dimensions measures |
| 🗂️ | Task catalog · Environment catalog | Generated inventories of all 88 tasks and 29 environments |
| 🧪 | Environments · M100 ExaData | Snapshot format, and the real Marconi100 bundles |
For researchers
| Page | What it gives you | |
|---|---|---|
| 📋 | Datasheet · Benchmark card | Provenance, composition, intended and out-of-scope use |
| ⚖️ | Limitations · Comparison | What AOBench cannot measure, and how it differs |
| 🔁 | Reproducing results · Versioning | What is pinned, and when scores are comparable |
| 📚 | Cite AOBench · Related work | BibTeX and the verified bibliography |
In this repository
ROADMAP · CHANGELOG · CONTRIBUTING · GOVERNANCE · SECURITY · Developer guide
Source: github.com/MSKazemi/aobench · Mirror: gitlab.com/mskazemi/aobench
The benchmark/qa/ directory embeds the AOBench-QA dataset — ~95 HPC
operational queries with role-specific variants and structured taxonomies. It
is consumed by the direct_qa baseline and seeds task design for the v1 HPC
task set.
Contributions are welcome, and the project is set up so you can start without asking permission first.
You do not need a cluster, a GPU, or an API key. The whole benchmark runs against
frozen snapshots on a laptop, and the direct_qa adapter needs no model provider.
The two most valuable contributions are data, not code:
- 📝 Write a task — 26 of the 50 QCAT × role cells have only one task, so the benchmark cannot tell "understands the category" from "got lucky". One JSON file against an existing snapshot. Guide: adding a task.
- 📊 Run a model and submit the numbers — independent results are what make a benchmark credible rather than a claim. Negative results are more useful than good ones.
Other places to start, easiest first:
| If you want to… | Start here |
|---|---|
| Fix something small and well-specified | Good first issues — each one names the files to touch, the tests to write, and an honest time estimate |
| See the whole map | #20 — Start here: where to contribute |
| Report a bug or ask for a feature | Open an issue |
| Ask a question or propose an idea | Discussions — questions are welcome and expected |
| Add a task or an environment | CONTRIBUTING.md § How to Add a Task |
| Point a coding agent at this repo | AGENTS.md — architecture, commands, invariants |
| Report a security issue | SECURITY.md — please don't open a public issue |
From clone to green tests in three commands:
git clone https://github.com/MSKazemi/aobench && cd aobench
make install # creates .venv and installs everything
make test # ~1510 tests should passWhat you can expect from us: a first response within 3 working days — even if that response is just "seen, I'll look properly on Friday". If a PR of yours goes quiet for over a week, ping it; that's our failure, not rudeness on your part.
What helps us: PRs under ~300 changed lines. Bug fixes, docs, tests, examples and new CLI flags need no prior discussion — just send them. Open an issue first only if you're changing the task schema, the scoring weights, the RBAC model, or a public CLI signature.
Every merged contribution earns a line in AUTHORS.md and a place on the contributor wall, whatever its size — including reviews, docs, and corpus work. AI assistance is welcome; see CONTRIBUTING.md § Using AI assistance.
You do not need HPC access or a cluster to contribute. The whole benchmark runs
against frozen snapshots on a laptop, and the direct_qa adapter needs no API key.
See CONTRIBUTING.md for the full guide and CODE_OF_CONDUCT.md for community expectations.
If you use AOBench in your research, please cite it — and please cite the version you actually ran, since the task corpus and scoring profiles change between minor versions.
- How to cite — BibTeX, the four fields to report alongside any score, and when to also cite the ExaData dataset.
- Reproducing results — what AOBench pins, what it cannot pin, and how to re-derive a published number.
Machine-readable metadata is provided in three formats, all kept in sync:
CITATION.cff (GitHub renders a "Cite this repository" button from it),
codemeta.json (CodeMeta 3.0, for research-software registries), and
.zenodo.json (archival deposition metadata).
Seyedkazemi Ardebili, Mohsen. AOBench: A Trace-Driven, Role-Aware Benchmark for
Agent Operations in Realistic Environments. https://github.com/MSKazemi/aobench
The source is mirrored at https://gitlab.com/mskazemi/aobench.
- KubeIntellect — a human-governed AI SRE for Kubernetes. Investigates with real tools, then pauses for your approval before it changes anything. Peer-reviewed, Journal of Grid Computing (2026).
- YazSes — offline voice dictation for Linux, macOS and Windows. Hold a key, speak, release; speech-to-text runs on your own CPU and nothing is sent to a server. Apache-2.0, and good first issues are tagged and waiting.
Apache 2.0 — see LICENSE.