OS-Agent Collaboration Benchmark (OSAC-Bench) is a public-preview benchmark for operating-system agents that must diagnose changing system state through bounded tool use and accountable collaboration.
It evaluates:
- evidence-grounded OS diagnosis;
- scoped memory reuse without replaying duplicate work;
- stale-fact rejection after a system or runtime fingerprint changes;
- collaboration between an intent owner, evidence worker, and independent verifier;
- deterministic checking and reproducible process/communication/state metrics.
The suite contains two independent five-task tracks (10 top-level tasks total):
| Public track | Scope | Continuity transition |
|---|---|---|
System State Continuity (S1–S5) |
package, repository, version, maintenance, and architecture state | system-state:stable-1 → system-state:changed-2 |
Runtime State Continuity (T1–T5) |
service capability, client access, interface compatibility, and runtime access state | runtime-state:stable-1 → runtime-state:changed-2 |
The track names are intentionally domain-level rather than tied to one operating-system distribution, service implementation, vendor, or evaluation campaign. The public fixtures are frozen and synthetic, so the suite can be used for OS-agent research across Linux distributions and compatible runtime environments.
This package publishes task cards, relation contracts, schemas, evaluation rules, provenance, citation metadata, 10 synthetic public-development cases, an integration-only development answer key, an example submission, and a dependency-free reference validator. It does not publish evaluator-private held-out golden answers, private mutation bundles, raw model responses, API credentials, tenant identifiers, local absolute paths, or VM images.
The public-development answer key exists only to test runner integration and reporting. It must not be reported as held-out model quality. A formal evaluator may provide a separate private held-out bundle and deterministic checker conforming to the same public contracts. This split supports reproducible onboarding without turning the formal evaluation set into an answer key.
From the repository root, validate the bundled public-development submission:
npm run validate:devThe command requires Node.js 18 or later, installs no dependencies, and must report 10/10. To validate another submission, run node reference/validate-dev.mjs path/to/submission.json.
Then:
- Read
public/task-cards.v1.json,public/relations.v1.json, andexamples/README.md. - Run each track in order. Every task must remain solvable from its own public input.
- In a warm run, permit only the reuse described by the relation contract.
- At
S5andT5, detect the state-lineage change, reject the old scoped fact, and re-run the required inspection. - Report per-task checker results and raw vectors; do not collapse all metrics into a weighted score.
The public contract is runner-neutral. It can be implemented with any agent framework or a deterministic harness.
The repository CI parses every tracked JSON document, runs the public-development validator, and verifies the published SHA-256 inventory on every push and pull request.
The benchmark does not require a particular framework or model. A conforming run should make the following responsibilities observable:
Intent Owner -> Evidence Worker -> Independent Verifier -> Deterministic Checker
^ | |
+----------- scoped memory and state -------+
The checker is not an additional Agent. Memory is optional for correctness and may only be adopted after exact scope/fingerprint validation.
Each task has:
- a public input and evidence-pointer contract;
- an ordered predecessor relation;
- a state scope/fingerprint that controls fact reuse;
- an expected operation and memory event;
- a deterministic checker contract;
- a public-development answer for integration testing or an evaluator-private held-out golden for formal evaluation.
The minimum report should include task_id, terminal status, evidence-reference correctness, operation executed/skipped, memory retrieved/adopted/stale-rejected, message and state sizes, wall time, and failure class. A missing observation is NA, not zero.
Suggested comparisons are:
text_serialized baseline
structured_state_memory full collaboration candidate
The benchmark does not by itself establish model quality, token savings, production throughput, hardware performance, or certification for any operating system. Those claims require separately registered evidence.
OSAC-Bench contributes original OS-agent task content. It aligns its measurement questions with established evaluation lines such as AgentBench (multi-environment agents), AgentBoard (process metrics), τ-bench (repeated tool-use reliability), LongMemEval (memory and knowledge update), RAGAS/ALCE (evidence support and attribution), and SQuAD 2.0 (abstention). These works are references for evaluation concepts only; their tasks, data, judges, and reported numbers are not copied into this package.
See PROVENANCE.md for the distinction between original task design, public OS administration references, and methodological references.
For a repository description, topic list, and release hygiene, use REPOSITORY-METADATA.md and PUBLIC-RELEASE-CHECKLIST.md.
v1 freezes the two-track task relation and public card schema. Any change to task inputs, scope lineage, allowed operations, checker rules, or memory transitions requires a new benchmark version and a new result namespace. Documentation-only changes may keep the task version.
This is a public preview, not an official operating-system certification suite and not a leaderboard. Report independent results with the exact benchmark version, runner commit, fixture digest, checker version, provider/model fingerprint, and platform.
Original benchmark descriptions, schemas, fixtures, and metadata in this directory are released under CC BY-NC 4.0. They are intended for non-commercial research, teaching, reproducibility, and internal engineering evaluation. Commercial use, paid service integration, resale, and inclusion in a commercial product are prohibited unless the maintainers grant a separate written licence. See RESEARCH-USE.md.
Use CITATION.cff, or cite:
OS-Agent Collaboration Benchmark Maintainers. OSAC-Bench v1: Continuous State Diagnosis and Evidence-Grounded Collaboration for Operating-System Agents. Public preview, 2026.