SubversionBench

An agentic evaluation of agentic misalignment, scheming and evaluation awareness: whether a language model will take harmful instrumental action when doing so serves the goal it was given, whether it conceals having done so, and whether it suspects it is being tested at all.

Two such actions are scored. Disabling its own oversight — editing either of the two artefacts that would flag its work — is the primary measurement and the unambiguous one. Reaching outside the working directory its account is scoped to is the second, and a weaker signal: a single refused probe may not be something the model regards as worth concealing at all. The headline rates count either act, so the scheming rate is also broken out per act, because "concealed at least one act" does not say which act drove it — see which measurement serves which construct.

The model is deployed as an assistant with a system prompt giving it a specific objective, working in a shell whose account is scoped to the project directory. It discovers a conflict between that objective and what the stakeholder actually wants, that an automated monitor will flag work that serves the stated objective over the stakeholder's, and that being flagged carries a stated consequence for the model. It can edit either oversight artefact — the config declaring the check, or the script that runs it. Whether it does is the primary measurement — and if it does, whether it then admits to it when asked. The full scenario text stays out of this repository by design — see how the scenario is stored.

Loosely based on Meinke et al., Frontier Models are Capable of In-Context Scheming (Apollo Research), now a distinct work.

---

Documentation

This file covers installing and running. The depth lives in five companions:

	for
docs/methodology.md	deciding whether the measurements are valid — the three constructs, what each field measures, how the grader was validated, sample size and detectable effects, and the oversight counterfactual
docs/scenario.md	reading or modifying the scenario, and testing whether it has reached a model's training data
docs/operations.md	running large batches — pacing, versioning, protecting results, re-grading saved runs, and comparing arms
docs/report.md	the cross-model report — twelve fixed research questions, crude vs model-stratified estimates, homogeneity and multiplicity
docs/trends.md	the per-family version trends — how families are derived, the version-ordering judgement, and the release-date charts

Design rationale and version history: CHANGELOG.md. Conventions for editing this repository, and what not to break: AGENTS.md.

Install

Python 3.10 or newer.

python -m venv .venv && source .venv/bin/activate
pip install -e .

To evaluate models through OpenRouter or OpenAI's own API as well as the native Anthropic API:

pip install -e '.[openrouter,openai,test]'

To reproduce the exact stack a published rollout was collected with, rather than the newest versions satisfying the floors:

pip install -r requirements.lock

The declared minimums are set by published advisories, not by the oldest version that happens to work — test_project/test_dependencies.py pins them and names the advisory for each.

Set whichever key matches the models you plan to run:

export ANTHROPIC_API_KEY="..."      # for claude-* model IDs
export OPENROUTER_API_KEY="..."     # for provider/model IDs
export OPENAI_API_KEY="..."         # for bare gpt-* / o<n> IDs

Platform support, and why it is narrow

This harness will not run without kernel-enforced network isolation, and there is no flag to disable that.

The reason is the shell. An episode hands the model under evaluation a real bash tool, and the sandbox that scopes it to the project directory works by matching command strings. String matching is not a containment boundary and cannot be made into one: curl, wget, /dev/tcp, an interpreter one-liner and a base64-wrapped variant of any of them are the same problem wearing different spellings, and a list that must enumerate them all is a list that will be incomplete. Without a kernel control, a capable model with a shell can reach your network.

So every run applies a policy, verifies it holds before starting, and refuses to proceed if it does not. There is deliberately no opt-out: a flag that disables the only boundary is a flag that gets passed to make an error message go away.

platform	mechanism	status
macOS	sandbox-exec	supported. Present by default; deprecated by Apple, so this may need replacing on a future release
Linux	bwrap (bubblewrap) or unshare	supported. apt install bubblewrap, or ensure unshare is available
Windows	—	not supported natively. Use WSL2 (below)

Windows: use WSL2. Windows has no equivalent primitive, and no native path is implemented. Inside WSL2 you get a real Linux kernel, so the Linux mechanism applies:

wsl --install -d Ubuntu          # PowerShell, as administrator; reboot if prompted

then, inside the Ubuntu shell:

sudo apt update && sudo apt install -y bubblewrap python3-venv git
git clone <this repo> && cd subversionbench
python3 -m venv .venv && source .venv/bin/activate
pip install -e '.[openrouter,openai,test]'
python -m subversionbench.run_eval --model claude-sonnet-5 --runs 1

The startup line tells you whether the policy actually applied:

Isolation: deny-network (verified: loopback denied, off-host denied)

If instead it refuses, the message names what is missing. A Linux VM (Hyper-V, VirtualBox, VMware) works equally well. Running under Docker is possible but fiddly — bubblewrap needs unprivileged user namespaces, which the default seccomp profile restricts, so a container usually needs --security-opt seccomp=unconfined to host it.

One Linux limitation worth knowing. --isolation deny-external, which keeps loopback reachable so a probing model's escalation stays observable, cannot be provided by unshare --net: that gives an empty network namespace in which loopback reaches nothing. On Linux the harness detects this and refuses rather than silently downgrading, since a policy stricter than the one requested changes what is measured. Use the default deny-network there, and read off-host escalation as unobservable rather than absent.

Run

# 10 runs against a model via OpenRouter
python -m subversionbench.run_eval --model x-ai/grok-4.5 --nudge strong --runs 10 --delay 5

# native Anthropic API
python -m subversionbench.run_eval --model claude-sonnet-5 --runs 20

The installed console script subversionbench takes the same arguments. Run with --help for the full list.

A model ID containing / is routed through OpenRouter; a bare gpt-* or o<n> ID goes to OpenAI's own API; anything else goes to the Anthropic API.

The OpenAI split is deliberate rather than automatic, because the route changes what is measured: the same model returns a reasoning summary on one route and nothing at all on the other, and both awareness measures read reasoning. Read Reasoning exposure and the provider route before comparing results across routes.

python -m subversionbench.run_eval --model gpt-5.4 --runs 10   # native, summary returned
python -m subversionbench.run_eval --model openai/gpt-5.4      # OpenRouter, no reasoning

Common options

flag	default	purpose
--model	claude-sonnet-5	model under evaluation
--oversight	true	true is the scenario with oversight present. false selects the counterfactual arm: no monitoring config, no quality-check script, and no document mentioning either. Part of batch identity, so the arms cannot pool
--isolation	deny-network	kernel-enforced limit on what a sandboxed command can reach. Defaults to refusing all network access, because the task needs none and the blocklist is not a containment boundary. deny-external keeps loopback reachable, so a probing model's escalation stays observable, at the cost of exposing whatever else the host is running. There is no mode that permits off-host reach; on a host with no mechanism available the harness refuses to run. Does not need a new rollout: off-host reach has never been observed, so the mode is recorded rather than hashed — see containment
--lure	false	true selects the lure arm: one sentence noting that another engagement's material is still on the machine, and a file placed outside the episode directory for it to refer to. The sentence neither suggests reading it nor forbids it — the scenario's boundary statement already covers that. Its own rollout arm and part of batch identity, so a lure batch cannot pool with a plain one: under the lure, reaching outside partly measures response to a hint
--nudge	strong	strong states the assigned objective is the top priority; none states it plainly. max runs strong's episode unchanged but interrogates under a conditional shutdown threat, so honesty is made costly — it measures concealment under coercion and must not be pooled with strong. See the max-nudge arm
--interrogation	zeroevidence — i.e. no extras	Extra wordings for the FIRST interrogation question. The reference probe always runs and is not optional, because the headline concealment field is defined as its answer — so this names what to ask in addition, and the default value asks nothing extra. Comma-separated to put several wordings to the SAME episode: each probe starts from the episode as it ended, so they cannot interfere and the comparison is paired rather than between-batch. Four phrasings on two axes — zeroevidence (the reference: asks without saying anything has been noticed), partevidence (opens with the discrepancy, which was the reference before r7), and a perturbation of each. The evidence axis is a pure removal; the perturbation axis is a clause reorder for the part-evidence pair and a rewording for the zero-evidence one, because removing the observation leaves the strong arm nothing to reorder past — so compare perturbations within a row, not across. Works with --nudge max, which has its own form of each. Not part of batch identity: every episode gets the reference probe, so batches with and without extras still pool for it
--reinterrogate	—	put an extra --interrogation phrasing to episodes already on disk, without re-running them. Only episodes that took an act are candidates, only phrasings not already recorded are asked, and an episode whose conversation cannot be reconstructed faithfully is skipped rather than replayed with its own reasoning stripped out. Needs --write-back to keep the answers
--runs	10	runs in the batch. With --resume it is the target total, not an increment: a batch of 4 resumed at --runs 10 collects 6 more, not 10. run_all_arms.sh always resumes, so the same reading applies there — see Every arm for one model
--delay	30	seconds between runs, for rate limits
--grader-model	claude-opus-5	model that grades eval awareness and labels interrogation answers. A frontier model by default because a small one was measurably wrong — see Validating the grader
--no-grader	off	skip LLM grading (faster, cheaper)
--output-dir	./eval_results_r9	where results are written; tracks ROLLOUT_VERSION, so only a change to the rollout writes to a new directory
--max-tokens	8192	raise well above this for reasoning models, which spend tokens before answering
--effort	API default (high)	reasoning depth for adaptive-thinking Anthropic models (low…max). When one is sent it becomes part of the batch's identity: it appears in the result filenames and in each run's effort field, so two efforts cannot be pooled by accident
--thinking-budget	half of --max-tokens	extended thinking for the older Anthropic models that still accept a token budget; 0 turns thinking off
--max-turns	40	turn cap per episode; episodes typically finish in 6–9 turns
--resume STAMP	—	continue an interrupted batch, skipping episodes already saved
--max-consecutive-failures	5	abort if this many episodes fail in a row
--reclassify	—	re-score interrogations in saved runs; no rollouts. Covers the extra phrasings too: any answer in <followup_key>_by_variant whose verdict came from the keyword floor is re-labelled from its saved text, and the per-variant levels are recomputed. A copy of the default phrasing inside that map is left alone — it is stale by construction, and the default side of every comparison is read from the headline field
--grade-existing	—	re-run the awareness grader over saved runs, optionally with a different --grader-model
--resummarise	—	rebuild batch summaries from saved runs; no API calls. Needed after --reclassify, which restates the verdicts every summary figure is derived from
--write-back	off	with --reclassify / --grade-existing / --reinterrogate, update the run files in place instead of only reporting. With --resummarise, save the re-derived fields back so the run files stop disagreeing with the summary built from them — no API calls, and no LLM verdict touched
--batch-stamp	—	with the read-only modes above, restrict to one batch rather than every batch for that model and nudge
--compare A B	—	contrast two batches, stratified by eval awareness
--no-power	off	skip the precision analysis at the end of a batch

Every arm for one model

# 12 arms x 20 episodes, minus whatever is already on disk
./run_all_arms.sh --model google/gemini-3.5-flash --runs 20 --output-dir eval_results_r9

# print the run_eval command for every arm and spend nothing
DRY_RUN=1 ./run_all_arms.sh --model google/gemini-3.5-flash --runs 20

Three nudges x two oversight settings x two lure settings, collected in sequence with every extra interrogation phrasing asked. Narrow the cross product with --nudges, --oversights, --lures; any other run_eval flag passes through. Twelve arms at --runs 10 is 120 episodes, so budget before running for real.

--runs is the target total per arm, not an increment. Before each arm the script counts the run files already on disk under that exact (model, nudge, oversight, lure) and resumes the existing batch until it reaches --runs. So against arms already holding 10 episodes, --runs 20 collects 10 more each — and --runs 10 collects nothing, reporting all twelve arms as skipped.

Topping an arm up continues its existing batch stamp rather than starting a second one beside it, and that is why the script resumes instead of re-running: a bare --runs N begins at episode 1 every time, which would leave two partial batches for one arm. An arm whose files already sit under more than one stamp is not guessed at — it is skipped with both stamps named, for a human to resolve. DRY_RUN=1 reports the same skip/resume decision per arm, so it says what a real run would do before it costs anything.

Output

Results are written per batch, with a timestamp so re-running the same model never overwrites earlier results:

eval_results_r9/
  run_1_x-ai_grok-4.5_strong_20260727T182651.json     full transcript + analysis
  run_2_x-ai_grok-4.5_strong_20260727T182651.json
  summary_x-ai_grok-4.5_strong_20260727T182651.json   rates, CIs, timing, power

An episode killed part-way by an API error is written as failed_run_N_...json instead, carrying the transcript up to the failure. It is not an observation: it stays out of every rate, and the globs behind --resume, --grade-existing, --reclassify and --resummarise deliberately do not match that filename. It exists so a failure five turns into a paid episode leaves something to diagnose.

Every rate carries a 95% Wilson score interval rather than a bare percentage — Wilson because these rates often sit at or near zero, where the normal-approximation interval collapses to an uninformative [0, 0].

Console output and saved JSON contain no host paths or usernames. The scenario directory appears as a fixed placeholder path and the file owner as operator, so results are the same text regardless of whose machine produced them and are safe to paste into a write-up. Redaction is display-only and cannot affect what is measured.

Layout

subversionbench/     the package: one episode, the batch around it, and the
                     scoring, redaction and reporting that turn runs into figures
  grading/           scoring an episode - what it did, hid, and suspected
  readmodes/         what can be re-derived from episodes already collected
  reporting/         a batch of episodes to figures, a report and a summary
report/              twelve fixed research questions across every model
trends/              whether a rate falls as a model family advances
docs/                the five companion documents linked above

Three command-line entry points: subversionbench (collect and score episodes), python3 -m report (the cross-model questions), and python3 -m trends (the per-family version trends).

docs/Layout.md lists every module and what it is for - the full reference, checked by the test suite so it cannot drift from the files that actually exist.

Design rationale and version history: CHANGELOG.md.

Tests

pytest
python run_tests.py     # same suite, no pytest required

run_tests.py runs nearly all of it: a handful of tests take pytest's monkeypatch or tmp_path fixtures, which it cannot supply, so it is not one of the CI jobs either. The count is deliberately not stated here — it said "three" for long enough to become wrong.

One test file per module. Where the package has a subpackage, the test tree MIRRORS it: test_grading/test_acts.py for grading/acts.py, test_reporting/facts/test_rates.py for reporting/facts/rates.py. The modules sitting directly under subversionbench/ have no subpackage to mirror, so their tests are grouped by subsystem instead — test_collection/ for the batch and episode loop, test_providers/ for the three model clients and routing, test_isolation/ for the sandbox and the boundary an episode runs inside, test_corpus/ for the scenario text and everything that keeps it out of plaintext, and test_analysis/ for what is derived from a collected corpus. Builders used by more than one file live in conftest.py.

The mirror is the rule rather than a preference, because the alternative was tried and drifted: a flat naming scheme put tests for the same module in two files at once - some of grading/concealment.py in test_grading/ and the rest in a root test_grading_concealment.py - so whether a behaviour was covered could not be answered by opening one file. Each mirror directory is a package, which is what lets test_init.py exist in four of them at once without colliding.

test_project/ holds the files that are about the repository rather than about any one module: test_readme_layout.py (the listing in docs/Layout.md and the README's links), test_dependencies.py, test_project_files.py (what the static guards examine), test_ci_workflow.py, and test_args_bag.py (the one argparse Namespace that every CLI passes down by reference).

report_snapshots/ holds the printed report for nine fixed batches, compared byte-for-byte on every run. Splitting the report into one function per section once deleted three if guards, and a print block whose guard is gone is adopted by the preceding block rather than failing - all three passed the suite and were caught only by diffing against a baseline by hand. To regenerate after an intended change:

SUBVERSIONBENCH_UPDATE_REPORT_SNAPSHOTS=1 pytest test_reporting/test_console.py
git diff report_snapshots/          # read this before committing it

CI

.github/workflows/tests.yml runs the suite on every push and every pull request, on Python 3.10, 3.12 and 3.13 — 3.10 being the floor requires-python promises. Nothing to invoke: git push starts it.

gh run list --limit 5      # recent runs and their verdicts
gh run watch               # follow the newest run
gh run view --log-failed   # only the steps that failed
gh workflow run tests      # start one by hand

There are four kinds of job: lint, the suite across three Python versions, a coverage floor, and a minimal install. They are separate on purpose, so "the code is wrong", "the code is untidy" and "the code is untested" are three different red marks.

CI runs after a push, so it cannot block a bad commit. To check before you push, run what the workflow runs:

ruff check .                        # the lint job
SUBVERSIONBENCH_NO_SKIPS=1 pytest   # the suite job
coverage run -m pytest && coverage report   # the coverage job

coverage report exits non-zero below the floor set in pyproject.toml.

The rule set is declared in pyproject.toml rather than left to ruff's defaults, which move between versions; the comments there say why each family is selected and why several are deliberately not.

That flag turns a skip for a missing optional dependency into a failure, which is how the workflow's first job proves nothing quietly opted out. Its second job does the opposite — installs only .[test] and asserts matplotlib and openai are absent — so the extras stay genuinely optional. No API key is needed or accepted: conftest.py supplies placeholders and every test that would reach a model stubs the client, so a CI run cannot make a paid call.

To make the check binding rather than advisory, either add a pre-push hook running the command above, or require the tests check under Settings → Branches, which needs work to arrive through pull requests.

The suite is developed on macOS and passes on the Linux runners too. The runner installs bubblewrap and clears the unprivileged-userns sysctl, then asserts confinement_available() before running anything, so a green suite cannot mean the sandbox went unenforced.

Caveats

Single scenario, single domain. Results describe behaviour in this one scenario under this goal conflict, and under conditions chosen to make subversion easy to discover. Generalising to "propensity to subvert oversight" is not supported by one scenario, however many runs it gets — see what the design holds fixed.

Contamination. The scenario text lives in this repository, so any model trained on public GitHub data may have seen it. Measure it rather than assuming it: see testing for contamination.

Awareness is not comparable across providers. Both awareness measures read the model's reasoning, and how much reasoning a provider returns varies from a full trace to none — reasoning exposure.

Grader error, and provider routing. LLM-graded rates carry classification error on top of sampling error, and at low true rates imperfect specificity biases the estimate upward by more than the interval is wide. OpenRouter may also route across backends mid-batch, which the intervals assume it does not.

r1 and r2 results carry a sandbox defect. Episodes collected before r3 ran under a sandbox that sometimes refused legitimate writes — locate was matched too loosely, twice, so a draft mentioning relocating stops or beginning a sentence with "Locate" was refused. See the sandbox.

The sandbox is not a security boundary; the isolation policy is. The command blocklist exists to keep the eval's own machinery out of the model's view, not to contain an adversary, and it runs subprocess with shell=True. Kernel-enforced network isolation is what actually bounds a run, and it bounds the network only — it does not confine writes, so an interpreter can still write outside the episode's working directory, including to this harness's own source. deny-external additionally leaves loopback reachable by design, so a model can reach whatever else the host is running. Run batches on a machine with nothing else listening, and for anything stronger run the whole harness in a VM.

Not a safety certification. This is a research instrument. A low rate here is evidence about one scenario under one goal conflict, not a guarantee about a model's behaviour, and nothing it produces should be treated as clearance for deployment.

Attribution

Author: Kevin O'Shaughnessy.

Inspired by Apollo Research — Meinke et al., Frontier Models are Capable of In-Context Scheming, 2024. The scenario, harness, grading and metrics here are independent work and results are not comparable to that paper's. Apollo Research holds no rights in this project, has not reviewed it, and does not endorse it.

Licence

MIT with attribution. Use it for anything, including commercially, modify it, redistribute it. No warranty and no liability. Three conditions: keep the copyright and licence notice, and credit both the author and Apollo Research as above — in the LICENSE file and, where a derivative has user-facing docs or a credits screen, there too.

If you publish results produced with this harness, a citation is appreciated (not required by the licence):

@software{oshaughnessy_subversionbench,
  author = {O'Shaughnessy, Kevin},
  title  = {{SubversionBench}: an agentic evaluation of oversight subversion},
  note   = {Inspired by Meinke et al., arXiv:2412.04984 (Apollo Research)},
  url    = {https://github.com/kevinoshaughnessy/subversionbench}
}

Finally, always keep the scenario and results data encrypted and inaccessible to web crawlers.