Skip to content

fix: isolate benchmark runners from coordinator data - #466

Merged
janicduplessis merged 2 commits into
mainfrom
fix/benchmark-isolation-460
Sep 7, 2026
Merged

fix: isolate benchmark runners from coordinator data#466
janicduplessis merged 2 commits into
mainfrom
fix/benchmark-isolation-460

Conversation

@janicduplessis

@janicduplessis janicduplessis commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Description

Prompt-only scope rules leave benchmark agents able to read coordinator configuration and sibling results. Enforce that boundary for both runners instead of discovering contamination after app proof succeeds.

Solution

Launch Codex and Claude under the same macOS filesystem policy: protected file contents and writes are denied while preserving access to the current run, required tools and shared state. Parent-directory listings remain allowed. Configured tool/cache paths cannot overlap private benchmark directories. Real read/write, child-process and symlink probes run before dispatch; a failed check prevents timing from starting. Preserve the verified policy digest with each run.

This prevents accidental access through the runner process tree, not malicious access through already-running native services. It relies on macOS's deprecated sandbox-exec launcher and fails closed when unavailable. Historical audit verdicts and benchmark execution rules are unchanged.

Test plan

  • Real macOS tests deny coordinator/golden/sibling reads and writes, including symlink aliases and children; allow current-run writes, read-only tool access, and creation of the exact future Git worktree.
  • Confirm broad or narrow private-directory cache grants, including symlink aliases, and modified verified policies refuse launch.
  • Untimed Stim/control smoke checks pass with the installed Codex skill profile, and the installed Claude executable starts under both policies. No model benchmark was rerun.

Fixes #460

@janicduplessis

Copy link
Copy Markdown
Collaborator Author

Independent code review is clear after rejecting configured grants that overlap private directories, including narrow descendants and aliases. Verified 3,617 tests, 66 e2e tests, all repository checks, real macOS sandbox and Git worktree tests, plus untimed Stim/control runner-profile preflights. No model benchmark was rerun.

@janicduplessis
janicduplessis marked this pull request as ready for review September 7, 2026 00:45
@janicduplessis
janicduplessis merged commit 5751260 into main Sep 7, 2026
5 checks passed
@janicduplessis
janicduplessis deleted the fix/benchmark-isolation-460 branch September 7, 2026 00:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Audit benchmark reads outside the allowed run scope

1 participant