fix: isolate benchmark runners from coordinator data - #466
Merged
Conversation
Collaborator
Author
|
Independent code review is clear after rejecting configured grants that overlap private directories, including narrow descendants and aliases. Verified 3,617 tests, 66 e2e tests, all repository checks, real macOS sandbox and Git worktree tests, plus untimed Stim/control runner-profile preflights. No model benchmark was rerun. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Prompt-only scope rules leave benchmark agents able to read coordinator configuration and sibling results. Enforce that boundary for both runners instead of discovering contamination after app proof succeeds.
Solution
Launch Codex and Claude under the same macOS filesystem policy: protected file contents and writes are denied while preserving access to the current run, required tools and shared state. Parent-directory listings remain allowed. Configured tool/cache paths cannot overlap private benchmark directories. Real read/write, child-process and symlink probes run before dispatch; a failed check prevents timing from starting. Preserve the verified policy digest with each run.
This prevents accidental access through the runner process tree, not malicious access through already-running native services. It relies on macOS's deprecated
sandbox-execlauncher and fails closed when unavailable. Historical audit verdicts and benchmark execution rules are unchanged.Test plan
Fixes #460