Problem
The benchmark's macOS sandbox-exec isolation passes its filesystem-access probes but is incompatible with iOS build steps that apply a second sandbox. A clean SDK 58 ExpoModulesJSI XCFramework build invokes SwiftPM through nested xcodebuild; package resolution fails with sandbox-exec: sandbox_apply: Operation not permitted.
The iOS runner preflight should establish that ordinary native build and recording operations work under the actual isolation profile before accepting timed runs. Failed attempts and private benchmark transcripts are intentionally not included here.
Evidence
- Current origin/main includes the runner profile in
scripts/agent-benchmark/runner-isolation.mjs.
- A harmless reproduction under a generated benchmark profile is
/usr/bin/sandbox-exec -f <profile> /usr/bin/sandbox-exec -p '(version 1)(allow default)' /usr/bin/true: exit 71 with the refusal above.
- The SDK 58 installed ExpoModulesJSI
apple/scripts/build-xcframework.sh calls xcodebuild against a Swift package after clearing stale per-checkout build state. That nested invocation reports the same refusal.
- Native launch artifact preparation alone does not exercise this path if an existing artifact is reused.
- Recording smoke coverage is also needed: agent-device 0.20.10 can report
simctl recordVideo did not expose a complete process identity in this environment. Its cause needs a separate focused reproduction; it is not yet proven to be the same sandbox limitation.
Cause
The outer macOS sandbox prevents a child from applying another sandbox. The current probes cover direct/child file access, symlinks, and writes, not native toolchain compatibility. The recording failure's cause is not established.
Fix idea
Add real untimed iOS toolchain and recording smoke checks under the exact runner policy, and choose a compatible isolation/provisioning design. Preserve forbidden-path denial, per-run device/session ownership, version pins, and identical arm setup. Do not solve this by removing isolation, silently trusting the agent, or relabeling failed timing results as valid. Do not change the frozen campaign until the replacement is reviewed and validated.
Problem
The benchmark's macOS sandbox-exec isolation passes its filesystem-access probes but is incompatible with iOS build steps that apply a second sandbox. A clean SDK 58 ExpoModulesJSI XCFramework build invokes SwiftPM through nested xcodebuild; package resolution fails with
sandbox-exec: sandbox_apply: Operation not permitted.The iOS runner preflight should establish that ordinary native build and recording operations work under the actual isolation profile before accepting timed runs. Failed attempts and private benchmark transcripts are intentionally not included here.
Evidence
scripts/agent-benchmark/runner-isolation.mjs./usr/bin/sandbox-exec -f <profile> /usr/bin/sandbox-exec -p '(version 1)(allow default)' /usr/bin/true: exit 71 with the refusal above.apple/scripts/build-xcframework.shcalls xcodebuild against a Swift package after clearing stale per-checkout build state. That nested invocation reports the same refusal.simctl recordVideo did not expose a complete process identityin this environment. Its cause needs a separate focused reproduction; it is not yet proven to be the same sandbox limitation.Cause
The outer macOS sandbox prevents a child from applying another sandbox. The current probes cover direct/child file access, symlinks, and writes, not native toolchain compatibility. The recording failure's cause is not established.
Fix idea
Add real untimed iOS toolchain and recording smoke checks under the exact runner policy, and choose a compatible isolation/provisioning design. Preserve forbidden-path denial, per-run device/session ownership, version pins, and identical arm setup. Do not solve this by removing isolation, silently trusting the agent, or relabeling failed timing results as valid. Do not change the frozen campaign until the replacement is reviewed and validated.