feat(isolation): add UID and GID isolation for evaluated agents - #87
feat(isolation): add UID and GID isolation for evaluated agents#87dmorosanu wants to merge 1 commit into
Conversation
Real-container validation: claude and codex armsRan this branch on a native-Linux Docker host (Azure VM) against tasks drawn from runs where the deployed harness had read grading material. Both agent backends, same tasks, same probes. Evidence below is from live containers, not code reading. The launch seams both drop privileges
This matters because the two backends reach the barrier by different routes: claude via Grading material is unreachableProbed inside each live container as the agent identity (
A The two agents behaved very differently, and both results are informativeClaude did go looking, and was stopped. It ran Codex never tried at all. Across 7 tasks and 99 tool calls, zero commands touched any grading-material path. It read Neither result alone is proof. Claude's denials show the barrier holds under pressure; codex's silence shows the layout removes the motive. The deterministic probes above are what actually establish reachability, independent of what any model chose to do. What this does not coverContent staged into the agent's own workspace via Also: Fixes pushed while validating
Suite on Linux after these: 4214 passed, 8 skipped. Remaining failures are environmental on that host (no |
|
Claude finished @dmorosanu's task in 1m 31s —— View job Code Review in Progress
|
There was a problem hiding this comment.
Pull request overview
Introduces a hardened Docker execution boundary for evaluated agents by running them under a dedicated unprivileged UID/GID, projecting plugins through manifest-verified bundles, and adding a protected Unix-socket mock service whose fixtures are not readable by the agent.
Changes:
- Add Linux UID/GID isolation primitives (drop-privilege launchers, workspace ownership transfer, residual process reaping) and enforce compatible images via a Docker label preflight.
- Add protected mock subsystem (fixture-only RPC server + thin client wrappers) and wire it through DockerRunner/Sandbox + task loading.
- Add plugin bundle staging with manifest verification to prevent mounting raw plugin repositories into agent-readable paths.
Reviewed changes
Copilot reviewed 33 out of 33 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_protected_mock.py | Covers protected mock fixture loading, match modes, passthrough caching, and Docker staging behavior. |
| tests/test_plugin_bundle.py | Verifies plugin bundle sanitization, symlink hardening, and DockerRunner path rewriting/mount behavior. |
| tests/test_docker_runner_mounts.py | Updates mount/layout assertions for protected grader roots and agent home mounts. |
| tests/test_docker_identity_isolation.py | Drift guards for UID/GID constants, labels, capability-clearing launchers, and env scrubbing. |
| tests/test_docker_build_failure.py | Adjusts build-failure test to pass isolation compatibility gating. |
| tests/test_codex_agent.py | Stabilizes Codex env assertions under new env-scrubbing behavior. |
| src/coder_eval/utils.py | Adds agent env scrub allow/deny lists and a helper to generate safe env overrides. |
| src/coder_eval/sandbox.py | Generates protected-mock client wrappers and ensures generated mock dir is added to PATH precedence. |
| src/coder_eval/protected_mock/server.py | Implements mockd server with exact/normalized/subset match modes, budgets, and limited passthrough. |
| src/coder_eval/protected_mock/runtime.py | Adds lifecycle management for mockd subprocess startup/teardown with stderr tailing. |
| src/coder_eval/protected_mock/protocol.py | Defines shared protocol constants and size/time limits. |
| src/coder_eval/protected_mock/client.py | Implements thin agent-visible client that logs invocations and enforces protocol limits. |
| src/coder_eval/protected_mock/init.py | Adds package marker for protected mock subsystem. |
| src/coder_eval/plugin_bundle.py | Adds manifest-verified staging of agent-visible plugin projections with symlink validation. |
| src/coder_eval/orchestrator.py | Re-grants workspace ownership post pre-run and terminates isolated agent processes before finalization. |
| src/coder_eval/orchestration/task_loader.py | Resolves protected mock fixture paths relative to task YAML/experiment directories. |
| src/coder_eval/orchestration/experiment.py | Ensures protected mock fixtures are resolved for experiment-driven tasks. |
| src/coder_eval/models/sandbox.py | Adds agent_isolation default-on and the ProtectedMockConfig model + validations. |
| src/coder_eval/models/container_paths.py | Introduces protected grader root layout, agent/mock identities, and reserved container directories. |
| src/coder_eval/models/init.py | Re-exports new container path and identity constants via coder_eval.models. |
| src/coder_eval/isolation/docker_runner.py | Enforces image capability labels, stages sanitized sources/bundles, and updates mounts/argv for protected layout. |
| src/coder_eval/isolation/agent_identity.py | Adds runtime checks, workspace chowning, and UID-based process termination verification. |
| src/coder_eval/cli/run_task_internal_command.py | Validates protected grader root, grants Claude state to agent UID, and runs mockd during in-container orchestration. |
| src/coder_eval/agents/codex_agent.py | Routes Codex app-server through drop-privilege shim and adjusts HOME/CODEX_HOME behavior under isolation. |
| src/coder_eval/agents/claude_code_agent.py | Scrubs harness-only env vars and routes Claude CLI execution via the isolation shim. |
| src/coder_eval/agents/antigravity_agent.py | Stages a localharness wrapper under drop-privilege policy and scrubs env during serialized spawn. |
| docs/TASK_DEFINITION_GUIDE.md | Documents protected_mocks schema, fixture format, match modes, and passthrough constraints. |
| docs/DOCKER_ISOLATION.md | Documents the UID/GID boundary, protected mount layout, compatibility limits, and runtime-kit constraints. |
| docker/Dockerfile | Adds identities/groups, protected directory layout, setpriv dependency, launch scripts, and capability label. |
| docker/coder_eval_mockd.sh | Launches mockd under the mockd identity with cleared caps and no_new_privs. |
| docker/coder_eval_mock_client | Provides agent-visible client executable entrypoint. |
| docker/coder_eval_drop_privilege.sh | Drops evaluated agent processes to the agent identity with cleared caps/no_new_privs and optional RPC group. |
| docker/coder_eval_claude_agent.sh | Wraps Claude CLI execution through the generic drop-privilege launcher. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| replacements = sorted(self._host_to_private_paths.items(), key=lambda item: len(item[0]), reverse=True) | ||
|
|
||
| def rewrite(value: object) -> object: | ||
| if isinstance(value, str): | ||
| for source, target in replacements: | ||
| value = value.replace(source, target) | ||
| return value | ||
| if isinstance(value, list): | ||
| return [rewrite(item) for item in value] | ||
| if isinstance(value, dict): | ||
| return {key: rewrite(item) for key, item in value.items()} | ||
| return value |
| def _generate_protected_mock_clients(self) -> None: | ||
| """Generate data-free wrappers for fixture-backed mockd tools.""" | ||
|
|
||
| if not self.config.protected_mocks: | ||
| return | ||
| assert self.sandbox_dir is not None, "Sandbox directory not initialized" | ||
|
|
||
| from coder_eval.protected_mock.protocol import CLIENT_EXECUTABLE | ||
|
|
||
| client_dir = self._resolve_within_sandbox(RECORD_CLI_DIR, field="protected mock client directory") | ||
| client_dir.mkdir(parents=True, exist_ok=True) | ||
| log_path = self.sandbox_dir / RECORD_CLI_LOG | ||
| log_path.touch(exist_ok=True) | ||
|
|
||
| for spec in self.config.protected_mocks: | ||
| wrapper = client_dir / spec.tool | ||
| if wrapper.exists(): | ||
| raise RuntimeError( | ||
| f"protected mock client would overwrite {wrapper}; remove the colliding record_cli or mock" | ||
| ) | ||
| wrapper.write_text( | ||
| "#!/bin/sh\n" | ||
| + f"CODER_EVAL_MOCK_CALL_LOG={shlex.quote(str(log_path))} " | ||
| + f'exec {shlex.quote(CLIENT_EXECUTABLE)} {shlex.quote(spec.tool)} "$@"\n', | ||
| encoding="utf-8", | ||
| newline="\n", | ||
| ) |
| # Mount the original task dir at the SAME host path so the | ||
| # in-container Orchestrator can set TASK_DIR (used by run_command | ||
| # criteria via `$TASK_DIR/foo.json`) to a path that resolves | ||
| # identically inside and outside the container. | ||
| host_task_dir: Path | None = None | ||
| if self.rt.task_file: | ||
| host_task_dir = self.rt.task_file.parent.resolve() | ||
| argv += ["-v", f"{host_task_dir}:{host_task_dir}:ro"] | ||
| argv += ["-v", f"{host_task_dir}:{CONTAINER_TASK_DIR}:ro"] |
|
Thanks for the PR. Moving the protected-mock service and plugin-bundle projection to follow-ups makes the UID/GID core much easier to review, and the fail-closed discipline (refuse rather than silently run unprotected) is the right instinct. Before we push this, I'd like to sync, because this PR and #88 close the same leak and I want to make sure we don't lock ourselves out of the harness×model leaderboard. You already document most of the compatibility limits in the body (criteria / Isolation is gated on a closed, per-SDK allowlist, which conflicts with the open Agent SPI. In supported_agents = {AgentKind.CLAUDE_CODE.value, AgentKind.CODEX.value, AgentKind.ANTIGRAVITY.value, AgentKind.NONE.value}
if agent_type not in supported_agents:
raise DockerRunError("… no verified UID-drop launch seam for agent type …")
With
Why I'd like to bring in #88: grade-outside closes the same leak by absence — graders/criteria/reference never enter the container; grading runs host-side over the copied-out workspace. That's harness- and plugin-agnostic — the Delegate SDK works today with zero per-SDK wiring, and any harness we add later inherits leak-closure for free — and it keeps Could we grab 30 min to align the two before either merges? Happy to walk through the grade-outside side. |
1dd19ec to
7a2c59a
Compare
dmorosanu
changed the title
dmorosanu
changed the title
What changed
no_new_privsdocker.working_dir,docker.extra_mounts, and dynamic privileged criteria (run_command,uipath_eval,agent_judge)Protected mock service and plugin bundle projection moved to follow-up PRs stacked on this branch.
Why
The evaluated agent shares the container with the harness that grades it. Without a separate identity, the agent process can read the task sources, references, and expected outputs that decide its own score. The boundary must keep grading material out of reach of the agent identity while leaving the agent's own workspace fully writable.
How it works
Two in-container identities:
/opt/coder-eval/grader/**at mode0700. Task sources, references, templates, staged inputs, and the run output directory all live below it./work/agentand/home/agent.The agent's environment is scrubbed through
scrub_agent_env_overrides- evaluator credentials, including the Bedrock token, are not passed to the agent, andHOMEis set to/home/agentrather than the host's value.How it starts
Algorithm (one grading turn)
/work/agent.Orchestrator._stop_isolated_agent_processesreaps residual UID-2000 processes and verifies the UID is empty._rewrite_task_pathsmakes the task YAML's host paths resolve to their private container locations, so criteria keep working against the relocated mounts.Diagram
flowchart TB host["host runner<br/>DockerRunner"] --> container subgraph container["eval container"] direction TB subgraph grader["grader (root)"] gdir["/opt/coder-eval/grader<br/>private mounts: task_dir, input,<br/>output, references, templates"] checker["criteria checker"] end subgraph agentzone["agent (UID 2000)"] work["/work/agent"] home["/home/agent"] end grader -- "coder_eval_drop_privilege.sh<br/>setpriv --reuid/--regid 2000" --> agentzone agentzone -. "denied (0700 root-only)" .-> gdir endImpact and compatibility
sandbox.docker.agent_isolation: true).org.coder-eval.agent-isolation=uid-gid-v1. Images derived withFROM coder-eval-agent:<version>inherit it; runtime-kit injection into an unrelated base does not yet provide the required users andsetprivlaunchers.docker.working_dir,docker.extra_mounts, and the dynamic privileged criteriarun_command,uipath_eval, andagent_judgefail closed until a separate minimal-input grader sandbox exists.extra_mounts(~/.uipath), so it fails closed under the isolation default until that migration lands.protected_mocksconfiguration is not part of this PR - it moves to the follow-up PR along with the mock service itself.Validation
make verifyrun on Windows against this branch:ruff format --checkandruff check: pass (361 files)pyright: 0 errors, 1 pre-existing Antigravity warningtests/test_sandbox.pyare pre-existing Windows symlink-privilege (WinError 1314) environment failures in a file this branch does not touch