Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 8 additions & 12 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -52,19 +52,13 @@ ARG CLAUDE_CODE_VERSION=2.1.177
RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
&& apt-get install -y --no-install-recommends nodejs \
&& rm -rf /var/lib/apt/lists/* \
&& npm config set prefix /usr/local \
&& npm install -g @anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}

# Every built-in agent backend routes its SDK-owned subprocess through the
# same setpriv policy. Claude's SDK accepts only one executable path, so it
# uses the small backend-specific wrapper below; Codex and Antigravity invoke
# the generic launcher directly.
# The trusted harness uses this launcher for one stateful worker containing
# the complete registry-selected Agent lifecycle. Every worker descendant
# inherits the same UID/GID, empty capability sets, and no-new-privileges bit.
COPY docker/coder_eval_drop_privilege.sh /usr/local/bin/coder_eval_drop_privilege.sh
COPY docker/coder_eval_claude_agent.sh /usr/local/bin/coder_eval_claude_agent.sh
RUN chmod 0555 \
/usr/local/bin/coder_eval_drop_privilege.sh \
/usr/local/bin/coder_eval_claude_agent.sh \
&& test "$(command -v claude)" = "/usr/local/bin/claude"
RUN chmod 0555 /usr/local/bin/coder_eval_drop_privilege.sh

# uv: matches host sandbox.py's `uv venv` + `uv pip install` fast path
RUN curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh
Expand Down Expand Up @@ -102,8 +96,10 @@ ARG SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="openai-codex-cli-bin,openai-codex
RUN export SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="${SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS}" && \
uv export --frozen --extra codex --extra antigravity ${CODER_EVAL_UV_EXTRAS} | uv pip install --system -r /dev/stdin

# Sanity check: the in-container entrypoint subcommand must be wired up.
RUN coder-eval _run-task-internal --help > /dev/null
# Sanity check the public container entrypoint and the worker module that the
# isolation proxy launches directly.
RUN coder-eval _run-task-internal --help > /dev/null \
&& python -I -m coder_eval.isolation.agent_worker < /dev/null

# Stamp the image with the installed coder_eval version so the host can do
# a pre-flight version assertion against `docker image inspect` BEFORE
Expand Down
4 changes: 0 additions & 4 deletions docker/coder_eval_claude_agent.sh

This file was deleted.

6 changes: 3 additions & 3 deletions docs/DOCKER_ISOLATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,11 @@ Both build `coder-eval-agent:<pkg-version>` and tag it `:latest`.

## Agent/grader identity boundary

`sandbox.docker.agent_isolation` defaults to `true`. The container harness and grader remain root, while every evaluated Claude, Codex, or Antigravity subprocess runs as `agent:agent` (`2000:2000`).
`sandbox.docker.agent_isolation` defaults to `true`. The container harness and grader remain root, while the complete lifecycle of the registry-selected agent runs in a stateful worker as `agent:agent` (`2000:2000`). This applies equally to the built-in agents and to third-party `AgentRegistry` plugins installed in the run image; there is no built-in-kind allowlist.

The agent launcher clears inheritable, ambient, and bounding capabilities and sets `no_new_privs`. Generated work is placed in `/work/agent`. Hidden task data, results, raw task/plugin/reference/template sources, and grader inputs live below root-only `/opt/coder-eval/grader`. Raw source bind mounts remain read-only and are never chmod/chowned; only disposable staging copies and the generated workspace are changed.

Older/custom images must declare `org.coder-eval.agent-isolation=uid-gid-v1`. A protected run rejects an image without that label before making an LLM call. Images derived with `FROM coder-eval-agent:<current-version>` inherit it. Runtime-kit injection into an unrelated base does not yet provide the required Linux users and `setpriv` launchers, so it is not compatible with protected mode.
Older/custom images must declare `org.coder-eval.agent-isolation=uid-gid-v1`. A protected run rejects an image without that label before making an LLM call. Images derived with `FROM coder-eval-agent:<current-version>` inherit it. Runtime-kit injection into an unrelated base does not yet provide the required Linux user and `setpriv` launcher, so it is not compatible with protected mode.

## Running a task in Docker

Expand Down Expand Up @@ -321,7 +321,7 @@ The host's run dir is bind-mounted read-write at `/opt/coder-eval/grader/output`

The host's `DockerRunner` rewrites host paths to protected container paths, renders `docker run`, and tails container stdout into `docker.log`. Inputs land at `/opt/coder-eval/grader/input`, output at `/opt/coder-eval/grader/output`, the raw task directory at `/opt/coder-eval/grader/task_dir`, and the agent workspace at `/work/agent`.

Inside the container, the root entrypoint verifies the protected parent. The standard Orchestrator prepares the workspace as root, grants only that generated tree to UID 2000, and launches the selected agent through the shared privilege-drop policy. The host reads the final result from the protected output mount and feeds the existing aggregation pipeline.
Inside the container, the root entrypoint verifies the protected parent. The standard Orchestrator prepares the workspace as root, grants only that generated tree to UID 2000, and launches one stateful agent worker through the shared privilege-drop policy. The worker loads the same registry, constructs the selected agent, and owns `start` / `communicate` / `stop` plus all descendants. Stream events and typed turn results cross back to the root orchestrator over a framed protocol; no agent implementation is instantiated in the privileged process. The host reads the final result from the protected output mount and feeds the existing aggregation pipeline.

Protected runs use Docker's init reaper and default to a 512-process limit when `limits.max_pids` is not specified. An explicit `max_pids` value takes precedence. Before trusted post-run/finalization begins, the harness stops the SDK, repeatedly kills every remaining UID-2000 process, and fails closed if that UID cannot be emptied.

Expand Down
68 changes: 11 additions & 57 deletions src/coder_eval/agents/antigravity_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,6 @@
import contextlib
import logging
import os
import shlex
import shutil
import tempfile
import time
from collections.abc import AsyncIterator, Callable
from contextlib import AsyncExitStack
Expand All @@ -42,10 +39,7 @@
TurnTimeoutError,
truncate_crash_message,
)
from coder_eval.isolation.agent_identity import agent_isolation_enabled
from coder_eval.models import (
AGENT_HOME,
CONTAINER_DROP_SHIM,
AgentKind,
AntigravityAgentConfig,
ApiRoute,
Expand All @@ -72,11 +66,7 @@
TurnEndStatus,
TurnStartEvent,
)
from coder_eval.utils import (
AGENT_ENV_SCRUB_PREFIXES,
AGENT_ENV_SCRUB_VARS,
expand_env_vars,
)
from coder_eval.utils import expand_env_vars


logger = logging.getLogger(__name__)
Expand Down Expand Up @@ -225,7 +215,6 @@ def __init__(
# Absolute dirs to prepend to PATH so sandbox mock CLIs shadow real ones
# for the harness's run_command tool — applied at spawn (see start()).
self._env_path_prepend: list[str] = []
self._drop_shim_dir: Path | None = None
# _state / _iteration / _iteration_was_incremented / pending_turn lifecycle
# bookkeeping lives on the Agent base class (shared defaults + helpers).
self._log = PrefixedAdapter(logger, {"prefix": instance_name})
Expand All @@ -234,22 +223,6 @@ def _effective_model(self) -> str:
"""Resolve the model: task ``agent.model`` > ``ANTIGRAVITY_MODEL`` > default."""
return self.config.model or settings.antigravity_model or _DEFAULT_MODEL

def _stage_localharness_drop_shim(self) -> Path:
"""Shadow localharness with a wrapper around the shared setpriv policy."""

real = shutil.which("localharness")
if real is None:
raise RuntimeError("agent isolation is enabled but localharness is not available on PATH")
shim_dir = Path(tempfile.mkdtemp(prefix="antigravity-drop-"))
wrapper = shim_dir / "localharness"
wrapper.write_text(
f'#!/usr/bin/env bash\nexec {CONTAINER_DROP_SHIM} {shlex.quote(real)} "$@"\n',
encoding="utf-8",
)
wrapper.chmod(0o555)
self._drop_shim_dir = shim_dir
return shim_dir

def _resolve_skills_paths(self, plugin_tools_dir: str | None) -> list[str]:
"""Resolve skill search-path roots for the harness's native ``skills_paths``.

Expand Down Expand Up @@ -343,8 +316,6 @@ async def start(
"""
self.working_directory = Path(working_directory)
self._env_path_prepend = list(env_path_prepend or [])
if agent_isolation_enabled():
self._env_path_prepend.insert(0, str(self._stage_localharness_drop_shim()))
self._state = AgentState.WORKING

try:
Expand Down Expand Up @@ -420,33 +391,20 @@ async def _harness_spawn_guard(self) -> AsyncIterator[None]:
PATH window, or its harness would inherit another task's mock dirs.
"""
async with _harness_spawn_lock():
scrubbed = {
name: os.environ.pop(name)
for name in list(os.environ)
if name in AGENT_ENV_SCRUB_VARS or name.startswith(AGENT_ENV_SCRUB_PREFIXES)
}
if not self._env_path_prepend:
yield
return
path_key = next((k for k in os.environ if k.upper() == "PATH"), "PATH")
original_path = os.environ.get(path_key)
original_home = os.environ.get("HOME")
if self._env_path_prepend:
os.environ[path_key] = os.pathsep.join([*self._env_path_prepend, original_path or ""])
self._log.debug("PATH prepend for harness spawn: %s", os.pathsep.join(self._env_path_prepend))
if agent_isolation_enabled():
os.environ["HOME"] = AGENT_HOME
original = os.environ.get(path_key)
os.environ[path_key] = os.pathsep.join([*self._env_path_prepend, original or ""])
self._log.debug("PATH prepend for harness spawn: %s", os.pathsep.join(self._env_path_prepend))
try:
yield
finally:
os.environ.update(scrubbed)
if self._env_path_prepend:
if original_path is None:
os.environ.pop(path_key, None)
else:
os.environ[path_key] = original_path
if agent_isolation_enabled():
if original_home is None:
os.environ.pop("HOME", None)
else:
os.environ["HOME"] = original_home
if original is None:
os.environ.pop(path_key, None)
else:
os.environ[path_key] = original

async def communicate(
self,
Expand Down Expand Up @@ -621,10 +579,6 @@ async def _teardown(self) -> None:
if stack is not None:
with contextlib.suppress(Exception):
await stack.aclose()
shim_dir, self._drop_shim_dir = self._drop_shim_dir, None
if shim_dir is not None:
with contextlib.suppress(Exception):
await asyncio.to_thread(shutil.rmtree, shim_dir, ignore_errors=True)


class _AntigravityTurnState:
Expand Down
13 changes: 2 additions & 11 deletions src/coder_eval/agents/claude_code_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,7 @@
format_timeout_reason,
)
from coder_eval.formatting import format_messages, format_payload
from coder_eval.isolation.agent_identity import agent_isolation_enabled
from coder_eval.models import (
AGENT_HOME,
CONTAINER_CLAUDE_SHIM,
AgentKind,
ApiRoute,
BedrockRoute,
Expand Down Expand Up @@ -73,7 +70,7 @@
TurnEndStatus,
TurnStartEvent,
)
from coder_eval.utils import dump_dataclass, process_plugins, scrub_agent_env_overrides
from coder_eval.utils import dump_dataclass, process_plugins


logger = logging.getLogger(__name__)
Expand Down Expand Up @@ -768,11 +765,9 @@ def _build_sdk_env(
Returns:
Tuple of (env_vars_dict, model_override_or_None).
"""
base_env: dict[str, str] = scrub_agent_env_overrides()
base_env: dict[str, str] = {}
if path := os.environ.get("PATH"):
base_env["PATH"] = path
if agent_isolation_enabled():
base_env["HOME"] = AGENT_HOME

if path_prepend:
prefix = os.pathsep.join(path_prepend)
Expand Down Expand Up @@ -1204,10 +1199,6 @@ def _build_claude_query(
if isinstance(self.config.claude_settings, dict)
else self.config.claude_settings,
mcp_servers=self._extra_mcp_servers,
# The SDK accepts a single CLI executable path. The baked wrapper
# invokes the real Claude binary through the same setpriv policy as
# the other backends (UID/GID drop, no capabilities, no_new_privs).
cli_path=CONTAINER_CLAUDE_SHIM if agent_isolation_enabled() else None,
**self.config.sdk_options,
)

Expand Down
50 changes: 7 additions & 43 deletions src/coder_eval/agents/codex_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,7 @@
TurnTimeoutError,
truncate_crash_message,
)
from coder_eval.isolation.agent_identity import agent_isolation_enabled, grant_agent_workspace
from coder_eval.models import (
AGENT_HOME,
CONTAINER_DROP_SHIM,
AgentKind,
ApiRoute,
AssistantMessage,
Expand All @@ -55,7 +52,7 @@
TurnEndStatus,
TurnStartEvent,
)
from coder_eval.utils import expand_env_vars, scrub_agent_env_overrides
from coder_eval.utils import expand_env_vars


logger = logging.getLogger(__name__)
Expand Down Expand Up @@ -663,7 +660,6 @@ def __init__(
self.working_directory: Path | None = None
self._env_path_prepend: list[str] = []
self._login_shell_home: Path | None = None
self._runtime_codex_home: Path | None = None
# _state / _iteration / _iteration_was_incremented / pending_turn lifecycle
# bookkeeping lives on the Agent base class (shared defaults + helpers).
self._log = PrefixedAdapter(logger, {"prefix": instance_name})
Expand Down Expand Up @@ -693,25 +689,14 @@ async def start(
self.working_directory = Path(working_directory)
self._env_path_prepend = list(env_path_prepend or [])
self._setup_login_shell_home()
if agent_isolation_enabled():
if self._login_shell_home is None:
self._login_shell_home = Path(tempfile.mkdtemp(prefix="coder-eval-codex-home-"))
self._runtime_codex_home = Path(AGENT_HOME) / ".codex"
await asyncio.to_thread(self._runtime_codex_home.mkdir, parents=True, exist_ok=True)
grant_agent_workspace(self._login_shell_home)
grant_agent_workspace(self._runtime_codex_home)
self._state = AgentState.WORKING

try:
from openai_codex import Codex, CodexConfig

# Build CodexConfig with environment variables for custom API configuration
env_override = self._build_codex_env()
launch_args_override = self._drop_privilege_launch_args()
if env_override is not None or launch_args_override is not None:
config = CodexConfig(env=env_override, launch_args_override=launch_args_override)
else:
config = None
config = CodexConfig(env=env_override) if env_override else None

# Initialize the Codex client (context manager compatible). Close any
# prior client first: start() is driven through execute_with_retry, so
Expand Down Expand Up @@ -1092,16 +1077,6 @@ def _effective_model(self) -> str | None:
"""
return self.config.model or settings.codex_model

@staticmethod
def _drop_privilege_launch_args() -> tuple[str, ...] | None:
"""Replace the SDK app-server argv with the shared UID-drop launcher."""

if not agent_isolation_enabled():
return None
from codex_cli_bin import bundled_codex_path

return (CONTAINER_DROP_SHIM, str(bundled_codex_path()), "app-server", "--listen", "stdio://")

def _build_codex_env(self) -> dict[str, str] | None:
"""Build the environment passed to the Codex app-server.

Expand All @@ -1116,7 +1091,7 @@ def _build_codex_env(self) -> dict[str, str] | None:
(and normalizes the PATH key case-insensitively), so a full PATH value
here safely replaces the inherited one.
"""
env: dict[str, str] = scrub_agent_env_overrides()
env: dict[str, str] = {}
api_key = os.getenv("CODEX_API_KEY")
if api_key:
env["CODEX_API_KEY"] = api_key
Expand All @@ -1141,12 +1116,6 @@ def _build_codex_env(self) -> dict[str, str] | None:
codex_home = self._codex_home()
codex_home.mkdir(parents=True, exist_ok=True)
env["CODEX_HOME"] = str(codex_home)
elif agent_isolation_enabled():
env["HOME"] = AGENT_HOME
env["ZDOTDIR"] = AGENT_HOME
codex_home = self._codex_home()
codex_home.mkdir(parents=True, exist_ok=True)
env["CODEX_HOME"] = str(codex_home)
return env if env else None

@staticmethod
Expand Down Expand Up @@ -1188,14 +1157,10 @@ def _setup_login_shell_home(self) -> None:
self._cleanup_login_shell_home()
if not (self._env_path_prepend and self._login_shell_profiles_supported()):
return
# The harness remains root in protected Docker runs. Its HOME and
# ZDOTDIR are private grader state and must never be restored by an
# agent login shell. Use the dedicated agent home as both the runtime
# home and the only profile source in that mode.
original_home = AGENT_HOME if agent_isolation_enabled() else os.environ.get("HOME", "")
original_home = os.environ.get("HOME", "")
# Where the user's REAL zsh dotfiles live: their own ZDOTDIR when set,
# else their home (zsh's fallback).
original_zdotdir = AGENT_HOME if agent_isolation_enabled() else os.environ.get("ZDOTDIR", "") or original_home
original_zdotdir = os.environ.get("ZDOTDIR", "") or original_home
# The profile only ever executes under a POSIX shell, so the PATH
# separator is ':' regardless of the host building it.
quoted_prepend = shlex.quote(":".join(self._env_path_prepend))
Expand Down Expand Up @@ -1801,10 +1766,9 @@ async def _recover_subagent_tool_calls(
# Best-effort: a recovery hiccup must never fail the turn.
self._log.debug("CodexAgent: sub-agent recovery failed for %s: %s", thread_id, exc)

def _codex_home(self) -> Path:
@staticmethod
def _codex_home() -> Path:
"""Codex data directory (rollouts live under ``<home>/sessions``)."""
if self._runtime_codex_home is not None:
return self._runtime_codex_home
return Path(os.environ.get("CODEX_HOME") or (Path.home() / ".codex"))

async def _await_rollout_file(self, home: Path, thread_id: str, *, attempts: int = 20) -> Path | None:
Expand Down
Loading