From bbeaa4005c15aacd0e31a6e4fd7c09274954f6ca Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Sun, 5 Jul 2026 05:17:26 +0200 Subject: [PATCH 1/2] docs(adr): record coding-agent target runtime contract --- ...18-coding-agent-target-runtime-contract.md | 243 ++++++++++++++++++ 1 file changed, 243 insertions(+) create mode 100644 docs/adr/0018-coding-agent-target-runtime-contract.md diff --git a/docs/adr/0018-coding-agent-target-runtime-contract.md b/docs/adr/0018-coding-agent-target-runtime-contract.md new file mode 100644 index 000000000..395a5af07 --- /dev/null +++ b/docs/adr/0018-coding-agent-target-runtime-contract.md @@ -0,0 +1,243 @@ +# 18. Coding-agent target runtime contract + +Date: 2026-07-05 + +## Status + +Accepted (2026-07-05). Tracks the coding-agent runtime closeout under Beads +`av-t2o5`, `av-y7eq.10`, `av-y7eq.9`, and `av-t2o5.2`. + +This ADR records the target runtime contract. It does not claim live provider +matrix completion; host-runtime dogfood evidence is owned by `av-y7eq.9`. +Profile and sandbox isolation evidence is deferred to `av-t2o5.1`. +Provider-agnostic recorded trajectory replay and removal of `provider: +copilot-log` from authored target YAML are owned by `av-t2o5.2`. + +## Context + +AgentV evaluates coding agents in real repositories. Those agents are not +ordinary LLM APIs: they run tools, read files from disk, mutate workspaces, +manage auth profiles, stream transcripts, and may spawn their own subprocesses. +If AgentV imports fragile agent SDKs directly into the main CLI/orchestrator +process, a provider bug can crash the run before AgentV finalizes +`.agentv/results//`, `summary.json`, `.internal/index.jsonl`, +transcripts, and grading artifacts. + +The product direction is repo-native, workspace-native evaluation with portable +run bundles as the source of truth. The default path should therefore evaluate +the real installed agent/profile on the host when that is what the user wants, +while still keeping provider instability inside target execution envelopes. + +Peer frameworks are evidence, not schema authority: + +- Promptfoo local clone `/home/entity/projects/promptfoo/promptfoo` at commit + `6bfc5a0c7f16f9c4717ac731d276b578e63d0769` separates coding-agent provider + families such as Codex SDK, Codex app-server, and Claude Agent SDK. Its + taxonomy explicitly says provider IDs should encode the runtime boundary. +- Promptfoo also shows useful optional dependency ergonomics: Claude Agent SDK + is loaded only for that provider, but the SDK still runs in Promptfoo's + provider process. AgentV adopts optional/lazy SDK loading, but adds an + AgentV-owned child-runner boundary. +- Vercel `agent-eval` at commit + `a9dcc9a8c53dbc22ececc967ded7ab3963f18e67` runs coding agents through + sandboxed CLI-like execution, records raw and parsed transcripts, and writes + result bundles under `results///`. +- Margin Evals at commit + `53fb2fd080689efaf7934573d8759d14fc1043e4` uses run-centric artifacts, + process/runtime logs, managed agent definitions, and trajectory hooks. +- Harbor uses container environments and ATIF trajectories for benchmark-grade + agent execution. Its 2026-06-18 change to run `harbor check` and + `harbor analyze` as Harbor trials instead of in-process Claude SDK calls + supports the same boundary: produce artifacts from real executions instead + of hiding work inside the coordinator process. +- Kata Symphony and Taskplane validate Pi RPC as a process/stdio control + boundary: `pi --mode rpc` is launched as a live subprocess locally, over + SSH, or through worker orchestration, rather than being collapsed into an + in-process SDK call. +- entireio/cli preserves native agent sessions and derives normalized + transcript/checkpoint metadata from provider-specific logs. That pattern + supports AgentV preserving raw logs as provenance and importing them into + provider-agnostic replay artifacts, not exposing one live `*-log` target + provider per backend. + +## Decision + +AgentV treats coding-agent targets as external runtimes to orchestrate, not +libraries to call in-process by default. + +Authored targets use this shape: + +```yaml +targets: + - id: codex-local + provider: codex-app-server + runtime: host + config: + command: ["codex", "app-server"] + model: gpt-5-codex +``` + +The fields mean: + +| Field | Meaning | +| --- | --- | +| `id` | Stable AgentV target identity used for CLI selection, artifacts, Dashboard, and comparisons. | +| `provider` | Adapter/control boundary such as `codex-cli`, `codex-app-server`, `pi-rpc`, `claude-cli`, or `copilot-sdk`. | +| `runtime` | Placement/isolation mode: `host`, `profile`, or `sandbox`; may be a string shorthand or an object with `mode`. | +| `config` | Provider-specific knobs such as `command`, `model`, `cwd`, `timeout_seconds`, auth endpoint settings, permission flags, and provider protocol settings. | + +Do not add competing top-level fields such as `isolation`, `sandbox`, +`profile`, `install`, `container`, `environment`, `executable`, `binary`, +`args`, or `arguments` for this contract. Process/protocol providers use +`config.command` as a non-empty argv array. General eval policy such as +concurrency belongs under `execution`, not inside a target definition. Grader +selection belongs to `defaults.grader`, CLI overrides, or evaluator-level target +selection, not to the system-under-test target. + +### Provider Boundaries + +Process and protocol providers are the preferred defaults: + +- `codex-app-server`: preferred Codex rich protocol/control boundary. +- `codex-cli`: simple Codex subprocess boundary for host/profile execution and + installed user shims. +- `pi-rpc`: preferred Pi rich control boundary over stdio/RPC. +- `pi-cli`: simple Pi subprocess boundary. +- `claude-cli`: default Claude path through the installed Claude CLI. +- `copilot-cli`: active Copilot execution through the installed CLI/protocol + path. + +SDK providers are explicit advanced paths: + +- `codex-sdk` +- `pi-sdk` +- `claude-sdk` +- `copilot-sdk` + +SDK transports run behind an AgentV child-runner process on the host. The parent +CLI/orchestrator starts the child with the target config and provider request, +receives structured events/logs and one final provider response envelope, and +maps child crashes, malformed child output, timeouts, and cancellation into +provider-scoped errors. The concrete SDK package is imported inside the child +runner only for the selected SDK target. + +This is process isolation for SDK dependency and crash containment. It is not +Docker/container isolation and does not make `runtime: host` equivalent to +`runtime.mode: sandbox`. + +### Runtime Modes + +| Runtime | Boundary | Use case | +| --- | --- | --- | +| `host` | Runs the installed CLI, protocol server, or SDK child runner on the current machine with the user's normal auth/profile unless provider config overrides it. | Local research, subscription OAuth, and evaluating the exact installed agent/profile an engineer uses manually. | +| `profile` | Runs a host process with isolated home/provider home/temp/env configuration, such as `HOME`, `CODEX_HOME`, provider homes, temp dirs, and explicit env allowlists. | Cleaner local evals without full container cost. | +| `sandbox` | Runs through a separate substrate such as Docker, a managed sandbox, remote worker, or future container backend. | CI, reproducibility, untrusted tasks, and stronger filesystem/runtime containment. | + +Host runtime is the first supported path for coding-agent targets. CLI, RPC, +and app-server transports run against the host-installed agent/profile. SDK +transports also run on the host, but through the AgentV child-runner process +described above. + +The current implementation supports Docker sandbox execution for generic +`provider: cli`. Sandbox-aware coding-agent providers are future work. When a +coding-agent provider is authored with `runtime.mode: sandbox` before a +sandbox-aware runner exists, AgentV should return a deliberate +`target_execution` error envelope rather than pretending the target ran or +crashing the evaluator. + +Codex `config.sandbox_mode` is a Codex provider permission/sandbox knob passed +to Codex. It is not AgentV `runtime.mode: sandbox`. + +### Failure Contract + +Coding-agent providers must report target failures through structured +`target_execution` envelopes whenever possible. That includes: + +- spawn failures and missing executables +- provider nonzero exits +- malformed provider output +- provider timeouts or cancellations +- SDK child-runner crashes before or after partial events +- sandbox infrastructure failures +- target task failures returned by a protocol provider +- partial transcripts or logs from a failed provider + +Target crashes are target results. They must not become AgentV orchestrator +crashes or prevent final run-bundle artifacts from being written. + +### Replay and Log Providers + +`provider: copilot-log` is removed from the authored live target surface before +beta. AgentV should not add `codex-log`, `claude-log`, `pi-log`, or other +provider-specific log target providers. + +Provider-native logs remain useful as raw provenance and import inputs. Copilot +`events.jsonl` parsing should feed import/normalization into a +provider-agnostic recorded trajectory replay contract. Replay is an +eval/orchestrator mode or generic replay target over AgentV trajectory artifacts, +not a live coding-agent runtime provider. Live Copilot targets remain +`copilot-cli` and `copilot-sdk`. + +This aligns with ADR 0008: raw native transcripts are preserved for debugging +and parser improvement, while normalized AgentV transcript/trajectory artifacts +are the durable input to grading, Dashboard inspection, and replay. + +## Consequences + +- Users can evaluate the same host-installed agent/profile they use manually. +- Provider IDs remain explicit about control boundary instead of collapsing + runtime choices into provider config flags. +- SDK providers stay available when SDK-native events or controls are worth the + extra complexity, but SDK dependency failures do not take down the parent CLI. +- `runtime: host` remains lightweight and zero-infra; stronger profile/sandbox + isolation can be added without changing target identity semantics. +- Generic Docker sandbox support through `provider: cli` remains valid, while + sandbox-aware coding-agent adapters are deliberately deferred. +- Offline grading/replay gets one provider-agnostic path instead of one + provider-specific `*-log` target surface per backend. + +## Alternatives Considered + +### Import coding-agent SDKs in the main AgentV process + +Rejected. Lazy SDK import is helpful for optional dependencies, but it is not a +runtime isolation boundary. Prior Pi SDK dogfood exposed stream teardown +failures that can outlive the apparent agent result and crash the parent +process. The parent process must own run finalization, timeout enforcement, and +artifact writing. + +### Make SDK providers the default because they are structured + +Rejected. SDKs can expose useful events, but the default AgentV path should +match the real installed CLI/profile where possible and keep the product +zero-infra. SDK providers are explicit advanced targets. + +### Copy Promptfoo provider naming wholesale + +Rejected. Promptfoo is useful evidence for explicit provider IDs and optional +provider dependencies, but AgentV keeps target identity and backend/control +boundary separate: `id` is stable AgentV target identity, while `provider` names +the adapter kind. AgentV does not copy Promptfoo's use of `label` as the target +identity field or carry compatibility aliases where the beta contract can be +cleaner. + +### Put runtime placement under provider-specific config + +Rejected. Runtime placement is cross-provider orchestration state. It belongs in +`runtime`, not in every provider's `config` with different names and precedence +rules. + +### Treat provider logs as live target providers + +Rejected. Passive logs do not run an agent and should not satisfy live +host-runtime dogfood. They are import/replay sources. Keeping them out of +authored live target YAML avoids a family of `*-log` providers and preserves a +single normalized replay contract. + +## Non-Goals + +- Implementing or validating the full live provider matrix. +- Implementing profile-mode or sandbox-aware coding-agent provider runners. +- Replacing the generic `provider: cli` sandbox path. +- Designing the full provider-agnostic replay cassette contract. +- Adding compatibility aliases for removed beta-only target provider names. From e3344f5f235a4be858d05e74c5afd26e5a67efde Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Sun, 5 Jul 2026 05:29:07 +0200 Subject: [PATCH 2/2] docs(adr): correct authored concurrency field --- docs/adr/0018-coding-agent-target-runtime-contract.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/adr/0018-coding-agent-target-runtime-contract.md b/docs/adr/0018-coding-agent-target-runtime-contract.md index 395a5af07..b529eb8fd 100644 --- a/docs/adr/0018-coding-agent-target-runtime-contract.md +++ b/docs/adr/0018-coding-agent-target-runtime-contract.md @@ -89,10 +89,10 @@ The fields mean: Do not add competing top-level fields such as `isolation`, `sandbox`, `profile`, `install`, `container`, `environment`, `executable`, `binary`, `args`, or `arguments` for this contract. Process/protocol providers use -`config.command` as a non-empty argv array. General eval policy such as -concurrency belongs under `execution`, not inside a target definition. Grader -selection belongs to `defaults.grader`, CLI overrides, or evaluator-level target -selection, not to the system-under-test target. +`config.command` as a non-empty argv array. Authored eval concurrency belongs +under `evaluate_options.max_concurrency`, not inside a target definition. +Grader selection belongs to `defaults.grader`, CLI overrides, or +evaluator-level target selection, not to the system-under-test target. ### Provider Boundaries