diff --git a/docs/adr/2026-06-17-harbor-runner-boundary.md b/docs/adr/2026-06-17-harbor-runner-boundary.md index 4ca7749c8..1d1446428 100644 --- a/docs/adr/2026-06-17-harbor-runner-boundary.md +++ b/docs/adr/2026-06-17-harbor-runner-boundary.md @@ -37,19 +37,45 @@ Harbor should own: - Harbor `task.toml` files and Harbor YAML config; - Opik trace upload through Harbor when enabled. +## Alignment with experiment separation + +The 2026-06-23 experiment/eval separation decision makes runtime binding an +experiment concern. Harbor execution should follow the same split: + +- AgentV eval YAML remains the authoring or selection surface for what benchmark + suite is being evaluated. +- AgentV experiment YAML selects or pins the Harbor runner, candidate + agent/model, run policy, and other runtime binding. +- Harbor-authored YAML remains Harbor's own config surface when the standard + suite needs Harbor-specific task packaging or verifier settings. + +This means the examples below describe the desired logical fields, but new +runtime fields should be placed on an experiment unless they are genuinely part +of the benchmark suite identity. Do not put candidate agent/model binding in the +eval file for new AgentV-native examples. + ## Minimal future config surface -The AgentV eval file should select Harbor with a nested runner config: +An AgentV eval suite can select the benchmark source without copying Harbor's +task schema or claiming to be the runtime runner: ```yaml -name: swebench-verified-codex +name: swebench-verified -execution: - runner: harbor - harbor: - dataset: swebench-verified - agent: codex - model: openai/gpt-5-mini +source: + type: harbor + dataset: swebench-verified +``` + +The corresponding experiment selects how that suite runs: + +```yaml +name: swebench-verified-codex +target: codex-gpt5-mini +evals: evals/swebench-verified.eval.yaml +runner: + type: harbor + options: opik: enabled: true ``` @@ -57,10 +83,9 @@ execution: For a Harbor-authored YAML file, use `config` instead of `dataset`: ```yaml -execution: - runner: harbor - harbor: - config: ./harbor/swebench-verified.yaml +source: + type: harbor + config: ./harbor/swebench-verified.yaml ``` The first implementation should accept exactly one Harbor source selector: @@ -68,11 +93,54 @@ The first implementation should accept exactly one Harbor source selector: file. There should be no precedence rule between them. If both are set, fail validation and ask the user to choose one. -Keep Harbor-specific options nested under `execution.harbor`. Do not add -top-level AgentV fields for Harbor task packaging, verifier images, task patches, -or Docker/Compose adapter settings. If a Harbor option becomes too specific to -standardize, users should put it in the referenced Harbor YAML file instead of -AgentV adding a pass-through field. +Do not combine Harbor suite selection with candidate binding in the eval file: + +```yaml +# Avoid in eval.yaml +execution: + runner: harbor + harbor: + dataset: swebench-verified + agent: codex + model: openai/gpt-5-mini +``` + +Split that shape across the suite and experiment instead: + +```yaml +# evals/swebench-verified.eval.yaml +name: swebench-verified +source: + type: harbor + dataset: swebench-verified +``` + +```yaml +# experiments/swebench-verified-codex.yaml +name: swebench-verified-codex +target: codex +model: openai/gpt-5-mini +evals: evals/swebench-verified.eval.yaml +runner: + type: harbor +``` + +Keep Harbor suite source selection under `source` in the eval suite. Keep +experiment-side runner selection under `runner.type`, with runner knobs under +`runner.options`. The eval suite answers "where do these cases come from?"; the +experiment answers "how is this run executed?" Do not use `execution.runner` in +new eval-suite examples because that name collides with the experiment runner. +Do not repeat the runner discriminator as `runner.harbor.options`; `type: +harbor` already provides that namespace. + +Do not add top-level AgentV fields for Harbor task packaging, verifier images, +task patches, or Docker/Compose adapter settings. If a Harbor option becomes too +specific to standardize, users should put it in the referenced Harbor YAML file +instead of AgentV adding a pass-through field. + +If the Harbor integration later changes the eval source or experiment runner +schema, this ADR should be updated with the final shape. The boundary decision +is stable: Harbor runtime binding is not an eval-case schema extension. ## CLI invocation strategy @@ -82,8 +150,9 @@ Native evals continue to run with the existing command: agentv eval evals/native.eval.yaml --target codex ``` -Harbor-backed evals should use the same top-level entrypoint and dispatch based -on `execution.runner`: +Harbor-backed evals should use the same top-level entrypoint. If no explicit +experiment runner is configured, AgentV may infer Harbor execution from +`source.type: harbor`: ```bash agentv eval evals/swebench-harbor.eval.yaml @@ -105,9 +174,9 @@ agentv results import harbor --job ``` Do not overload native `--target` semantics in the first Harbor runner slice. -Harbor `agent`, `model`, and matrix behavior should come from -`execution.harbor` or the referenced Harbor YAML until repeated usage proves a -shared AgentV flag is needed. +Harbor `agent`, `model`, and matrix behavior should come from the experiment or +the referenced Harbor YAML until repeated usage proves a shared AgentV flag is +needed. ## Unsupported fields and non-goals @@ -128,8 +197,9 @@ standard-suite path. ## Implementation sequencing 1. Document the native-vs-Harbor boundary and commit alias rules. -2. Add schema validation for optional `execution.runner` and - `execution.harbor`, with no changes to native workspace acquisition. +2. Add schema validation for eval-suite `source.type: harbor` and exactly one of + `source.dataset` or `source.config`, plus experiment `runner.type` and + `runner.options`, with no changes to native workspace acquisition. 3. Add a Harbor launch adapter that records job identity and status. 4. Add a Harbor result importer that maps rewards, exceptions, timings, artifacts, and Opik trace URLs into AgentV run bundles. diff --git a/docs/adr/2026-06-23-experiments-vs-eval-separation.md b/docs/adr/2026-06-23-experiments-vs-eval-separation.md index aae9955f8..a74af472e 100644 --- a/docs/adr/2026-06-23-experiments-vs-eval-separation.md +++ b/docs/adr/2026-06-23-experiments-vs-eval-separation.md @@ -37,10 +37,22 @@ This decision must also preserve AgentV's existing product boundary: ## Vocabulary -An eval is a frozen task definition. It includes the prompt or dataset, expected -behavior, task-owned workspace fixtures, and assertions. AgentV's LLM-judge, -code-grader, deterministic assertions, and hidden or explicit evaluation -criteria belong here. +An eval suite is a frozen task-definition boundary. It includes suite metadata, +shared prompt/context, case references, shared assertions or graders, and +task-owned workspace fixtures. AgentV's LLM-judge, code-grader, deterministic +assertions, and hidden or explicit evaluation criteria belong here. + +An eval case is one atomic task inside a suite. It includes the case id, prompt +or input, criteria, expected output or reference behavior, case metadata, and +case-specific workspace overrides. A suite can inline cases, point to +`cases.yaml`/JSONL, or use a directory convention where each case owns files such +as `TASK.txt`, `PROMPT.md`, `answer/`, or `grader.test.ts`. + +In that directory-convention form, `EVAL.yaml` may be thin or inferred by a +loader, but the suite layer is still present conceptually: the directory +convention plus runner adapter is the suite contract. This distinction matters +because AgentV is a reusable framework, not a single benchmark harness whose +suite semantics can live only in code. An experiment is a committed or generated run definition. It declares which agent, target, provider, model, harness options, setup steps, run count, timeout, @@ -62,6 +74,13 @@ task inputs, datasets, assertions, and task fixtures. They should not be the canonical place for which agent, model, harness, setup injection, sandbox, or run matrix executes the task. +For simple projects, an eval-only run remains valid. AgentV treats the implicit +experiment label as `default` unless a committed experiment is configured. For +specialized harnesses that already have a strong directory contract, AgentV may +support loaders that infer the suite from the directory instead of requiring a +separate YAML file, but those loaders must still lower into the same suite/case +concepts. + Experiment files will live under `experiments/` by convention. AgentV will support YAML as the canonical authoring path for the abstraction story and TypeScript as the power-user escape hatch: @@ -85,6 +104,48 @@ setup: - script: cp skills/copilot/AGENTS.md AGENTS.md ``` +## Workspace boundary + +Workspace config belongs with the eval suite or case when it defines the task +scenario being replayed. Examples: + +- clone `org/repo` at a specific `commit` or `base_commit`; +- copy starter files, failing tests, fixtures, or issue prompts; +- run task-owned setup hooks that prepare the repo state required by the case; +- declare per-case repo pins or fixture overrides. + +Experiment setup belongs with the experiment when it changes the runtime +condition being compared. Examples: + +- choose `codex` versus `claude` targets; +- inject an `AGENTS.md`, skill, guideline file, or tool config for an A/B run; +- choose repeat/run policy, timeout, workers, budget, or sandbox mode; +- select a subset of suites or cases for a run campaign. + +Rule of thumb: if changing it changes the task being evaluated, put it in the +suite or case workspace. If changing it changes the candidate or run condition +measured against the same task, put it in the experiment. + +## Directory-style evals + +Convex-style harnesses are a useful counterexample to requiring YAML for every +case. A product-specific benchmark can encode each case as a directory with a +task prompt, reference solution, and executable grader. In AgentV terms, that is +not `experiment -> eval case` with no suite; it is an implicit suite contract +provided by the loader: + +```text +evals/// + TASK.txt # case input + answer/ # reference fixture + grader.test.ts # code-grader assertion +``` + +AgentV should support this as an import/loader shape when useful, but the core +contract remains `experiment -> eval suite -> eval case`. The experiment applies +runtime bindings to the selected suites/cases; it does not own the prompt, +expected behavior, or grading contract. + `config.yaml` will gain a default experiment pointer so existing `agentv eval` usage keeps working: