diff --git a/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md b/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md index 786cb6fbc..a4a41dd14 100644 --- a/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md +++ b/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md @@ -15,9 +15,12 @@ Status note (2026-07-04): implementation settled the grader vocabulary after this ADR was accepted. Current authored executable graders use `type: script`. `llm-rubric` is the promptfoo-compatible free-form rubric judge. Structured and multi-criteria rubric judging uses `g-eval` where itemized rubric semantics are -needed. `grading.json` exposes `assertion_results` plus `score`, `verdict`, and -`evidence`; do not teach `grading.json.assertions[]` as the current artifact -contract. +needed. The current output contract is owned by ADR 0017 and the active Beads: +authored YAML uses `assert`, `assert-set`, and `llm-rubric`, while `grading.json` +describes evaluated `graders[]` and nested `checks[]` with aggregate `pass`, +`score`, and `reason`. Do not teach `assertion_results`, `assertions`, +`passed`-only aliases, top-level `checks`, or dynamic one-grader artifact shapes +as the public contract. ## Context @@ -46,15 +49,14 @@ keep AgentV's only where its semantics are genuinely better.** AgentV extension rather than being forced into `llm-rubric`. Structured AgentV rubric criteria are preserved, not flattened into a single text blob: criteria objects keep `weight`, `operator`, `required`, - `score_ranges`, and `min_score`. Artifact assertion rows are the generic - AgentV grader contract: `grading.json.assertion_results[]` holds flattened - assertion evidence, and nested grader entries keep their own - `assertion_results[]` breakdown with score, verdict, and evidence. - Deterministic graders usually emit one row, while multi-aspect graders emit - one row per authored check or result unit. Structured rubric criteria - therefore populate one assertion row per criterion so the Dashboard can show - criterion-level evidence, using the same mechanism as script graders, field - accuracy, execution metrics, and tool trajectory. + `score_ranges`, and `min_score`. Result artifacts use the ADR 0017 grader + contract: `grading.json.graders[]` records each evaluated grader, and + `graders[].checks[]` records criterion- or component-level results when the + grader produces them. Deterministic graders usually emit no checks or one + check, while multi-aspect graders emit one check per authored criterion or + result unit. Structured rubric criteria therefore populate checks so the + Dashboard can show criterion-level evidence, using the same mechanism as + script graders, field accuracy, execution metrics, and tool trajectory. 3. **Grader execution**: `javascript` in-process (Bun `import`), `python` subprocess, `script` = the subprocess power tool (workspace-`cwd`, arbitrary language). `javascript` is NOT desugared to `script`. diff --git a/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md b/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md index ca5a3f04f..13303632a 100644 --- a/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md +++ b/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md @@ -15,8 +15,21 @@ Companion to [ADR 0016](0016-promptfoo-superset-eval-authoring-contract.md). We reviewed the output formats of promptfoo, margin-lab, vercel-agent-eval, and agentskills, and the workspace-acquisition models of SWE-bench, margin, Harbor, and -Inspect AI (`docs/plans/…` §6, §11.1). Two decisions follow: the canonical result -bundle, and how a workspace is acquired. +Inspect AI (`docs/plans/…` §6, §11.1). The active implementation scope is tracked in +Beads; PRs, ADRs, and plans summarize those Beads for review, but the Bead descriptions, +acceptance criteria, and notes remain the implementation source of truth. + +Promptfoo evidence was checked locally at `/home/entity/projects/promptfoo/promptfoo` +commit `6bfc5a0c7f16f9c4717ac731d276b578e63d0769`. Promptfoo's authored eval YAML is +the compatibility target where AgentV overlaps. Its result/export model is useful +inspiration, not AgentV's canonical artifact format: `EvaluateResult` carries +prompt/test/provider identity, `success`, `score`, `namedScores`, token usage, and a +full `gradingResult`; `GradingResult` carries `pass`, `score`, `reason`, optional +`componentResults`, and assertion metadata; `EvaluateSummaryV3`/`ResultsFile` wrap +results, prompts, stats, config, author, and variables. AgentV borrows the clean +aggregate grading vocabulary and component-breakdown idea, then writes them into a +filesystem/Git-native split run bundle instead of one consolidated DB/export file. +Two decisions follow: the canonical result bundle, and how a workspace is acquired. ## Decision — output/artifact contract (best-of-each, split, no DB) @@ -24,24 +37,30 @@ bundle, and how a workspace is acquired. maintained consolidated single-file export (generate on demand if ever needed). 3 of 4 references split; only promptfoo consolidates (for its DB/hosted model). 2. **Queryable aggregate ← margin-lab**: run-root `summary.json` is a rich `jq`-queryable - `Summary` (run_id, status breakdown, per-case **pass@k**, per-instance summaries, - usage, infra-failure taxonomy) — widen AgentV's current thin summary to this. Plus + `Summary` (run_id, status breakdown, `pass_rate`, `pass_count`, `sample_count`, + per-case `passed`/`pass_any` where applicable, per-instance summaries, usage, and + infra-failure taxonomy) — widen AgentV's current thin summary to this. Reserve + `pass_at_k`/pass@k vocabulary for explicit sampling metrics with a true `k`; do not + use `pass_at_1` when the value actually means "any sample passed". Plus `.internal/index.jsonl` (one row per case) for streaming/line queries. No database. 3. **Transcript + metrics ← vercel**: two-layer transcript (raw + normalized) with a canonical cross-agent `tool_name` enum and precomputed `transcript_summary`, the summary **inlined into each result row** for cheap trajectory/metrics assertions; transcript referenced **by path**. -4. **Per-assertion grading ← agentskills**: `grading.json` = `assertion_results[{ text, - passed, evidence }]` + `summary` counts, PLUS AgentV's superset — top-level **string - `verdict` (`pass`|`fail`|`skip`)** + fractional **`score`** (not a boolean; needs skip - + fractional). These rows are the generic AgentV grader evidence channel: every - grader returns `assertions[]`; deterministic graders typically return one row, - while multi-aspect graders return one row per distinct criterion/aspect. The - artifact preserves both the flattened rows and each grader's nested rows. - Default judge = skeptical evidence-by-path (opt-out via - explicit `prompt`); grader target selection flows through the config graph - (`defaults.grader`) or assertion-level target selection, not a system-under-test - target field. Evidence stays in `grading.json`. +4. **Per-attempt grading sidecar**: `grading.json` is an AgentV-native public contract + that keeps Promptfoo's aggregate grading vocabulary while preserving AgentV's richer + nested breakdown. It exposes top-level `pass`, `score`, `reason`, optional + `threshold`/`details`, and an always-present `graders[]` array. Each grader exposes + `name`, `type`, `pass`, `score`, `reason`, optional `threshold`/`details`, and + optional `checks[]`. Each check exposes `id?`, `text`, `pass`, optional `score`, + `reason`, and optional `evidence` only when the evidence is distinct from `reason`. + There are no public top-level `checks`, no dynamic single-grader shortcut, and no + public `assertion_results`, `assertions`, `passed`-only aliases, or + evidence-as-reason aliases. Authored YAML uses `assert`, `assert-set`, and + `llm-rubric`; result artifacts describe evaluated graders and checks. Default judge = + skeptical evidence-by-path (opt-out via explicit `prompt`); grader target selection + flows through the config graph (`defaults.grader`) or assertion-level target + selection, not a system-under-test target field. Evidence stays in `grading.json`. 5. **Bundle layout / naming**: machine files move under per-run **`.internal/`** (`index.jsonl`, `progress.json`, `events.jsonl`, `bundle.json`); run root stays clean (`summary.json` + per-case dirs). Rename the reference field `manifest_path` → @@ -61,9 +80,57 @@ Confirms ADR-0009 + ADR-0012 (not a new decision): ### Artifact filenames (locked — accuracy over cosmetic consistency) - **`summary.json`** (run-root AND per-case) — the aggregate. Kept over margin's `results.json`: it's a *summary*, not the full results (those are the per-case dirs + `index.jsonl`); avoids the `results//results.json` stutter; symmetric at both levels (run aggregates cases, case aggregates samples); vercel-aligned. We match margin on the aggregate *concept/shape*, not the filename. -- Per-sample triad (distinct, all kept): **`result.json`** (what happened), **`grading.json`** (verdict = `assertion_results`+`verdict`+`score`), **`metrics.json`** (duration+tokens+cost+execution/trajectory; the `timing.json` merge). +- Per-sample triad (distinct, all kept): **`result.json`** (what happened), **`grading.json`** (aggregate `pass`/`score`/`reason` plus `graders[]`/`checks[]`), **`metrics.json`** (duration+tokens+cost+execution/trajectory; the `timing.json` merge). - **`grading.json`** kept (not `grades.json`) — source-consistent with agentskills (whose file is `grading.json`), and "grading" names the grading *result*. +### `grading.json` wire-format example +```json +{ + "pass": false, + "score": 0.62, + "reason": "The answer names the right API but misses the rollback condition.", + "threshold": 0.8, + "details": { + "aggregation": "weighted_mean" + }, + "graders": [ + { + "name": "rubric", + "type": "llm-rubric", + "pass": false, + "score": 0.62, + "reason": "Two of three rubric checks passed.", + "threshold": 0.8, + "checks": [ + { + "id": "api", + "text": "Identifies the API used to publish result bundles.", + "pass": true, + "score": 1, + "reason": "Correctly identifies the publish command." + }, + { + "id": "rollback", + "text": "Explains when to roll back a failed publish.", + "pass": false, + "score": 0, + "reason": "Mentions retrying but not rollback criteria.", + "evidence": "The response says to rerun the command after any failure." + } + ] + } + ] +} +``` + +Summary and index guidance: use `pass_rate`, `pass_count`, and `sample_count` for run +and case aggregates; use `passed` for one execution outcome and `pass_any` when any +sample in a repeated case passed. Use `pass_at_k` only when the metric is an explicit +sampling metric with a real `k` and the calculation is documented on the summary row. +Index rows should stay lightweight: identity/outcome/named score/token usage fields +plus paths such as `result_path`, `grading_path`, `metrics_path`, `transcript_path`, +and `outputs_path`, not a full embedded grading tree. + ### Full results-tree layout (two levels — no per-run `.indexes`) ``` .agentv/results/ @@ -80,12 +147,12 @@ Confirms ADR-0009 + ADR-0012 (not a new decision): ``` - Per-run index (rows = cases) = `/.internal/index.jsonl`; **no separate per-run `.indexes`** — `.internal` already holds it. Cross-run catalog (rows = runs) = `.agentv/results/.indexes/runs.jsonl`. Names signal scope: `.internal` = one bundle; `.indexes`/`.cache` = across runs. Both dot-prefixed (skipped by discovery). - **Cross-run filtering needs `cases.jsonl`, not just `runs.jsonl`.** `runs.jsonl` (one row/run) answers "which runs match"; **case-level cross-run** queries ("every `fizzbuzz` across runs", "failing cases with tag X over last 10 runs", "trend of `test_id` T") need one row per (run x case) → `.indexes/cases.jsonl`, rebuilt by concatenating every `/.internal/index.jsonl` + run metadata. Join key for trends = the layered identity (content-hash `test_id` + author governance tag, ADR-0016 pt8). Both catalogs are derived/rebuildable; if JSONL scanning outgrows laptop scale, a rebuildable SQLite **view** is the escape hatch (optional adapter, never core — exploitbench pattern, Phoenix boundary intact). -- **margin-lab consistency & divergence:** matches on the *filesystem* substance — top-level queryable aggregate (`results.json`=`summary.json`), `internal/` machine folder (we dot-prefix `.internal/`), per-execution-unit dirs, one pure `Build()` for pass@k, `instance_key = test_id#sample_index`. **Divergences (deliberate):** (1) margin's *runner* uses a persistent **`RunStore` (in-memory / Postgres, NOT SQLite)** for scheduling + queries; **AgentV declines a store entirely** (laptop-first; resumability via `index.jsonl` + `--rerun-failed`). (2) hierarchical `/sample-N/` vs margin's flat `instances/#/`. (3) `timing`→`metrics` merge. The **rebuildable derived index/view** idea (JSONL `.indexes/`, optional SQLite escape hatch) is from **exploitbench** (`import`/`export` bijection), not margin — margin's store is the operational source during a run, not a filesystem-derived index. (Nuance: margin *can* rehydrate a run's completed-work state from its run-dir for **resume** — `LoadProgressSnapshot` + `loadSavedResumeBundle` + `carryForwardLocalCases` — but that's targeted carry-forward, not a general `import` that rebuilds the multi-run query DB from files; the memory store is ephemeral, the Postgres store persists independently. **AgentV follows exploitbench's model** — filesystem is source of truth, `.indexes/*.jsonl` are derived/rebuildable — with `--rerun-failed` reading `index.jsonl` from fs and no store to rehydrate.) +- **margin-lab consistency & divergence:** matches on the *filesystem* substance — top-level queryable aggregate (`results.json`=`summary.json`), `internal/` machine folder (we dot-prefix `.internal/`), per-execution-unit dirs, one pure `Build()` for pass rates and explicit sampling metrics, `instance_key = test_id#sample_index`. **Divergences (deliberate):** (1) margin's *runner* uses a persistent **`RunStore` (in-memory / Postgres, NOT SQLite)** for scheduling + queries; **AgentV declines a store entirely** (laptop-first; resumability via `index.jsonl` + `--rerun-failed`). (2) hierarchical `/sample-N/` vs margin's flat `instances/#/`. (3) `timing`→`metrics` merge. The **rebuildable derived index/view** idea (JSONL `.indexes/`, optional SQLite escape hatch) is from **exploitbench** (`import`/`export` bijection), not margin — margin's store is the operational source during a run, not a filesystem-derived index. (Nuance: margin *can* rehydrate a run's completed-work state from its run-dir for **resume** — `LoadProgressSnapshot` + `loadSavedResumeBundle` + `carryForwardLocalCases` — but that's targeted carry-forward, not a general `import` that rebuilds the multi-run query DB from files; the memory store is ephemeral, the Postgres store persists independently. **AgentV follows exploitbench's model** — filesystem is source of truth, `.indexes/*.jsonl` are derived/rebuildable — with `--rerun-failed` reading `index.jsonl` from fs and no store to rehydrate.) - **Dashboard default view is sensible, never odd/empty:** because `tags.experiment` is value-defaulted to the eval/suite name (always populated), the default view groups by `experiment` (real names, no "(none)" wall) or a recent-runs list; the grouping key is a user preference they can change, not the absence of a default. ### Run organization: cross-run index, repeat naming, experiment-as-tag -- **Cross-run index (rebuildable cache, not source of truth):** keep per-run `index.jsonl` (rows = cases); add a cross-run catalog `.agentv/results/.indexes/runs.jsonl` (already-reserved `.indexes/` namespace) — **one row per run** (run_id, timestamp, targets, `tags` incl experiment, aggregate pass@k). Derived by scanning `*/summary.json`, rebuildable, optional (Dashboard can glob summaries as fallback). JSONL (append per run), **not `index.json`**. -- **Repeat folder = `sample-N`, not `run-N`.** "run" is overloaded (`run_id` = the whole invocation). Rename `run-${attempt+1}` → `sample-1`, `sample-2`, … (matches margin `samples_per_case`/`sample_index`, pass@k, and AgentV's `repeat`; Inspect's `epoch` is the ML-jargon alt). Keep the metadata split: `sample_index` = repeats, `retry_index` = infra retries. +- **Cross-run index (rebuildable cache, not source of truth):** keep per-run `index.jsonl` (rows = cases); add a cross-run catalog `.agentv/results/.indexes/runs.jsonl` (already-reserved `.indexes` namespace) — **one row per run** (run_id, timestamp, targets, `tags` incl experiment, aggregate `pass_rate`/`pass_count`/`sample_count`, and explicit `pass_at_k` only when present). Derived by scanning `*/summary.json`, rebuildable, optional (Dashboard can glob summaries as fallback). JSONL (append per run), **not `index.json`**. +- **Repeat folder = `sample-N`, not `run-N`.** "run" is overloaded (`run_id` = the whole invocation). Rename `run-${attempt+1}` → `sample-1`, `sample-2`, … (matches margin `samples_per_case`/`sample_index`, explicit sampling metrics, and AgentV's `repeat`; Inspect's `epoch` is the ML-jargon alt). Keep the metadata split: `sample_index` = repeats, `retry_index` = infra retries. - **`experiment` has no *structural* privilege, but its *value* is auto-defaulted.** No storage dir (already `/`), no top-level field (`tags.experiment`), no special schema; tag keys sort **alphabetically**; the default grouping/compare **key** is a user preference (any tag — AgentV blesses none). `--experiment X` = sugar for `--tag experiment=X`. **The one convenience:** the harness auto-populates the `experiment` tag's **value** when unset, deriving it from the eval/suite name (ADR-0009: `--experiment` > authored `tags.experiment` > eval/suite name). So every run always has a meaningful `experiment` value and is groupable — without the author setting anything. This is a default *value*, not a privileged *key*. ## Decision — workspace resolver (provenance vs acquisition) diff --git a/docs/plans/promptfoo-aligned-eval-restructure.md b/docs/plans/promptfoo-aligned-eval-restructure.md index 48809b76f..cc5d50d86 100644 --- a/docs/plans/promptfoo-aligned-eval-restructure.md +++ b/docs/plans/promptfoo-aligned-eval-restructure.md @@ -7,6 +7,15 @@ workspaces and `workspace_mode` are removed from the user-facing contract, and `--workspace-path` / `execution.workspace_path` are the only static local workspace override. +Supersession note (2026-07-04): this plan predates the `av-kfik.28.1` grading +artifact decision. The current public `grading.json` contract is ADR-0017 plus +the active Beads: aggregate `pass`/`score`/`reason`, optional +`threshold`/`details`, always-present `graders[]`, nested `checks[]`, and no +public `assertion_results`, `assertions`, `passed`-only aliases, top-level +`checks`, or dynamic one-grader shortcut. Its broad pass@k language is also +stale unless it refers to an explicit sampling metric with a real `k`; use +`pass_rate`, `pass_count`, `sample_count`, and `passed`/`pass_any` otherwise. + Sources analyzed (all cloned locally, read-only): - promptfoo v0.121.17 — `/home/christso/projects/promptfoo-clone` (authoring format — the thing we clone) - Margin-Lab/evals — `/home/christso/projects/margin-lab-evals` (runner, I/O contracts, workspace, analytics) @@ -322,7 +331,12 @@ Borrow vercel's judge model, which is stronger than a prompt-stuffed rubric: - **Judge pinning** knob: `grader_target` = `{ agent?, model }` with self-grade default. AgentV already has `grader_target`; formalize the `{model}`-required pinning for apples-to-apples comparison. - **Gap to fix vs vercel:** they capture no token/cost — AgentV already does; keep it. -### 5.3 `grading.json` contract — reconciled with agentskills (owner-flagged main risk) +### 5.3 `grading.json` contract — stale historical design + +Stale after `av-kfik.28.1` and ADR-0017's 2026-07-04 update. Keep this section +as historical context only; implementation workers must use the current +`graders[]`/`checks[]` contract documented in ADR-0017 and in the Bead notes. + The output contract originates from agentskills' [evaluating-skills](https://github.com/agentskills/agentskills/blob/main/docs/skill-creation/evaluating-skills.mdx). Its `grading.json` is: ```json { "assertion_results": [ { "text": "…", "passed": true, "evidence": "…" } ], diff --git a/docs/plans/promptfoo-grading-reference-output-alignment.md b/docs/plans/promptfoo-grading-reference-output-alignment.md index 3882629d7..5ab11c1f5 100644 --- a/docs/plans/promptfoo-grading-reference-output-alignment.md +++ b/docs/plans/promptfoo-grading-reference-output-alignment.md @@ -1,6 +1,6 @@ # Promptfoo Grading Reference Output Alignment Plan -Status: draft review summary. Beads are the source of truth for scope, owner locks, acceptance, and closure. This document is a human-reviewable sequencing aid for `av-kfik.28` and must not replace the child Bead descriptions or acceptance criteria. +Status: draft review summary. Beads are the implementation source of truth for scope, owner locks, acceptance, sequencing, and closure. This document and the PRs that update it are human-reviewable summaries of the Beads; they must not replace child Bead descriptions, acceptance criteria, or notes. ## Summary @@ -11,12 +11,13 @@ Finalized contract: - Authored reference answers live in `vars.expected_output`. - `vars.expected_output` is passive reference data and does not imply grading. - Explicit `assert` entries own pass/fail. -- The Promptfoo-compatible low-friction pattern is a suite-level `default_test` / `defaultTest` `assert` entry with `type: llm-rubric` and `value` containing `{{ expected_output }}`. +- The Promptfoo-compatible low-friction pattern in AgentV wire format is a suite-level `default_test` `assert` entry with `type: llm-rubric` and `value` containing `{{ expected_output }}`. - `llm-rubric` should parse Promptfoo-style judge output `{reason, pass, score}`. -- Public grading artifacts use aggregate `{pass, score, reason, graders[]}`. -- Each grader uses `{name, type, pass, score, reason, checks?}`. -- Checks use `{id?, text, pass, score?, reason, evidence?}`. +- Public grading artifacts use aggregate `{pass, score, reason, threshold?, details?, graders[]}` with `graders[]` always present. +- Each grader uses `{name, type, pass, score, reason, threshold?, details?, checks?}`. +- Checks use `{id?, text, pass, score?, reason, evidence?}`, with `evidence` present only when distinct from `reason`. - Do not emit top-level `checks`, public `assertion_results`, a public `passed` alias, or a dynamic one-grader shortcut. +- Summary and index vocabulary should prefer `pass_rate`, `pass_count`, `sample_count`, and `passed`/`pass_any`; reserve `pass_at_k`/pass@k for explicit sampling metrics with a real `k`. Wire formats remain `snake_case`; internal TypeScript remains `camelCase` with boundary translation. @@ -28,7 +29,7 @@ Promptfoo evidence checked locally at clone commit `6bfc5a0c7f16f9c4717ac731d276 | --- | --- | --- | --- | --- | | `av-kfik.28` | Parent epic for Promptfoo-compatible reference answers and public grading result contract. | Parent under `av-kfik`; coordinates `av-kfik.28.1` through `av-kfik.28.7`. | This plan PR records branch/PR/commit only; do not close the epic or child Beads from this PR. | Draft plan PR first. Implementation PRs follow child Bead order and keep Beads canonical. | | `av-kfik.28.1` | Specify the final public grading result contract. | None inside this sub-epic. | Contract says aggregate `pass`, `score`, `reason`, always-present `graders[]`, nested `checks[]`, and no public legacy aliases. | First implementation/spec PR; blocks all artifact, SDK, parser, and dashboard work. | -| `av-kfik.28.2` | Migrate authored `expected_output` to `vars.expected_output` and reject normal authored top-level/test `expected_output`. | `av-kfik.28.1`; must avoid colliding with `av-kfik.27` input hard-deprecation and `av-kfik.15` broad codemod. | Parser/codemod/errors prove `vars.expected_output` is passive and explicit `assert` owns grading; examples use `default_test`/`defaultTest` `llm-rubric` with `{{ expected_output }}` where semantic grading is intended. | Stack after `av-kfik.15`, or proceed only on isolated expected-output parser/codemod paths that do not rewrite input fixtures/docs/examples already owned by `av-kfik.15`/`av-kfik.16`. | +| `av-kfik.28.2` | Migrate authored `expected_output` to `vars.expected_output` and reject normal authored top-level/test `expected_output`. | `av-kfik.28.1`; must avoid colliding with `av-kfik.27` input hard-deprecation and `av-kfik.15` broad codemod. | Parser/codemod/errors prove `vars.expected_output` is passive and explicit `assert` owns grading; examples use AgentV wire-format `default_test` `llm-rubric` with `{{ expected_output }}` where semantic grading is intended. | Stack after `av-kfik.15`, or proceed only on isolated expected-output parser/codemod paths that do not rewrite input fixtures/docs/examples already owned by `av-kfik.15`/`av-kfik.16`. | | `av-kfik.28.3` | Parse Promptfoo-compatible `llm-rubric` judge output and normalize it into the new contract. | `av-kfik.28.1`. | Tests cover `{reason, pass, score}`, coercion/failure cases, optional `checks[]`, rubric arrays, and no public legacy fields. | Can proceed after `av-kfik.28.1` in non-overlapping grader/parser areas, using prompt/vars fixtures that do not touch input hard-deprecation migration. | | `av-kfik.28.4` | Update SDK and script grader result APIs for `pass`/`reason`/`checks`. | `av-kfik.28.1`. | SDK schemas/helpers, script grader docs/examples, and tests use aggregate plus checks; internal APIs keep camelCase and translate at boundaries. | Can proceed after `av-kfik.28.1`; coordinate with `av-kfik.28.6` before artifact fixtures are regenerated. | | `av-kfik.28.6` | Rewrite run artifacts, JSONL/result exports, validators, and samples to stable `graders[]`/`checks[]`. | `av-kfik.28.1`, `av-kfik.28.3`, `av-kfik.28.4`. | Artifact contract tests cover single grader, multiple graders, no checks, scored checks, and failed grader parse errors; public artifacts reject legacy `assertion_results`/`passed`-only shape. | Artifact PR after parser and SDK PRs. Keep sample regeneration separate from input/example migration unless stacked after `av-kfik.15`. | @@ -51,7 +52,7 @@ README stays out of scope for this plan except to keep Promptfoo comparison in p Implementation workers should choose the smallest checks that prove their Bead acceptance criteria, but the full sub-epic needs: -- Unit, schema, parser, loader, and runtime tests for authored `vars.expected_output`, explicit `assert` ownership, rejection/migration of authored `expected_output`, and `default_test`/`defaultTest` inheritance. +- Unit, schema, parser, loader, and runtime tests for authored `vars.expected_output`, explicit `assert` ownership, rejection/migration of authored `expected_output`, and AgentV wire-format `default_test` inheritance. - Artifact contract tests for aggregate `{pass, score, reason, graders[]}`, nested `checks[]`, and rejection of public `assertion_results`, `passed`, top-level `checks`, or one-grader dynamic shapes. - SDK and script grader tests for aggregate-only results, checks with scores, checks without scores, and boundary translation between TypeScript internals and public wire format. - Docs and examples validation after docs/examples migrate to prompts plus vars and current grading artifact vocabulary.