diff --git a/.agentv/config.yaml b/.agentv/config.yaml index 083f7a508..7ad8b9982 100644 --- a/.agentv/config.yaml +++ b/.agentv/config.yaml @@ -11,12 +11,6 @@ eval_patterns: - "**/evals/**/*.eval.yml" - "**/evals/**/*.eval.ts" -# Publish completed run bundles to the git-backed results branch. -results: - path: . - branch: agentv/results/v1 - auto_push: true - # Execution defaults (overridden by CLI flags) execution: verbose: false diff --git a/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx b/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx index 1ba49e3f4..684e0ff80 100644 --- a/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx +++ b/apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx @@ -457,7 +457,18 @@ Project-local YAML config takes precedence over home/global YAML config. AgentV Within one config directory, AgentV reads `config.yaml` first and `config.local.yaml` second. The local overlay wins: plain objects deep-merge, arrays replace, and scalar values from `config.local.yaml` override `config.yaml`. -Use `config.yaml` for portable defaults and shared eval-definition fields that can be committed with the eval project. Use `config.local.yaml` for machine-local overrides such as private paths, local result remotes, Dashboard project registry entries, or temporary execution defaults. Project-local `config.local.yaml` is gitignored by default. +Use project-local `config.yaml` for portable defaults and shared +eval-definition fields that can be committed with the eval project. Do not put +operator-specific result publishing remotes in a shared eval project's +`.agentv/config.yaml`: different users and CI identities often publish to +different result repositories or do not publish at all. Put those project +registry and result-sync preferences in the user/global AgentV config pair +(`$AGENTV_HOME/config.yaml` or `$AGENTV_HOME/config.local.yaml`, default +`~/.agentv/...`) unless the project intentionally owns one canonical public +results repo. Use project-local `config.local.yaml` only for local overrides +that must stay beside the checkout, such as temporary execution defaults or +private workspace paths. Project-local `config.local.yaml` is gitignored by +default. ### YAML config (`config.yaml` plus optional `config.local.yaml`) @@ -653,7 +664,11 @@ set AGENTV_HOME=D:\agentv-config When unset, AgentV uses `~/.agentv`. -For local workspaces, put portable registry defaults in `$AGENTV_HOME/config.yaml` and machine-local project paths or result remotes in `$AGENTV_HOME/config.local.yaml`: +For local workspaces, prefer registering projects and result remotes in the +user/global AgentV config pair instead of committing them to the eval project. +Use `$AGENTV_HOME/config.yaml` for your normal personal or CI registry. Use +`$AGENTV_HOME/config.local.yaml` when the values are temporary, machine-local, or +an overlay on a shared mounted global config. ```yaml projects: @@ -664,6 +679,10 @@ projects: branch: agentv/results/v1 ``` +Only commit `projects[].results` in a project-local `.agentv/config.yaml` when +that repository deliberately owns a shared canonical results destination, such +as a public demo results repo. + For larger registries, `$AGENTV_HOME/config.yaml` can point `projects` at a bare project array: diff --git a/docs/adr/0007-conflict-free-results-sync-without-force-push.md b/docs/adr/0007-conflict-free-results-sync-without-force-push.md index c245a5087..cb74364d2 100644 --- a/docs/adr/0007-conflict-free-results-sync-without-force-push.md +++ b/docs/adr/0007-conflict-free-results-sync-without-force-push.md @@ -41,6 +41,17 @@ Phoenix, hosted DB, or inbound webhook server at runtime. Replace the force-push path with a two-layer, no-force-push design. +**Configuration placement — result publishing is user/global by default.** +Remote result sync describes where a particular operator or CI identity publishes +artifacts, so it is normally user-specific. Do not commit AgentV's own result +publishing remote, local result checkout path, or `auto_push` preference into the +AgentV repo or a shared eval project's `.agentv/config.yaml` unless the project +intentionally owns one canonical public results repo. Configure operator-specific +result remotes in the user/global AgentV config pair (`$AGENTV_HOME/config.yaml` +or `$AGENTV_HOME/config.local.yaml`, default `~/.agentv/...`) under `projects:`. +This keeps clones and contributors from inheriting someone else's publication +destination while still allowing CI or demos to mount a prepared global config. + **Layer 1 — auto-merge the common case.** On a non-fast-forward results push, run a bounded `fetch → merge → push` loop using artifact-aware Git merge drivers: `merge=union` for the append-only `index.jsonl` and a small `agentv-json` diff --git a/docs/adr/0009-eval-path-result-identity-and-default-experiment.md b/docs/adr/0009-eval-path-result-identity-and-default-experiment.md index f8abde8bd..7a81a095e 100644 --- a/docs/adr/0009-eval-path-result-identity-and-default-experiment.md +++ b/docs/adr/0009-eval-path-result-identity-and-default-experiment.md @@ -100,10 +100,14 @@ source root. Dashboard and other readers should display this value as `Eval`. They should also display `test_id`, `target`, and `variant` when present so users can distinguish rows with overlapping test IDs. -`suite` and `name` are display metadata. They may help humans group or label -results, and `suite` may participate in opaque row-id collision avoidance, but -they must not drive visible storage hierarchy, semantic routing, Dashboard -detail selection, rerun lookup, import identity, or artifact discovery. +`suite`, `name`, and any authored test `description` are display metadata. They +may help humans group, scan, or label results, and `suite` may participate in +opaque row-id collision avoidance, but they must not drive visible storage +hierarchy, semantic routing, Dashboard detail selection, rerun lookup, import +identity, or artifact discovery. `description` is especially not a substitute +for `test_id`: Promptfoo allows duplicate test descriptions, so AgentV keeps +`test_id` as the durable result identity and treats description as a non-unique +label. `index.jsonl` is authoritative for all bundle-relative artifact paths. Per-row directories are exposed with `result_dir`. Sidecar paths such as `test_dir`, diff --git a/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md b/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md index 91c196765..2612d70f5 100644 --- a/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md +++ b/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md @@ -31,6 +31,14 @@ scope and either inline or by `file://` reference. `workspace` and workspace-named code or docs that model the same testbed concept are migration debt unless they refer only to internal mutable directories or result storage. +Status note (2026-07-06): keep authored `tests[].id` / flattened `test_id` as +AgentV's durable test-case identity. Promptfoo-compatible `description` remains +display metadata only. PR-679 Promptfoo parity research found Promptfoo allows +duplicate test descriptions in the Dashboard, so `description` is not a safe +identity key for AgentV reruns, trends, result links, or cross-run comparison. +This deliberately diverges from Promptfoo's display-first row model while +keeping Promptfoo config importable. + ## Context AgentV's eval-authoring surface diverged from industry primitives. We are re-basing @@ -89,9 +97,19 @@ keep AgentV's only where its semantics are genuinely better.** `${{ ENV }}`. Rationale beyond superset-compat: `{{ env.VAR }}` **does not collide with runtime shell `${VAR}`** — CLI-target commands can carry `$VAR`/`${VAR}` that must reach the shell at runtime untouched; a `${ENV}` config sigil would clobber them. -8. **Optional test `id`**, layered identity: content identity = `test_id` (content hash, - derived when unauthored); governance/trend identity = an author `tag`/`metadata` key - (Dashboard keys comparison on this); display label = `description` → vars → `Test #n`. +8. **Test identity stays `id` / `test_id`; `description` is display-only.** + `tests[].id` is the preferred authored stable case identifier, and the + flattened `test_id` is the result/API/gate identity used for reruns, result + links, artifact lookup, trends, and Dashboard comparison. Imported or + ad-hoc Promptfoo-style rows that omit `id` may receive a deterministic + generated fallback, but generated IDs must be marked as generated and should + not be taught as the preferred curated-benchmark authoring style. + `description` is a human label and may duplicate across tests. Dashboard + display may prefer `description` where present, then vars, then `test_id` or + `Test #n`, but Dashboard selection and comparison must key on + `eval_path + test_id + target + variant`, not on description text. Content + fingerprints belong in separate provenance/change-detection metadata, not in + the public test identity. 9. **Keep AgentV where better**: first-class `expected_output` as passive gold/reference data (DeepEval-aligned; not moved into `vars`, and not sent to target prompts unless the author separately places it in `vars`). A specific grader may use diff --git a/docs/solutions/best-practices/name-portable-config-endpoints-by-user-intent.md b/docs/solutions/best-practices/name-portable-config-endpoints-by-user-intent.md index aefc2567c..0d8b35fe4 100644 --- a/docs/solutions/best-practices/name-portable-config-endpoints-by-user-intent.md +++ b/docs/solutions/best-practices/name-portable-config-endpoints-by-user-intent.md @@ -23,7 +23,7 @@ applies_when: While adding Git-backed dashboard result remotes, the initial nested results repository config used `results.repo.url` for the Git endpoint URL and left `remote` available for the local Git remote-name implementation detail. That split was technically precise but did not match how users talk about portable Git sync configuration: in related tools such as Beads, `remote` names the shared endpoint URL, not a machine-local alias. -For a checked-in AgentV project config, requiring a local Git remote alias would make the file depend on per-machine setup. The portable contract only needs the endpoint URL, a local path, and the branch/ref that stores artifacts. +For a checked-in AgentV project config, requiring a local Git remote alias would make the file depend on per-machine setup. The portable contract only needs the endpoint URL, a local path, and the branch/ref that stores artifacts. Even with portable endpoint names, result publishing is often an operator preference: different users, CI jobs, and private-review flows may publish to different remotes or skip publishing entirely. Shared eval projects should not commit those user-specific result destinations unless the project intentionally owns one canonical results repo. ## Guidance @@ -48,6 +48,12 @@ Internally, the runtime can still create or use a Git remote name such as `origi Keep established neighboring contracts intact unless they are part of the same user-facing block. Source project repositories already use `projects[].repo.url`, so changing results repository config to `results.repo.remote` does not require renaming source project `repo.url`. +For user-specific result publishing, prefer the user/global AgentV config pair +(`$AGENTV_HOME/config.yaml` or `$AGENTV_HOME/config.local.yaml`, default +`~/.agentv/...`) over a repo-local `.agentv/config.yaml`. That keeps the eval +project portable while letting each operator choose their own result remote, +local checkout path, branch, and `auto_push` policy. + ## Why This Matters Portable config is commonly checked into a repository, copied into Docker development environments, or generated by setup scripts. A field that accepts a machine-local alias forces every machine and container to create the same Git remote name before the config works. @@ -59,6 +65,7 @@ Using `remote` for the endpoint URL in the results repository block matches the - A config file describes Git sync, artifact publishing, or result storage that should work from a fresh clone or container. - A field could mean either a remote endpoint URL or a local Git remote-name alias. - A setup script is expected to generate config that a developer can run with `bun run ...` from a latest checkout. +- A result remote reflects a user's publishing destination rather than a project-owned canonical artifact repository. ## Examples