Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions .agentv/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,6 @@ eval_patterns:
- "**/evals/**/*.eval.yml"
- "**/evals/**/*.eval.ts"

# Publish completed run bundles to the git-backed results branch.
results:
path: .
branch: agentv/results/v1
auto_push: true

# Execution defaults (overridden by CLI flags)
execution:
verbose: false
Expand Down
23 changes: 21 additions & 2 deletions apps/web/src/content/docs/docs/next/evaluation/running-evals.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -457,7 +457,18 @@ Project-local YAML config takes precedence over home/global YAML config. AgentV

Within one config directory, AgentV reads `config.yaml` first and `config.local.yaml` second. The local overlay wins: plain objects deep-merge, arrays replace, and scalar values from `config.local.yaml` override `config.yaml`.

Use `config.yaml` for portable defaults and shared eval-definition fields that can be committed with the eval project. Use `config.local.yaml` for machine-local overrides such as private paths, local result remotes, Dashboard project registry entries, or temporary execution defaults. Project-local `config.local.yaml` is gitignored by default.
Use project-local `config.yaml` for portable defaults and shared
eval-definition fields that can be committed with the eval project. Do not put
operator-specific result publishing remotes in a shared eval project's
`.agentv/config.yaml`: different users and CI identities often publish to
different result repositories or do not publish at all. Put those project
registry and result-sync preferences in the user/global AgentV config pair
(`$AGENTV_HOME/config.yaml` or `$AGENTV_HOME/config.local.yaml`, default
`~/.agentv/...`) unless the project intentionally owns one canonical public
results repo. Use project-local `config.local.yaml` only for local overrides
that must stay beside the checkout, such as temporary execution defaults or
private workspace paths. Project-local `config.local.yaml` is gitignored by
default.

### YAML config (`config.yaml` plus optional `config.local.yaml`)

Expand Down Expand Up @@ -653,7 +664,11 @@ set AGENTV_HOME=D:\agentv-config

When unset, AgentV uses `~/.agentv`.

For local workspaces, put portable registry defaults in `$AGENTV_HOME/config.yaml` and machine-local project paths or result remotes in `$AGENTV_HOME/config.local.yaml`:
For local workspaces, prefer registering projects and result remotes in the
user/global AgentV config pair instead of committing them to the eval project.
Use `$AGENTV_HOME/config.yaml` for your normal personal or CI registry. Use
`$AGENTV_HOME/config.local.yaml` when the values are temporary, machine-local, or
an overlay on a shared mounted global config.

```yaml
projects:
Expand All @@ -664,6 +679,10 @@ projects:
branch: agentv/results/v1
```

Only commit `projects[].results` in a project-local `.agentv/config.yaml` when
that repository deliberately owns a shared canonical results destination, such
as a public demo results repo.

For larger registries, `$AGENTV_HOME/config.yaml` can point `projects` at a
bare project array:

Expand Down
11 changes: 11 additions & 0 deletions docs/adr/0007-conflict-free-results-sync-without-force-push.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,17 @@ Phoenix, hosted DB, or inbound webhook server at runtime.

Replace the force-push path with a two-layer, no-force-push design.

**Configuration placement — result publishing is user/global by default.**
Remote result sync describes where a particular operator or CI identity publishes
artifacts, so it is normally user-specific. Do not commit AgentV's own result
publishing remote, local result checkout path, or `auto_push` preference into the
AgentV repo or a shared eval project's `.agentv/config.yaml` unless the project
intentionally owns one canonical public results repo. Configure operator-specific
result remotes in the user/global AgentV config pair (`$AGENTV_HOME/config.yaml`
or `$AGENTV_HOME/config.local.yaml`, default `~/.agentv/...`) under `projects:`.
This keeps clones and contributors from inheriting someone else's publication
destination while still allowing CI or demos to mount a prepared global config.

**Layer 1 — auto-merge the common case.** On a non-fast-forward results push, run
a bounded `fetch → merge → push` loop using artifact-aware Git merge drivers:
`merge=union` for the append-only `index.jsonl` and a small `agentv-json`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -100,10 +100,14 @@ source root. Dashboard and other readers should display this value as `Eval`.
They should also display `test_id`, `target`, and `variant` when present so
users can distinguish rows with overlapping test IDs.

`suite` and `name` are display metadata. They may help humans group or label
results, and `suite` may participate in opaque row-id collision avoidance, but
they must not drive visible storage hierarchy, semantic routing, Dashboard
detail selection, rerun lookup, import identity, or artifact discovery.
`suite`, `name`, and any authored test `description` are display metadata. They
may help humans group, scan, or label results, and `suite` may participate in
opaque row-id collision avoidance, but they must not drive visible storage
hierarchy, semantic routing, Dashboard detail selection, rerun lookup, import
identity, or artifact discovery. `description` is especially not a substitute
for `test_id`: Promptfoo allows duplicate test descriptions, so AgentV keeps
`test_id` as the durable result identity and treats description as a non-unique
label.

`index.jsonl` is authoritative for all bundle-relative artifact paths. Per-row
directories are exposed with `result_dir`. Sidecar paths such as `test_dir`,
Expand Down
24 changes: 21 additions & 3 deletions docs/adr/0016-promptfoo-superset-eval-authoring-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,14 @@ scope and either inline or by `file://` reference. `workspace` and
workspace-named code or docs that model the same testbed concept are migration
debt unless they refer only to internal mutable directories or result storage.

Status note (2026-07-06): keep authored `tests[].id` / flattened `test_id` as
AgentV's durable test-case identity. Promptfoo-compatible `description` remains
display metadata only. PR-679 Promptfoo parity research found Promptfoo allows
duplicate test descriptions in the Dashboard, so `description` is not a safe
identity key for AgentV reruns, trends, result links, or cross-run comparison.
This deliberately diverges from Promptfoo's display-first row model while
keeping Promptfoo config importable.

## Context

AgentV's eval-authoring surface diverged from industry primitives. We are re-basing
Expand Down Expand Up @@ -89,9 +97,19 @@ keep AgentV's only where its semantics are genuinely better.**
`${{ ENV }}`. Rationale beyond superset-compat: `{{ env.VAR }}` **does not collide with
runtime shell `${VAR}`** — CLI-target commands can carry `$VAR`/`${VAR}` that must reach
the shell at runtime untouched; a `${ENV}` config sigil would clobber them.
8. **Optional test `id`**, layered identity: content identity = `test_id` (content hash,
derived when unauthored); governance/trend identity = an author `tag`/`metadata` key
(Dashboard keys comparison on this); display label = `description` → vars → `Test #n`.
8. **Test identity stays `id` / `test_id`; `description` is display-only.**
`tests[].id` is the preferred authored stable case identifier, and the
flattened `test_id` is the result/API/gate identity used for reruns, result
links, artifact lookup, trends, and Dashboard comparison. Imported or
ad-hoc Promptfoo-style rows that omit `id` may receive a deterministic
generated fallback, but generated IDs must be marked as generated and should
not be taught as the preferred curated-benchmark authoring style.
`description` is a human label and may duplicate across tests. Dashboard
display may prefer `description` where present, then vars, then `test_id` or
`Test #n`, but Dashboard selection and comparison must key on
`eval_path + test_id + target + variant`, not on description text. Content
fingerprints belong in separate provenance/change-detection metadata, not in
the public test identity.
9. **Keep AgentV where better**: first-class `expected_output` as passive gold/reference
data (DeepEval-aligned; not moved into `vars`, and not sent to target prompts
unless the author separately places it in `vars`). A specific grader may use
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ applies_when:

While adding Git-backed dashboard result remotes, the initial nested results repository config used `results.repo.url` for the Git endpoint URL and left `remote` available for the local Git remote-name implementation detail. That split was technically precise but did not match how users talk about portable Git sync configuration: in related tools such as Beads, `remote` names the shared endpoint URL, not a machine-local alias.

For a checked-in AgentV project config, requiring a local Git remote alias would make the file depend on per-machine setup. The portable contract only needs the endpoint URL, a local path, and the branch/ref that stores artifacts.
For a checked-in AgentV project config, requiring a local Git remote alias would make the file depend on per-machine setup. The portable contract only needs the endpoint URL, a local path, and the branch/ref that stores artifacts. Even with portable endpoint names, result publishing is often an operator preference: different users, CI jobs, and private-review flows may publish to different remotes or skip publishing entirely. Shared eval projects should not commit those user-specific result destinations unless the project intentionally owns one canonical results repo.

## Guidance

Expand All @@ -48,6 +48,12 @@ Internally, the runtime can still create or use a Git remote name such as `origi

Keep established neighboring contracts intact unless they are part of the same user-facing block. Source project repositories already use `projects[].repo.url`, so changing results repository config to `results.repo.remote` does not require renaming source project `repo.url`.

For user-specific result publishing, prefer the user/global AgentV config pair
(`$AGENTV_HOME/config.yaml` or `$AGENTV_HOME/config.local.yaml`, default
`~/.agentv/...`) over a repo-local `.agentv/config.yaml`. That keeps the eval
project portable while letting each operator choose their own result remote,
local checkout path, branch, and `auto_push` policy.

## Why This Matters

Portable config is commonly checked into a repository, copied into Docker development environments, or generated by setup scripts. A field that accepts a machine-local alias forces every machine and container to create the same Git remote name before the config works.
Expand All @@ -59,6 +65,7 @@ Using `remote` for the endpoint URL in the results repository block matches the
- A config file describes Git sync, artifact publishing, or result storage that should work from a fresh clone or container.
- A field could mean either a remote endpoint URL or a local Git remote-name alias.
- A setup script is expected to generate config that a developer can run with `bun run ...` from a latest checkout.
- A result remote reflects a user's publishing destination rather than a project-owned canonical artifact repository.

## Examples

Expand Down
Loading