docs(experiment): label runs via tags.experiment, drop top-level experiment - #1590
Merged
Conversation
…riment
A top-level `experiment:` key does not set the run's experiment namespace (it
falls through to the suite name / eval filename), so the docs were misleading.
Present `tags.experiment` as the single way to label a run/experiment, matching
the merged promptfoo-shaped tags support.
- README: eval examples and prose use `tags.experiment`; `defineEval` example
uses `tags: { experiment: ... }`; remove the misplaced `max_budget_usd` from
the target example (budget belongs under `evaluate_options`).
- docs: experiments.mdx + eval-files.mdx now describe `tags.experiment` as the
grouping label and state there is no top-level `experiment` field; fixed the
namespace precedence wording (`--experiment` > `tags.experiment` > default).
- sdk: `defineEval` `tags` accepts the promptfoo map form
(`Record<string,string>`) in addition to the selection list; `experiment` is
marked deprecated in favor of `tags.experiment`. Verified live that
`defineEval({ tags: { experiment } })` resolves the namespace via tags.
Confirmed via subagent that the Dashboard Experiments tab groups on the row
`experiment` field, which the resolver keeps equal to `tags.experiment`, so
`tags.experiment` drives the experiments view end to end.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploying agentv with
|
| Latest commit: |
98943df
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://88472810.agentv.pages.dev |
| Branch Preview URL: | https://av-4hzh-docs-tags-experiment.agentv.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #1589. A top-level
experiment:key does not set a run's experiment namespace — it falls through to the suite name / eval filename (verified live:ns_source: eval_filename). The docs presented it as the grouping label, which was misleading. This makestags.experimentthe single documented way to label a run.tags.experiment; thedefineEvalexample usestags: { experiment: ... }; removed the misplacedmax_budget_usdfrom the target example (budget belongs underevaluate_options).experiments.mdx+eval-files.mdxdescribetags.experimentas the grouping label and state there is no top-levelexperimentfield; corrected the namespace precedence wording to--experiment>tags.experiment> default.defineEvaltagsnow accepts the promptfoo map form (Record<string,string>) alongside the selection list, andexperimentis@deprecatedin favor oftags.experiment. Verified live thatdefineEval({ tags: { experiment } })resolves the namespace via tags (ns_source: tags).Dashboard verification
Confirmed (via investigation) that the Dashboard Experiments tab groups on the per-row
experimentfield (serve.tshandleExperiments→manifest.ts:337). Because the resolver keepsexperimentequal totags.experiment, settingtags.experimentmakes runs appear under that experiment in the Experiments tab end to end. (The Dashboard does not yet read the promptfootagsmap directly — grouping is via the lockstepexperimentfield.)Note
evaluate({ experiment })(programmatic run-label param) still works and is kept as-is — it is the run-label argument, not an eval-file top-level field.Tests
SDK:
defineEvalpreserves the map form and the list form oftags(toEvalYamlObject). Full SDK suite (92) green; core + CLI typecheck green.🤖 Generated with Claude Code