Skip to content

feat(evaluation): support scenario file references - #1696

Merged
christso merged 2 commits into
mainfrom
examples/av-7bnx-scenarios
Jul 6, 2026
Merged

feat(evaluation): support scenario file references#1696
christso merged 2 commits into
mainfrom
examples/av-7bnx-scenarios

Conversation

@christso

@christso christso commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add field-local top-level scenarios file refs/globs and flatten them before scenario lowering.
  • Add parser, validator, schema, and generated eval-writer schema coverage for scenario refs and malformed scenario files.
  • Add examples/features/scenarios with inline plus glob-loaded translation scenarios and a deterministic local CLI target.

Validation

  • bun install
  • Red check before implementation: bun apps/cli/src/cli.ts validate <tmp scenario eval> rejected scenarios[0] with "Scenario must be an object".
  • bun --filter @agentv/core build
  • bun apps/cli/src/cli.ts validate examples/features/scenarios/evals/suite.yaml -> valid 1 / invalid 0
  • bun run validate:examples -> 109 valid / 0 invalid
  • bun apps/cli/src/cli.ts eval run examples/features/scenarios/evals/suite.yaml --targets examples/features/scenarios/targets.yaml --target translation-cli --output /tmp/agentv-scenarios-run-LjjGCq -> PASS, 4/4 scored 100%
  • Manifest inspection confirmed lowered rows: inline-portuguese-hello, french-hello, spanish-hello-world, spanish-thank-you, all status ok, score 1
  • bun run lint
  • bun --filter @agentv/core test test/evaluation/loaders/jsonl-parser.test.ts test/evaluation/validation/eval-validator.test.ts test/evaluation/validation/eval-file-schema.test.ts test/evaluation/validation/eval-schema-sync.test.ts -> 217 pass / 0 fail

Notes

  • This stays field-local to scenarios; it does not add universal arbitrary-field file expansion.
  • No providerPromptMap, scenario result grouping, artifact primitive, or Dashboard behavior changes.
  • Live provider/grader dogfood was not used because the runtime change is parser/validator/example-only and the example uses deterministic local CLI output plus deterministic assertions.

@christso
christso marked this pull request as ready for review July 6, 2026 08:12
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 6, 2026

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: 253aa6d
Status: ✅  Deploy successful!
Preview URL: https://0d3ac008.agentv.pages.dev
Branch Preview URL: https://examples-av-7bnx-scenarios.agentv.pages.dev

View logs

@christso
christso merged commit 6d95fc9 into main Jul 6, 2026
8 checks passed
@christso
christso deleted the examples/av-7bnx-scenarios branch July 6, 2026 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant