Skip to content

Align repeat authoring with Promptfoo numeric repeat - #1724

Merged
christso merged 3 commits into
mainfrom
feat/av-s96i-numeric-repeat
Jul 7, 2026
Merged

Align repeat authoring with Promptfoo numeric repeat#1724
christso merged 3 commits into
mainfrom
feat/av-s96i-numeric-repeat

Conversation

@christso

@christso christso commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • remove the public object-shaped repeat authoring surface (count, strategy, early_exit, cost_limit_usd) from core validation/parsing, CLI run-policy wiring, and SDK authoring
  • keep numeric evaluate_options.repeat: N and numeric scoped repeat overrides while preserving internal sample artifact metadata
  • update public docs, examples, ADR/plans, and agent guidance to describe numeric repeat only and mark the old strategy surface as superseded

Validation

  • bun --filter @agentv/core build
  • bun --filter @agentv/sdk build
  • bun --filter agentv build
  • bun test packages/sdk/test/eval-authoring.test.ts packages/core/test/evaluation/experiment.test.ts packages/core/test/evaluation/eval-inline-experiment.test.ts packages/core/test/evaluation/validation/eval-validator.test.ts packages/core/test/evaluation/validation/eval-file-schema.test.ts packages/core/test/evaluation/loaders/jsonl-parser.test.ts packages/core/test/evaluation/loaders/ts-eval-loader.test.ts apps/cli/test/eval.integration.test.ts
  • bun run lint
  • git diff --check
  • bun apps/cli/src/cli.ts validate examples/features/trials/evals/suite.yaml
  • bun --filter @agentv/web build
  • bun --filter @agentv/web check:routes

Live dogfood

  • Ran a temporary live eval using LOCAL_OPENAI_PROXY_BASE_URL=http://127.0.0.1:10531/v1, provider local-openai, grader provider local-openai-grader, and evaluate_options.repeat: 2.
  • Result: PASS, 1/1 scored 100%.
  • Bundle confirmed two samples: sample-1 and sample-2, with samples[].sample_index 0 and 1 and aggregate total_samples: 2.

Beads: av-s96i, av-s96i.1, av-s96i.2.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 7, 2026

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: 14a60e9
Status: ✅  Deploy successful!
Preview URL: https://bf4e2224.agentv.pages.dev
Branch Preview URL: https://feat-av-s96i-numeric-repeat.agentv.pages.dev

View logs

@christso
christso merged commit d89518e into main Jul 7, 2026
9 checks passed
@christso
christso deleted the feat/av-s96i-numeric-repeat branch July 7, 2026 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant