feat(eval): ondemand simulate — replay a dataset, evaluate synchronously

[jariy17]

What

Adds eval ondemand simulate — the synchronous twin of batch-evaluation simulate. Replay a dataset against a runtime, then grade the resulting sessions client-side via the Evaluate API and print scores inline (no async job to poll).

Each dataset example already owns its invoker and its neutral ground truth. On-demand simulate reuses that replay wholesale and just swaps the grader: invokeDataset → getTracesForAgent → evaluate.

No new core method

The command is a pure handler composition of three existing CoreEvalClient methods:

invokeDataset (replay)  →  getTracesForAgent (gather)  →  evaluate (sync grade)

Composing them in core would break "a core method never calls a sibling client," so the chain lives in the handler. The only new logic is a small ground-truth adapter.

The ground-truth adapter

invokeDataset returns neutral InlineGroundTruth per session; evaluate wants EvaluationReferenceInput[] correlated by context.spanContext.sessionId. Session-level fields map 1:1:

function toReferenceInputs(s: InvokedSession): EvaluationReferenceInput[] {
  const gt = s.groundTruth;
  if (!gt?.assertions?.length && !gt?.expectedTrajectory) return [];
  return [{
    context: { spanContext: { sessionId: s.sessionId } },
    ...(gt.assertions?.length && { assertions: gt.assertions }),                 // {text}[]  1:1
    ...(gt.expectedTrajectory && { expectedTrajectory: gt.expectedTrajectory }), // {toolNames} 1:1
  }];
}

evaluate's groupRefsBySession re-groups the flat array by session, so this just works.

Command

agentcore eval ondemand simulate \
  --runtime-id <id> --payload-template '{"prompt":"{input}"}' \
  --dataset ./dataset.jsonl \
  --evaluator Builtin.Correctness --evaluator Builtin.Helpfulness \
  [--qualifier --header --bearer-token --user-id --dataset-version] --json

Same replay flags as batch; drops --name/--description/--kms-key-arn (no job). Returns { results[], sessionsRequested, sessionsEvaluated, examplesInvoked, examplesFailed }.

Deliberately deferred

v1 = session-level ground truth only (assertions + trajectory, which are exact). Per-turn expectedResponse is trace-level on the Evaluate API and would require correlating each turn to its traceId by order — the same alignment hazard as the batch sparse-turn bug. Left to a follow-up; batch-evaluation simulate still covers turn-level expectations.

Testing

• bun run typecheck: 0 errors. bun test: 1552 pass, 0 fail.
• Handler test (TestCoreClient): composition (invoke → traces → evaluate), sessionId passthrough, required-flag rejection, refuse-when-nothing-invoked, inline output shape, and a golden locking the adapted EvaluationReferenceInput[] (session-level 1:1, no-GT dropped, turn-level dropped).
• Design plan: https://artifactory.beta.harmony.a2z.com/view/7463f244-949d-4aa5-bf9f-c5cd6b370ace

Stacking

Draft, based on feat/eval-invoke-dataset-pr (#2032) since it reuses invokeDataset. Rebase onto refactor once #2032 lands. A real-account bug-bash (Strands deploy + on-demand simulate, adversarial) runs next; findings will be appended.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.11%. Comparing base (708c074) to head (94a16ac).

Additional details and impacted files

@@                       Coverage Diff                       @@
##           feat/eval-invoke-dataset-pr    #2071      +/-   ##
===============================================================
+ Coverage                        97.10%   97.11%   +0.01%     
===============================================================
  Files                              388      389       +1     
  Lines                            23171    23271     +100     
===============================================================
+ Hits                             22500    22600     +100     
  Misses                             671      671              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.