eval

eval

eval compares two prompt or system-prompt variants under the same model and execution settings. It runs control and treatment through the iii harness, evaluates each result with an iii function, and reports correctness together with tokens, cost, latency, function-call, trace, and span metrics.

Start an evaluation

{
  "dimension": "prompt",
  "model": {
    "model": "codex/gpt-5.6-luna",
    "provider": "openai-codex",
    "system_prompt_strategy": "override"
  },
  "control": {
    "label": "baseline",
    "prompt": "Reply with exactly OK.",
    "system_prompt": "Follow the user request exactly."
  },
  "treatment": {
    "label": "candidate",
    "prompt": "Return exactly the text OK.",
    "system_prompt": "Follow the user request exactly."
  },
  "evaluator": {
    "function_id": "eval::assert::exact",
    "arguments": {
      "expected": "OK"
    }
  }
}

Call eval::start with the request. It returns an evaluation_id immediately. Use eval::status for occasional progress checks and eval::result for the terminal report, or bind to the eval::completed trigger type.

Three runs per variant are scheduled by default. Their order alternates by pair to reduce order bias. A candidate is eligible only when every treatment run passes and its pass count does not regress against control. Efficiency metrics are descriptive and never select a winner automatically.

Public functions

• eval::start — validate, persist, and enqueue an evaluation.
• eval::list — list recent evaluations as lightweight summaries.
• eval::status — inspect progress without loading the full report.
• eval::result — read the normalized request and terminal comparison report.
• eval::cancel — cancel the active harness session and remaining runs.
• eval::delete — delete a terminal evaluation and its session indexes.
• eval::assert::exact — built-in deep JSON/string equality evaluator.

Evaluator functions receive the output, harness::metrics, run identity, and caller-supplied arguments. They return { passed, score?, reason?, details? }. Evaluators should be deterministic and idempotent because durable delivery is at-least-once.

Console UI

When the console worker is running, eval injects an eval page at #/ext/eval-benchmarks. The page creates prompt or system-prompt comparisons, tracks durable progress, restores recent reports after reload, and compares correctness, token, cost, latency, function-call, trace, and span metrics.

The model picker reads the live router::models::list catalog and falls back to manual model/provider entry when the catalog is unavailable. Exact-value evaluation is built in; any evaluator function can also be selected by id with JSON arguments. Harness policies and output/metadata options remain collapsed until needed.

Boundaries

The worker intentionally does not implement an agent loop, model router, metrics collector, trace collector, test DSL, or LLM judge. Those concerns remain in the harness, engine, and user-provided evaluator functions.