Skip to content

Enforce orchestration evidence and add evaluation benchmarks#6

Merged
whchi merged 2 commits into
mainfrom
codex/orchestration-evidence-evals
Jul 15, 2026
Merged

Enforce orchestration evidence and add evaluation benchmarks#6
whchi merged 2 commits into
mainfrom
codex/orchestration-evidence-evals

Conversation

@whchi

@whchi whchi commented Jul 15, 2026

Copy link
Copy Markdown
Owner

What changed

  • enforce delegation-scoped evidence, observed file-change tracking, configurable envelope enforcement, and operator-declared verifier commands
  • add routing-contract, checker-recall, mutation-fixture, and statistical benchmark evaluation surfaces
  • replace the simple four-domain prompts with one canonical delegation-required task catalog and hard-fail zero-child extraction
  • document the runtime decisions in ADR 0005 and the evaluation protocol in ADR 0006

Why

The existing runtime could trust maker self-reporting, lose evidence attribution across child sessions, and discover malformed envelopes only after execution. The benchmark pipeline also used single runs and default-pass labels that could overstate quality.

Impact

Runtime verification and trace diagnostics become stricter and more observable. Benchmark results require explicit acceptance data, report cost/time before token detail, and can evaluate routing and checker quality independently.

Validation

  • bun test — 193 passed, 0 failed
  • bun run build — passed
  • git diff --check — passed

@whchi
whchi marked this pull request as ready for review July 15, 2026 11:50
@whchi
whchi merged commit c74b85e into main Jul 15, 2026
1 check passed
@whchi
whchi deleted the codex/orchestration-evidence-evals branch July 15, 2026 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant