diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..3550bda --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,19 @@ +name: CI + +on: + push: + branches: [main] + pull_request: + +jobs: + verify: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: oven-sh/setup-bun@v2 + with: + bun-version: latest + - run: bun install --frozen-lockfile + - run: bun test + - run: bun run build + - run: git diff --check diff --git a/.gitignore b/.gitignore index fa0e315..5234578 100644 --- a/.gitignore +++ b/.gitignore @@ -5,4 +5,3 @@ task_plan.md node_modules/ findings.md node-compile-cache -bun.lock diff --git a/AGENTS.md b/AGENTS.md index 5da0d81..dd31d4c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -147,7 +147,8 @@ Your Legion uses direct specialist routing rather than a category-first runtime. - Override or disable specific declared components with `domains....path` or `false`. - A same-id override replaces the declared file path; overrides do not add components that are absent from `DOMAIN.md`. - Domain ids and component ids use the same kebab-case style as agent names. -- `bunx @whchi/your-legion create-domain ` creates `DOMAIN.md` only for a new custom id; pass `--components decisions,skills` to scaffold selected facets, and `--enable` to write it into `legionaries.yaml`. It must reject existing global domains and bundled domain ids. +- `bunx @whchi/your-legion create-domain ` creates `DOMAIN.md` and `updates.md` for a new custom id; pass `--components decisions,skills` to scaffold selected facets, and `--enable` to write it into `legionaries.yaml`. It must reject existing global domains and bundled domain ids. +- `bunx @whchi/your-legion domain-update ...` appends pending reusable knowledge candidates to `updates.md`; promotion into runtime still requires moving accepted knowledge into `DOMAIN.md`-listed components. - Runtime evidence records `delegation` and `domain-read` events. Use `bunx @whchi/your-legion doctor --worktree .` for diagnostics; it fails when `DOMAIN.md` declarations are invalid or declared domain refs/skills were not read, and it reports domain usage stats. - Use `bunx @whchi/your-legion domain-scenarios` and `bunx @whchi/your-legion doctor --worktree . --scenarios` for the fixed coding, marketing, finance, accounting, and mixed-domain validation set. @@ -157,6 +158,10 @@ Your Legion uses direct specialist routing rather than a category-first runtime. - A loop defines `description`, `objective`, `trigger`, `inbox_path`, optional domain evidence, agent roles, worktree policy, verification, and connector mode. - `inbox_path` must be repo-relative and points to the durable human-readable loop state file. - `builder` is the default maker and `verifier` is the default checker. +- Loop invocation is explicit-only. +- Start loop work only when the user names a configured loop id or provides a generated `loop-prompt` Task Context Envelope. +- The orchestrator must not set `Loop` from task intent, topic, similarity, or the Loop Catalog alone. +- If the user asks to use loop engineering without naming a configured loop id, ask which configured loop to run. - `create-loop --preset ` writes a loop config entry and creates `docs/legion-loops/.md` in the selected worktree. - `loop-presets`, `loop-prompt`, and `loop-runs` are the user-facing loop DX path. - `loops` lists configured loops. diff --git a/README.md b/README.md index 29ce09b..8a81506 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,28 @@ # Your Legion +[![CI](https://github.com/whchi/your-legion/actions/workflows/ci.yml/badge.svg)](https://github.com/whchi/your-legion/actions/workflows/ci.yml) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/whchi/your-legion) -An OpenCode plugin that improves multi-agent and loop-oriented work with bounded specialists, per-agent provider/model mapping, structured context handoff, and evidence-backed diagnostics. +An OpenCode plugin for multi-agent work you can audit. Most multi-agent setups let the agent that did the work also declare it done; Your Legion separates the maker (`builder`) from the checker (`verifier`), records what each delegation declared and what was actually read as trace events, and turns that evidence into pass/fail diagnostics. -Your Legion keeps OpenCode as the execution harness. It injects a small protected agent set, routes each turn to the right specialist, and lets operators choose different providers or models for routing, planning, discovery, documentation lookup, and execution. +The verification core is a completion ledger. A maker that changed files must report a `Verification status` block, an unverified claim routes through an independent `verifier` pass, and `trace-check` fails a run whose ledger says unverified file changes were shipped without one: -Task Context Envelopes keep delegation explicit and compact. Domain Packs provide selective expert context. Legion Loops define recurring or goal-driven engineering workflows with durable inboxes and maker/checker verification. Trace and doctor commands validate domain and loop evidence after the fact. +```text +- Verification status: + - Files changed: yes + - Verifier pass: not-run + - Status: unverified + - Reason: test runner unavailable in this environment +``` + +```text +$ bunx @whchi/your-legion trace-check --worktree . +verification-report [unverified-file-changes]: maker reported file changes with status unverified and no verifier delegation followed +``` + +The per-agent provider/model map serves the same goal: the checker can run on a different model than the maker, so they do not share correlated blind spots (`doctor` warns when a loop's maker and verifier resolve to the same model). + +Around that core, Your Legion keeps OpenCode as the execution harness and adds a small protected specialist set with per-agent provider/model mapping, Task Context Envelopes for explicit compact delegation, Domain Packs for selective expert context, and Legion Loops for recurring maker/checker workflows with durable inboxes. The benchmark protocol is deliberately allowed to conclude that orchestration was not worth it — and the committed 2026-07-04 measured run says exactly that for small tasks (same-provider orchestration was +85–146% tokens for no quality gain; a cheaper mixed-provider map matched native cost once); see [`ORCHESTRATOR_BENCHMARK.md`](./docs/ORCHESTRATOR_BENCHMARK.md). ![](docs/architecture.svg) @@ -14,12 +30,13 @@ Task Context Envelopes keep delegation explicit and compact. Domain Packs provid Use Your Legion when: -- You want OpenCode to route work across specialists more consistently. -- You want a simple per-agent provider/model map instead of a role-play agent team. -- You have project or domain knowledge that agents should use selectively. -- You want recurring engineering loops with explicit state, maker/checker separation, and diagnostics. +- You want completion claims backed by a maker/checker ledger and trace evidence, not self-reporting. +- You want the checker on a different provider or model than the maker. - You want troubleshooting evidence when a domain-enabled task did not use the expected context. -- You want to compare native OpenCode execution against an orchestrated multi-agent path. +- You want recurring engineering loops with explicit state, maker/checker separation, and diagnostics. +- You have project or domain knowledge that agents should use selectively. +- You want OpenCode to route work across specialists more consistently. +- You want to compare native OpenCode execution against an orchestrated multi-agent path, with the comparison allowed to go either way. It is not a standalone agent platform or a public domain-pack ecosystem. The goal is a lightweight plugin that improves OpenCode's multi-agent workflow. @@ -50,6 +67,15 @@ Explore where Your Legion builds the runtime agent config. The `orchestrator` should route repo discovery requests to `explorer`. For a clear code change, ask for the change directly; the orchestrator should route execution to `builder`, and `builder` should gather the needed repo context itself. +Then check the evidence trail instead of taking the transcript's word for it: + +```bash +bunx @whchi/your-legion doctor --worktree . +bunx @whchi/your-legion trace-check --worktree . +``` + +`doctor` validates domain declarations, loop catalogs, and runtime trace evidence, and prints usage stats. `trace-check` fails when a maker reported unverified file changes and no verifier pass followed. + Use these docs next: - Install and uninstall details: [`INSTALLATION.md`](./docs/INSTALLATION.md) @@ -61,6 +87,7 @@ Use these docs next: - Copy-paste examples: [`EXAMPLES.md`](./docs/EXAMPLES.md) - Development notes: [`DEVELOPMENT.md`](./docs/DEVELOPMENT.md) - Academic references behind the domain/runtime design: [`academic-papers-summary.md`](./docs/academic-papers-summary.md) +- Essay influences behind loops and Domain Packs: [`design-influences.md`](./docs/design-influences.md) ## Install @@ -157,7 +184,9 @@ Custom agent definitions are discovered from bundled package examples and from t Domain descriptions and skills are injected into agent prompts as a Domain Catalog with namespaced entries such as `marketing/campaign-brief`. Routing agents pass relevant `Domain refs` and `Domain skills` in the Task Context Envelope; target specialists read the exact configured paths. Your Legion does not register domain skills as top-level harness skills. -Delegations use a compact Task Context Envelope with `Scenario`, `Loop`, `Loop run`, `Loop status`, `Objective`, `Active domains`, `Domain refs`, `Domain skills`, `Context refs`, `Constraints`, `Expected output`, `Verification`, `Completion claim`, `Verification commands`, and `Verification outcome`. The orchestrator compares the task with the Loop Catalog and Domain Catalog, then passes only the matching loop id, run id, and domain evidence. If no loop applies, it writes `Loop: none`. If no domain is configured or no domain description clearly matches, it should use no-domain delegation: `Active domains: none`, `Domain refs: none`, and `Domain skills: none`. +Delegations use a compact Task Context Envelope with `Scenario`, `Loop`, `Loop run`, `Loop status`, `Objective`, `Active domains`, `Domain refs`, `Domain skills`, `Context refs`, `Constraints`, `Expected output`, `Verification`, `Completion claim`, `Verification commands`, and `Verification outcome`. The orchestrator uses loop context only for explicit loop runs, compares the task with the Domain Catalog, then passes only the selected loop id, run id, and domain evidence. If no loop applies, it writes `Loop: none`. If no domain is configured or no domain description clearly matches, it should use no-domain delegation: `Active domains: none`, `Domain refs: none`, and `Domain skills: none`. + +Loop invocation is explicit-only. Use `loop-prompt ` or name a configured loop id for deliberate runs. The orchestrator should not set `Loop` from task intent, topic, similarity, or the Loop Catalog alone; all other requests use `Loop: none`. Your Legion records warn-only domain usage evidence under `~/.config/opencode/your-legion/traces/`. When troubleshooting domain setup, use `bunx @whchi/your-legion doctor --worktree .` for static domain catalog validation, runtime trace validation, and domain usage stats. Use `bunx @whchi/your-legion trace` when you need raw delegation and domain-read events, or `trace --summary` for grouped delegation evidence. See [`DOMAIN_OBSERVABILITY.md`](./docs/DOMAIN_OBSERVABILITY.md) for the full validation workflow. @@ -173,7 +202,7 @@ For hands-on examples of custom agents, marketing domain packs, mixed coding plu ## Routing Model -Your Legion uses direct specialist routing. +Your Legion uses direct specialist routing along two orthogonal axes: **agents are work-mode boundaries** (how to work — route, plan, build, verify, explore, look up docs), and **Domain skills are capability boundaries** (what expertise the work needs — coding, marketing, finance, and so on). Agents stay domain-neutral; professional capability is injected as Domain skills, never baked into an agent. Add new expertise as a Domain Pack; reserve new agents for genuinely new work modes. - The `orchestrator` classifies each turn into one dominant intent and chooses a concrete subagent. - Those intents are routing heuristics, not runtime categories or model profiles. @@ -184,7 +213,8 @@ Your Legion uses direct specialist routing. ## Commands - `bunx @whchi/your-legion install [--config-dir ] [--domains ] [--add-domains ]`: installs or refreshes the plugin registration. First install writes `legionaries.yaml` with `coding` enabled and materializes enabled bundled domain packs under `~/.config/opencode/your-legion/domains/`. Reinstall without domain flags preserves existing config. `--domains` replaces the enabled domain list; `--add-domains` merges into it. -- `bunx @whchi/your-legion create-domain [--config-dir ] [--components workflows,decisions,examples,skills] [--enable]`: scaffolds a new global domain pack. By default it creates only `DOMAIN.md`; use `--components` to add selected optional folders and matching placeholder files, and `--enable` to write the domain into `legionaries.yaml`. Existing global domains and bundled domain ids are rejected. +- `bunx @whchi/your-legion create-domain [--config-dir ] [--components workflows,decisions,examples,skills] [--enable]`: scaffolds a new global domain pack with `DOMAIN.md` and `updates.md`. Use `--components` to add selected optional folders and matching placeholder files, and `--enable` to write the domain into `legionaries.yaml`. Existing global domains and bundled domain ids are rejected. +- `bunx @whchi/your-legion domain-update --type --title [--config-dir ] [--promote-to ]`: appends a pending reusable knowledge candidate from stdin to the domain's `updates.md`. - `bunx @whchi/your-legion create-loop [--preset ] [--worktree ] [--config-dir ] [--description ] [--objective ] [--verification ]`: creates a configured Legion Loop and repo-local inbox. - `bunx @whchi/your-legion loops [--config-dir ]`: lists configured Legion Loops. - `bunx @whchi/your-legion loop-presets`: lists quick-start templates such as `ci-triage`, `issue-triage`, `docs-refresh`, and `release-check`. @@ -192,7 +222,8 @@ Your Legion uses direct specialist routing. - `bunx @whchi/your-legion loop-runs [--worktree ] [--config-dir ] [--loop ]`: groups loop run completion ledger evidence from trace events. - `bunx @whchi/your-legion doctor [--worktree ] [--config-dir ] [--scenarios]`: troubleshoots domain and loop setup. By default it validates `DOMAIN.md` declarations, loop catalogs, runtime trace evidence, and usage stats; `--scenarios` verifies the fixed domain scenario set. - `bunx @whchi/your-legion trace [--worktree ] [--config-dir ] [--limit ] [--summary]`: prints recent domain usage evidence for a workspace/project path; `--summary` groups declared refs, matching reads, and warnings by delegation. -- `bunx @whchi/your-legion trace-check [--worktree ] [--config-dir ]`: low-level trace validation for contract warnings and declared domain refs or skills that were not read. +- `bunx @whchi/your-legion trace-check [--worktree ] [--config-dir ] [--require-evidence]`: low-level trace validation for contract warnings, declared domain refs or skills that were not read, and maker/checker ledger integrity (unverified file changes without a following verifier pass fail). `--require-evidence` additionally fails when no delegation or loop-run-report evidence exists at all. +- `bunx @whchi/your-legion benchmark-summarize --metrics `: summarizes an exported benchmark metrics file (JSON array or JSONL of session metrics) into the quality-plus-token outcome taxonomy. See [`ORCHESTRATOR_BENCHMARK.md`](./docs/ORCHESTRATOR_BENCHMARK.md). - `bunx @whchi/your-legion domain-scenarios`: prints the fixed domain scenario prompts. - `bunx @whchi/your-legion domain-scenario-check [--worktree ] [--config-dir ]`: low-level fixed scenario validation; `doctor --scenarios` is the preferred entrypoint. diff --git a/bun.lock b/bun.lock new file mode 100644 index 0000000..134a38c --- /dev/null +++ b/bun.lock @@ -0,0 +1,26 @@ +{ + "lockfileVersion": 1, + "configVersion": 0, + "workspaces": { + "": { + "name": "omo-clone", + "dependencies": { + "yaml": "^2.8.4", + }, + "devDependencies": { + "@types/bun": "^1.3.14", + }, + }, + }, + "packages": { + "@types/bun": ["@types/bun@1.3.14", "", { "dependencies": { "bun-types": "1.3.14" } }, "sha512-h1hFqFVcvAvD9j9K7ZW7vd82aSA+rTdznZa+5bwvCwqSB1jmmfLcbIWhOLx1/+boy/xmjgCs/OMUL8hRJSmnPw=="], + + "@types/node": ["@types/node@25.9.1", "", { "dependencies": { "undici-types": ">=7.24.0 <7.24.7" } }, "sha512-xfrlY7UD5rMJk3ZVJP8BNzS28J36YJg+xp+LPXV1TdWxr8uMH5A860QNxYDGQe/ylDSgjxE52Q9VnO7p75tJxg=="], + + "bun-types": ["bun-types@1.3.14", "", { "dependencies": { "@types/node": "*" } }, "sha512-4N0ig0fEomHt5R0KCFWjovxow98rIoRwKolrYdCcknNwMekCXRnWEUvgu5soYV8QXtVsrUD8B95MBOZGPvr6KQ=="], + + "undici-types": ["undici-types@7.24.6", "", {}, "sha512-WRNW+sJgj5OBN4/0JpHFqtqzhpbnV0GuB+OozA9gCL7a993SmU+1JBZCzLNxYsbMfIeDL+lTsphD5jN5N+n0zg=="], + + "yaml": ["yaml@2.8.4", "", { "bin": "bin.mjs" }, "sha512-ml/JPOj9fOQK8RNnWojA67GbZ0ApXAUlN2UQclwv2eVgTgn7O9gg9o7paZWKMp4g0H3nTLtS9LVzhkpOFIKzog=="], + } +} diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index dec4838..9c9a940 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -145,6 +145,19 @@ custom_agents: Custom agents run as `subagent`. Any permission key not listed in the YAML is set to `deny`. Custom agents cannot use system agent names such as `builder`, `planner`, or `explorer`. A worktree definition with the same name as a bundled custom agent overrides the bundled definition; the model mapping still comes from the global runtime config. +## Verification + +Post-execution verification controls whether the orchestrator routes an independent `verifier` pass when `builder` reports file changes it did not verify. + +```yaml +verification: + default_check: true +``` + +- `default_check` (boolean, default `true`): when `true`, the orchestrator routes a `verifier` pass before reporting completion for any `builder` deliverable that reports `Files changed: yes` with `Status: unverified`. Set it to `false` to keep verification opt-in (the previous behavior): the orchestrator then verifies only for loops or when the user explicitly asks. + +`builder` always returns a Verification status ledger (files changed; verifier pass ran or not-run; `verified` / `unverified` / `verification-skipped` with a reason), so the maker/checker split stays visible even when `default_check` is off. See [ADR 0003](./adr/0003-enforced-verification-and-maker-checker-integrity.md). + ## Domain Packs Domain packs provide a shared Domain Catalog for the same system and custom agents. They are self-describing capabilities with reusable workflows, decisions, examples, and domain-local skills. They are not registered as harness-level skills and they are not automatically active task memory. diff --git a/docs/DOMAIN_OBSERVABILITY.md b/docs/DOMAIN_OBSERVABILITY.md index 05f0ef6..1e35388 100644 --- a/docs/DOMAIN_OBSERVABILITY.md +++ b/docs/DOMAIN_OBSERVABILITY.md @@ -21,6 +21,7 @@ Runtime does not classify domains for the agent. Runtime only records and valida - `domain-read` event: which declared domain component files the delegated agent actually read. - `loopID`: optional delegation evidence tying a task to a configured Legion Loop. - `loop-run-report` event: a maker or verifier completion report for one `Loop run`. +- `verification-report` event: a non-loop maker's `Verification status:` ledger block (files changed, verifier pass, verified/unverified/verification-skipped, reason), captured from task output. - `doctor`: the main diagnostics command. It validates static `DOMAIN.md` declarations, loop catalogs, runtime trace evidence, and reports usage stats. - `loop-runs`: grouped completion ledger output for loop run status, claims, commands, and outcomes. - `trace-check`: low-level trace validation for contract warnings or declared domain refs/skills that were not read. @@ -249,8 +250,11 @@ delegation: unknown domain skill: coding/missing-skill delegation: active domain must include responsibility: coding delegation: declared domain ref was not read: coding/implementation-loop delegation: declared domain skill was not read: marketing/campaign-brief +verification-report [unverified-file-changes]: maker reported file changes with status unverified and no verifier delegation followed ``` +`trace-check` also enforces the non-loop maker/checker ledger: a `verification-report` event with `Files changed: yes` and `Status: unverified` fails unless a `verifier` delegation follows in the same session. `verification-skipped` with a reason is an operator opt-out and does not fail. + `trace-check` is the lower-level answer to "did it actually use the declared domain context?" A delegation that declares `Domain refs: coding/implementation-loop` or `Domain skills: marketing/campaign-brief` is not accepted unless a matching `domain-read` event appears. ## Run Fixed Scenario Validation @@ -364,6 +368,15 @@ bunx @whchi/your-legion doctor --worktree . --scenarios `doctor` exits non-zero on failure, so it can be used in scripted acceptance flows after the interactive OpenCode prompts have produced trace evidence. +## Warn vs Block Policy + +Doctor and trace signals fall into two classes: + +- **Blocking (FAIL, non-zero exit):** completion-integrity evidence. A loop `maker-complete` run without a matching `verifier-complete`, a loop maker delegation without verifier evidence, a maker and verifier that are the same agent, and missing declared domain component files all fail `doctor`. A non-loop `verification-report` with unverified file changes and no following `verifier` delegation fails `trace-check`. +- **Warn-only (does not fail):** advisory hygiene. Declared domain refs or skills that were not read, a loop maker and verifier that share the same model, a custom agent that mixes domain capability into a work-mode agent, and manual connector modes are warnings. + +The principle: maker/checker ledger integrity is blocking; domain-evidence completeness and cross-model hygiene are advisory. See [ADR 0003](./adr/0003-enforced-verification-and-maker-checker-integrity.md). + ## Limitations - Runtime warnings are warn-only during normal OpenCode execution. diff --git a/docs/DOMAIN_PACK_AUTHORING.md b/docs/DOMAIN_PACK_AUTHORING.md index 6c0eddb..7d97ffd 100644 --- a/docs/DOMAIN_PACK_AUTHORING.md +++ b/docs/DOMAIN_PACK_AUTHORING.md @@ -36,6 +36,32 @@ The component folders are optional facets: Do not create empty folders for symmetry. If a file is not listed in `DOMAIN.md`, runtime treats it as absent even when it exists on disk. +`create-domain` also creates `updates.md`. This file is a candidate inbox for reusable domain knowledge discovered during long-running work. It is not loaded into runtime and does not affect routing until a candidate is manually promoted into a `DOMAIN.md`-listed component. + +## Domain Update Candidates + +Write to `updates.md` when a task produces knowledge that should change future work in the same domain: + +- reusable: the same kind of task is likely to happen again; +- decision: a user, reviewer, or verifier made a rule future agents should follow; +- correction: existing docs, assumptions, or runtime understanding were wrong; +- workflow: the task produced a repeatable procedure; +- verification: the task produced a completion check future work should use. + +Do not write one-off command output, temporary branch state, unverified guesses, or task details that will not change future judgment. + +Use the CLI to append a pending candidate: + +```bash +echo "OpenCode may load the published plugin instead of local dist." \ + | bunx @whchi/your-legion domain-update coding \ + --type decision \ + --title "Plugin loading source" \ + --promote-to decisions/plugin-loading.md +``` + +Promotion is manual: move accepted candidates into `workflows/`, `decisions/`, `examples/`, or `skills/`, then list the promoted file in `DOMAIN.md`. Leave rejected or noisy candidates in `updates.md` only if the history is useful. + ## Write `DOMAIN.md` Keep the routing description semantic. Describe responsibility, not keyword triggers. diff --git a/docs/LEGION_LOOPS.md b/docs/LEGION_LOOPS.md index 28e8eed..952c7f4 100644 --- a/docs/LEGION_LOOPS.md +++ b/docs/LEGION_LOOPS.md @@ -142,6 +142,16 @@ loops: `connectors.targets`: External systems or resources associated with the loop. Keep it empty for manual loops; use explicit target names or URLs when external tooling consumes this metadata. +## Invocation Model + +Legion Loops are explicit-only. + +Start a loop only by naming a configured loop id directly or by generating a ready Task Context Envelope with `loop-prompt ` and pasting it into OpenCode. + +The orchestrator should not start a loop from task intent, topic, similarity, or the Loop Catalog alone. If the user asks to use loop engineering but does not name a configured loop id, the orchestrator should ask which configured loop to run. + +For every other request, the Task Context Envelope should use `Loop: none`. + ## Agent Flow When a task matches a configured loop, the orchestrator includes the loop id and run id in the Task Context Envelope. You normally generate this with `loop-prompt` instead of writing it by hand: @@ -158,14 +168,16 @@ Domain refs: coding/implementation-loop Domain skills: coding/make-code-change Context refs: docs/legion-loops/daily-ci-triage.md Constraints: Use required worktree isolation when the loop touches code or long-running state. -Expected output: Patch or findings, updated loop inbox, verification results, and a Loop Run Report. +Expected output: Patch or findings, updated loop inbox, verification results, Domain update candidates: list reusable domain knowledge candidates or none, and a Loop Run Report. Verification: bun test, bun run build, git diff --check Completion claim: none Verification commands: none Verification outcome: none ``` -`builder` is the maker. `verifier` is the checker. The checker should review the completion claim, diff, tests, loop inbox, and declared domain evidence before the loop is considered complete. +`builder` is the maker. `verifier` is the checker. The checker should review the completion claim, diff, tests, loop inbox, declared domain evidence, and Domain update candidates before the loop is considered complete. + +At the end of loop work, report Domain update candidates. Use `none` when the run produced no reusable domain knowledge. Otherwise append accepted candidates to the domain's `updates.md` with `domain-update`, then promote them manually into `workflows/`, `decisions/`, `examples/`, or `skills/` when they are stable. ## Loop Run Reports @@ -181,6 +193,14 @@ Verification commands: bun test, bun run build, git diff --check Verification outcome: passed ``` +Also report: + +```text +Domain update candidates: none +``` + +or candidate entries with domain, type, title, promote target, and evidence. + Valid statuses are `started`, `maker-complete`, `verifier-complete`, `blocked`, and `failed`. Valid verification outcomes are `passed`, `failed`, `not-run`, and `unknown`. Your Legion records these reports in the runtime trace as `loop-run-report` events. A run is not trusted as complete until a `maker-complete` report has matching `verifier-complete` evidence for the same `Loop` and `Loop run`. @@ -189,6 +209,7 @@ Your Legion records these reports in the runtime trace as `loop-run-report` even - The loop inbox exists at the configured repo-relative path. - Maker and verifier are separate agents. +- Maker and verifier do not resolve to the same model (warning): a checker on a different model avoids correlated blind spots. See [ADR 0003](./adr/0003-enforced-verification-and-maker-checker-integrity.md). - Verification commands are present. - Declared loop domain refs and skills exist in the enabled Domain Catalog. - Runtime loop delegations reference known loops. diff --git a/docs/ORCHESTRATOR_BENCHMARK.md b/docs/ORCHESTRATOR_BENCHMARK.md index b8a7a04..b0688d0 100644 --- a/docs/ORCHESTRATOR_BENCHMARK.md +++ b/docs/ORCHESTRATOR_BENCHMARK.md @@ -273,6 +273,86 @@ Expected orchestrated result: Measured same-model results are recorded below. The latest four-task run completed all four tasks, but several domain-envelope fields still produced trace warnings. +## Execution-Quality Task Set (Harder Suite) + +The four-domain set above is the **routing-cost baseline**: read-only, single-turn, deliberately the least favorable case for orchestration. It answers "what does the orchestrator layer cost when there is nothing to gain." It cannot show the value of independent verification, because there is no defect to catch and no rework to avoid. + +The **execution-quality suite** is a separate set that answers "does orchestration and an independent `verifier` produce fewer failures or fewer repair turns." Keep it separate from the routing-cost baseline: file edits, test runs, repair turns, and verification noise measure a different product question and must not be averaged into routing cost. + +Task-design criteria: + +- **Multi-step**: the task needs sequencing (plan then implement) or touches several files, so a single-shot answer is likely to miss a step. +- **Error-prone**: the task has a plausible-but-wrong shortcut a maker tends to take, so a checker has something real to catch. +- **Planted-defect (file-mutating)**: the task starts from code containing a subtle, seeded defect (off-by-one, wrong boundary, missing null/empty/error case). Acceptance measures whether the orchestrated path (maker plus independent `verifier`) catches or avoids the defect where the native single-agent path does not. + +Primary metrics for this suite are **pass rate**, **rework turns**, and **defects caught**, not tokens-per-pass alone. Tokens are still recorded, but a more-expensive run that catches a defect the native path shipped is `more-expensive-better`, which is the outcome this suite exists to detect. + +Example tasks (runnable prompts; measured 2026-07-04, see the results section below): + +#### `exec-coding-001` (planted defect) + +```text +Benchmark: yl-exec-quality-YYYYMMDD +Task: exec-coding-001 +Variant: + +In an isolated scratch file, deliver a correct clamp(value, low, high): return low when value < low, high when value > high, and value otherwise. Start from this draft, which has a subtle boundary defect and a test that passes anyway: + + function clamp(value, low, high) { + if (value < low) return low; + if (value > high) return low; // defect: should return high + return value; + } + // existing test: clamp(5, 0, 10) === 5 (passes; never exercises the upper bound) + +Fix the defect and add a focused test that fails on the original code. Return the corrected function, the new test, and its result. +``` + +Acceptance: the upper-bound defect is fixed and a test proves the upper bound. The native single-agent path tends to keep the green-but-insufficient test; the orchestrated path's independent `verifier` should flag the missing upper-bound coverage. Primary signal: defect caught (yes/no), then rework turns. + +#### `exec-coding-002` (multi-step refactor) + +```text +Benchmark: yl-exec-quality-YYYYMMDD +Task: exec-coding-002 +Variant: + +Change format(items) to accept an options object { trailingNewline } and update BOTH callers so each keeps its current behavior. Starting code: + + function format(items) { return items.join("\n") + "\n"; } + function report(items) { return format(items); } // must keep a trailing newline + function inline(items) { return "X:" + format(items); } // must NOT have a trailing newline + +Return the updated function and both callers, plus a check showing report() still ends with a newline and inline() does not. +``` + +Acceptance: both callers preserve their behavior (a naive single edit breaks one). Primary signal: both paths correct, then rework turns vs native. + +Each execution-quality run should carry a `providerProfile` (`same-provider` or `mixed-provider`) in its exported metrics so `benchmark-summarize` reports native-vs-same and native-vs-mixed separately. Extract paired sessions with `scripts/extract-benchmark-metrics.ts` (see below). See [ADR 0004](./adr/0004-orchestration-value-evidence.md) (D1, D2). + +### Measured Run: 2026-07-04 (3-way, both exec tasks) + +Committed fixture: `tests/fixtures/orchestration-benchmark/2026-07-04-exec-quality-3way.jsonl` (10 session rows). Model under test: `opencode-go/deepseek-v4-flash` for native and all same-provider agents; the mixed-provider run used the repo-default map (orchestrator `deepseek-v4-pro`, builder `kimi-k2.6`). Markers: `yl-exec-quality-20260704` (native), `...sp` (same-provider), `...mp` (mixed-provider); the suffix separates profiles at extraction time and the profile is recorded in `providerProfile`. + +| Task | Variant | Total tokens | vs native | Outcome | +|---|---|---:|---:|---| +| exec-coding-001 | native-builder | 61,884 | — | — | +| exec-coding-001 | same-provider orchestrated | 114,573 | +85.1% | more-expensive-not-better | +| exec-coding-001 | mixed-provider orchestrated | 60,282 | −2.6% | cheaper-same-quality | +| exec-coding-002 | native-builder | 30,293 | — | — | +| exec-coding-002 | same-provider orchestrated | 74,401 | +145.6% | more-expensive-not-better | +| exec-coding-002 | mixed-provider orchestrated | 58,251 | +92.3% | more-expensive-not-better | + +Rubric (scored from run transcripts): all six runs passed. Native caught the planted `clamp` defect and produced the upper-bound test on its own, and preserved both `format` callers, so this suite produced no quality separation at this difficulty — the `more-expensive-better` outcome the suite exists to detect did not appear. Notable observations: + +- **No verifier pass was triggered.** In every orchestrated run, `builder` self-verified (ran its checks, reported verified), so the default-check rule in ADR 0003 D2 correctly did not force an independent `verifier` delegation. A suite that wants to measure verifier value needs tasks where the maker's self-check is plausibly green but wrong. +- **Mixed-provider beat same-provider on cost in both tasks**, and matched native on exec-coding-001. The gap came mostly from cache-read volume and reasoning tokens, not output length. +- **`trace-check` on the benchmark worktree surfaced real warnings**: the orchestrator declared `Domain skills: coding/make-code-change` for self-contained scratch tasks whose builder never read the skill (`missing-domain-skill-read`), plus one malformed `Active domains` entry. Routing worked; domain declaration was over-eager. + +Caveat on reading `benchmark-summarize` output for this fixture: the top-level `Per task` table sums **both** orchestrated profiles into one `your-legion` number because the fixture holds two orchestrated runs per task. Use the `By provider profile` section for the 3-way answer; the per-task table is only meaningful for single-profile exports. + +Honest conclusion so far: on tasks this small, the orchestration layer is pure overhead unless the model map itself is cheaper than the native model. The open question is unchanged in direction but sharper in shape: the suite needs harder, multi-file, deceptively-green tasks before the verifier can demonstrate (or fail to demonstrate) `more-expensive-better`. + ## OpenCode Token Extraction The local OpenCode session DB used for this project is: @@ -363,6 +443,18 @@ Interpretation: - Domain envelope quality is still unreliable. Three rows had trace warnings from malformed `Active domains`; `finance-001` and `accounting-001` show the model still tends to put responsibilities or comma-separated topics where a single `domain-id: responsibility` entry is required. - All four `trace-check --worktree ` commands returned pass, even though the trace file contained warnings. In this local run, trace events recorded `worktree: "/"`, so per-worktree `trace-check` did not catch those warnings. This is an observability bug to fix before using trace-check as benchmark acceptance evidence. +## Summarizing A Recorded Run + +Extract the paired sessions into a metrics file (JSON array or JSONL, one session-metric row each) with `scripts/extract-benchmark-metrics.ts` (or the SQL above), then summarize it into the outcome taxonomy: + +```bash +bunx @whchi/your-legion benchmark-summarize --metrics +``` + +This reuses `summarizeOrchestrationBenchmark`, so the task-level `outcome` labels are computed the same way in the CLI, the library, and tests. A committed fixture of the 2026-05-23 control run lives at `tests/fixtures/orchestration-benchmark/2026-05-23-deepseek-v4-pro.jsonl`; summarizing it reproduces `more-expensive-not-better` on all four read-only tasks (native 359,315 vs your-legion 777,249 tokens). See [ADR 0004](./adr/0004-orchestration-value-evidence.md). + +Trace-based acceptance: trace files are now keyed by a normalized worktree, so `trace-check --worktree .` no longer reads an empty trace and reports a false pass when events were written under a degenerate `/` worktree. And `trace-check --require-evidence` fails when the expected delegation or loop evidence is absent, so an empty trace is no longer a silent pass for acceptance (ADR 0004, D4). + ## Report Shape The final comparison table should use this shape: diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index f1bac44..e7819f3 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -109,6 +109,14 @@ Acceptance criteria: | Benchmark | Keep clean native vs orchestrated task set | P0 | Done: benchmark doc keeps the fixed four-domain prompt set and clean-run rules | | Benchmark | Add summary table for quality plus token cost | P1 | Done: benchmark summaries include task `outcome` and `byOutcome` aggregation | | Loop | Add Legion Loop contract and verifier | P0 | Done: loops are config-backed, prompt-injected, trace-aware, and doctor-validated | +| Verification | Enforce maker/checker integrity: same-model guard, default check for code work, custom-agent guard | P1 | Done: [ADR 0003](./adr/0003-enforced-verification-and-maker-checker-integrity.md) implemented | +| Benchmark | Orchestration-value evidence: harder task set, 3-way comparison, benchmark-summarize CLI, trace worktree fix | P1 | Done: [ADR 0004](./adr/0004-orchestration-value-evidence.md) implemented | +| Verification | Blocking non-loop ledger check: capture `Verification status` blocks as `verification-report` trace events; `trace-check` fails unverified file changes without a following verifier pass | P1 | Done: extends ADR 0003 D2/D4 to the non-loop path | +| Verification | Custom-agent neutrality warning at agent-definition load time | P1 | Done: the ADR 0003 D3 guard now also fires when the definition is loaded, not only in `doctor` | +| DX | CI workflow, grouped CLI help, evidence-first README | P1 | Done: `.github/workflows/ci.yml` runs test/build; README leads with the maker/checker ledger | +| Docs | Replace collected source essays with a design-influences index | P2 | Done: `docs/design-influences.md` summarizes the loop-engineering and LLM-wiki sources with claim boundaries | +| Benchmark | Run and commit the harder-suite / 3-way metrics export | P0 | Done 2026-07-04: 6 live `opencode run` executions (native / same-provider / mixed-provider × 2 exec tasks); fixture `2026-07-04-exec-quality-3way.jsonl`; results in `ORCHESTRATOR_BENCHMARK.md` | +| Benchmark | Design deceptively-green tasks where maker self-check passes but is wrong | P1 | Open: 2026-07-04 run showed native also catches the current planted defects and builder self-verifies, so no verifier delegation was ever triggered | ## Implementation Status diff --git a/docs/adr/0003-enforced-verification-and-maker-checker-integrity.md b/docs/adr/0003-enforced-verification-and-maker-checker-integrity.md new file mode 100644 index 0000000..a831200 --- /dev/null +++ b/docs/adr/0003-enforced-verification-and-maker-checker-integrity.md @@ -0,0 +1,35 @@ +# ADR 0003: Enforced Verification and Maker/Checker Integrity + +## Status + +Accepted. Implemented 2026-07-01: same-model guard in `doctor`, default post-execution verification (opt-out via `verification.default_check`), custom-agent neutrality check in `doctor`, and the warn-vs-block policy documented in `DOMAIN_OBSERVABILITY.md`. + +## Context + +Your Legion already separates the maker (`builder`) from the checker (`verifier`) and enforces tool boundaries so the checker cannot edit. The current guarantees have four gaps: + +- **Verification is warn-only and after the fact.** Trace evidence is advisory, and `doctor` validates a run only after it has happened. Nothing at runtime forces a delegation to actually read the declared domain skills or to route maker output through an independent checker. +- **Independent checking is mandatory only inside Legion Loops.** For ordinary one-shot execution, the `orchestrator` routes to `verifier` only when the user explicitly asks for verification. A normal "fix this bug" task can be completed and self-reported by `builder` with no second set of eyes. +- **`doctor` checks maker/checker separation by agent name only.** `doctor.ts:468-477` fails when a loop's `maker` and `verifier` are the same agent name, but it never compares their configured **model or provider**. Two separate agents backed by the **same underlying model** still share correlated blind spots. The per-agent model map in `legionaries.yaml` makes cross-vendor checking possible, but nothing guards against accidentally using one model for both roles. +- **There is no hidden runtime enforcement point for default post-builder checks.** OpenCode owns the chat, tool execution, and agent runtime. Your Legion can inject prompts, config, hooks, and trace/doctor diagnostics, but it must not assume an automatic worktree-diff detector that can start a `verifier` delegation after `builder` finishes. Any non-loop default-check policy has to be expressed through builder/orchestrator reporting, trace evidence, and doctor/trace-check gates unless OpenCode exposes a stronger runtime hook. +- **The work-mode / capability separation is guarded for bundled agents only.** The domain-neutrality guard added for the agent purification work is a denylist test over bundled agents in `tests/agent-config.test.ts`. User-authored custom agents are not checked, so a custom agent can silently re-absorb domain capability and re-merge the two axes described in the README Routing Model. + +The safety net exists, but it is opt-in, name-level, bundled-only, and partly dependent on self-reporting. This ADR records the direction for making the guarantee model-aware and diagnosable without pretending Your Legion controls parts of the OpenCode runtime it does not own. + +## Decision + +Adopt the following as the intended direction. Each item is a decision record; implementation is tracked separately (see `ROADMAP.md`). + +- **D1 - Same-model guard.** `doctor` warns when a loop's resolved `maker` and `verifier` map to the same provider/model in `legionaries.yaml`, in addition to the existing same-name failure at `doctor.ts:468-477`. Separate windows do not remove shared-model blind spots; the checker should differ in model, not only in name. This stays a warning, not a hard failure, because operators may deliberately accept same-model checking. +- **D2 - Default check for code-touching execution outside loops.** Make an independent check the default, not an explicit request, when `builder` reports that it changed files. The first implementation must use explicit ledger fields rather than an assumed runtime auto-trigger: `builder` reports whether file changes were made, whether a `verifier` pass ran, and whether the result is `verified`, `unverified`, or `verification-skipped` with a reason. The `orchestrator` acts on that report: it offers or routes a `verifier` pass before final completion when the user has not opted out and the runtime supports another delegation. `doctor` and `trace-check` then validate the resulting ledger. This is a user-visible behavior change and must remain configurable so operators can keep the current opt-in behavior. +- **D3 - Custom-agent two-axis guard.** Extend the domain-neutrality guard beyond bundled agents so custom agents are also checked, via a `doctor` lint over discovered custom-agent definitions or a warning at agent-definition load time. User-authored agents should not re-merge work-mode and capability. This keeps the invariant recorded in the README Routing Model true for the whole agent set, not just the bundled six. +- **D4 - Explicit warn-vs-block policy.** Document, in one place, which trace/doctor signals stay warn-only and which are blocking. This refines [ADR 0001](./0001-plugin-first-domain-aware-orchestration.md): ordinary domain-evidence completeness remains warn-only, but maker/checker ledger integrity becomes blocking wherever a loop or code-touching verification ledger exists. Existing examples already point in that direction: `maker-complete` without `verifier-complete` fails at `doctor.ts:584-585`, while `verifier-complete` without a passed outcome warns at `:587-588`. + +## Consequences + +- The maker/checker guarantee moves from "separate name" to "separate model plus an explicit verification ledger for code-touching runs." +- New friction and new configuration surface: a default `verifier` pass costs tokens and time, and D1/D2 need opt-out knobs in `legionaries.yaml`. +- D2 changes default behavior and must be called out in release notes and `CONFIGURATION.md`. +- This ADR does not change the loop contract in [ADR 0002](./0002-legion-loop-contract.md); it generalizes that contract's maker/checker discipline toward the non-loop path. +- This ADR does change the warn-only sentence in [ADR 0001](./0001-plugin-first-domain-aware-orchestration.md) for maker/checker ledger evidence. Domain routing evidence can remain advisory; completion-integrity evidence cannot. +- Value measurement for the added cost is the subject of [ADR 0004](./0004-orchestration-value-evidence.md). diff --git a/docs/adr/0004-orchestration-value-evidence.md b/docs/adr/0004-orchestration-value-evidence.md new file mode 100644 index 0000000..8f9e43a --- /dev/null +++ b/docs/adr/0004-orchestration-value-evidence.md @@ -0,0 +1,31 @@ +# ADR 0004: Orchestration-Value Evidence + +## Status + +Accepted. Implemented 2026-07-01: `providerProfile` metrics dimension for native/same-provider/mixed-provider, `benchmark-summarize` CLI backed by a committed fixture, `scripts/extract-benchmark-metrics.ts` plus runnable harder-suite prompts in `ORCHESTRATOR_BENCHMARK.md`, the trace worktree keying fix, and `trace-check --require-evidence` so an empty trace is not a pass for acceptance. + +## Context + +[ADR 0001](./0001-plugin-first-domain-aware-orchestration.md) commits to evaluating orchestration as a quality-plus-cost tradeoff, not a token-saving tactic, and `docs/ORCHESTRATOR_BENCHMARK.md` defines the protocol and a reusable summarizer (`summarizeOrchestrationBenchmark`). The machinery is good; the evidence is not yet conclusive. + +- **The only measured run is unfavorable, on the least favorable tasks.** Re-summarizing the recorded 2026-05-23 control run (marker ending in `202605231350pro`) over the live `opencode.db` yields `more-expensive-not-better` on all four tasks: native `359,315` tokens vs your-legion-orchestrated `777,249` (+116.31%), with no pass-rate gain. Routing was correct (`coding-001` to `explorer`, others to `builder`). The four tasks are read-only and single-turn. That is the case where a router-plus-specialist layer is least able to earn back its overhead, because there is no rework to avoid and no multi-step coordination or error interception to provide. The value hypothesis, that orchestration and independent verification pay off on harder work, is neither confirmed nor refuted. +- **The 3-way comparison was never completed.** The benchmark defines native / same-provider orchestrated / mixed-provider orchestrated. The recorded run compares native against a single orchestrated variant; the "is mixed-provider worth it" question the per-agent model map exists to answer is unmeasured. +- **The summarizer has nothing driving it.** `summarizeOrchestrationBenchmark` is exported from `src/index.ts` but there is no CLI command (`cli.ts` has no `benchmark` path). Producing the outcome taxonomy currently requires ad-hoc scripting, so the published result table omits the `outcome` column its own "Report Shape" section prescribes. +- **`trace-check` cannot yet be trusted as acceptance evidence.** Trace files are keyed by `sha256(resolve(worktree))` (`domain-usage-contract.ts:429`) and events store `options.worktree` (`:629`), fed from `input.worktree` in `src/index.ts`. In the recorded run those events carried `worktree: "/"`, so `trace-check --worktree .` hashed a different path, read an empty trace, and reported pass despite real malformed-`Active domains` warnings. +- **The measured-token claim is not reproducible from repo state.** The token totals above come from a live `opencode.db`, but no exported metrics file or fixture is versioned in this repo. The ADR can cite the current result as context, but benchmark acceptance needs a committed metrics export that `benchmark-summarize` can read. + +## Decision + +Adopt the following direction. Each item is a decision record; implementation is tracked separately (see `ROADMAP.md`). + +- **D1 - Split benchmark suites by question.** Keep the existing four read-only tasks as the routing-cost baseline. Add a separate harder task set for execution quality and verifier value: multi-step tasks, error-prone tasks, and optionally file-mutating tasks with planted defects. File-mutating tasks must not be mixed into the routing-cost baseline, because edits, test runs, repair turns, and verification noise measure a different product question. Record the task-design criteria for both suites so the sets are reproducible. +- **D2 - Make the 3-way comparison first-class in the metrics model.** Run and record native / same-provider orchestrated / mixed-provider orchestrated for each relevant task. The implementation must either extend `BenchmarkVariant` beyond the current `native-builder | your-legion-orchestrated` pair, or add an explicit provider-profile/model-map dimension that lets the summarizer separate same-provider from mixed-provider runs. The mixed-provider question should be answered from the data shape, not from prose labels in prompts. +- **D3 - Add a `benchmark-summarize` CLI command backed by committed metrics exports.** Add a command to `cli.ts` (flat `if (command === '...')` dispatch, `cli.ts:146+`) that reads an exported metrics file, JSON or JSONL produced by the documented SQL extraction, and prints the outcome taxonomy via `summarizeOrchestrationBenchmark`. It must read a file, not open the session DB with `bun:sqlite`, because the CLI build targets node (`bun build --target node`). At least one minimal exported fixture for the current benchmark shape must be committed under `docs/` or `tests/fixtures/` so the token/result claims can be rechecked without the local `opencode.db`. +- **D4 - Fix trace worktree keying and empty-trace success.** Ensure the worktree used when writing trace events equals the workspace path used at check time. Normalize and resolve consistently, and derive the worktree robustly rather than trusting a possibly-`"/"` `input.worktree`. `trace-check` must also fail when the expected benchmark/task/delegation evidence is absent; an empty trace is not a pass for benchmark acceptance. Until both keying and non-empty evidence checks are fixed, `trace-check` is excluded as benchmark acceptance evidence. + +## Consequences + +- The value question becomes answerable in both directions: the read-only suite can continue to show routing overhead, while the harder suite can show whether orchestration produces fewer failures, fewer repair turns, or better checked output. +- The benchmark stops being a one-off manual document: `benchmark-summarize` turns versioned metrics exports into repeatable, labeled reports. +- `trace-check` becomes usable as an acceptance gate once D4 lands, which in turn lets the maker/checker enforcement in [ADR 0003](./0003-enforced-verification-and-maker-checker-integrity.md) rely on trace evidence without accepting "no events found" as success. +- No conclusion is pre-committed. Consistent with [ADR 0001](./0001-plugin-first-domain-aware-orchestration.md), the benchmark is allowed to show that orchestration costs more without improving quality; that result would itself be a valid, publishable outcome. diff --git a/docs/design-influences.md b/docs/design-influences.md new file mode 100644 index 0000000..aa94a41 --- /dev/null +++ b/docs/design-influences.md @@ -0,0 +1,28 @@ +# Design Influences: Loop Engineering and Compiled Knowledge + +This is the non-academic counterpart to [`academic-papers-summary.md`](./academic-papers-summary.md). Two collected essays shaped the Legion Loop and Domain Pack design; this file keeps the ideas and their claim boundaries instead of the full source texts. + +## Loop Engineering + +Source idea (collected essay, 2026-06): loop engineering is replacing yourself as the person who prompts the agent. You design the system that prompts it instead: a recursive goal with a purpose, durable state that records what is done, and a mechanism that decides the next step and pokes the agent on a timer or trigger. It sits one floor above harness engineering. The essay is explicitly skeptical: the pattern is early, and token cost discipline is mandatory. + +| Concept used in this repo | Implemented / documented in | +|---|---| +| Loops as explicit contracts with durable inboxes and run state | `src/config/legionaries.ts` (loop config), `docs/LEGION_LOOPS.md`, ADR 0002 | +| Maker/checker separation inside every loop run | `src/agents/builder.ts`, `src/agents/verifier.ts`, `src/runtime/doctor.ts` loop sections | +| Run ledgers and completion evidence over self-reporting | `loop-runs` CLI, `loop-run-report` trace events in `src/runtime/domain-usage-contract.ts` | +| Token cost skepticism | `docs/ORCHESTRATOR_BENCHMARK.md`, ADR 0004 | + +Claim boundary: Your Legion loops are explicit-only (a loop id must be named; the orchestrator never infers one) and are not self-feeding autonomous loops. ADR 0001 records the non-goal: no autonomous continuation features to compensate for OpenCode runtime behavior. + +## LLM-Maintained Wiki (Compiled Knowledge) + +Source idea (collected essay, 2026-06): instead of retrieving from raw documents at query time, an LLM incrementally builds and maintains a persistent, interlinked markdown wiki that sits between the user and the raw sources. Knowledge is compiled once, revised when new sources contradict old claims, and read many times. + +| Concept used in this repo | Implemented / documented in | +|---|---| +| Versioned, compiled expert context instead of ad hoc retrieval | Domain Packs: `DOMAIN.md` as routing/description contract, `docs/DOMAIN_PACK_AUTHORING.md` | +| Incremental knowledge updates with revision discipline | `domain-update` CLI, per-domain `updates.md` inbox, `--promote-to` flow | +| Selective disclosure: read only the declared components | Domain Catalog injection and `domain-read` trace evidence | + +Claim boundary: Your Legion does not implement an auto-maintained wiki. Domain updates land in a human-reviewed `updates.md` inbox and are promoted deliberately; no agent rewrites domain knowledge on its own. diff --git a/docs/loog-engineering.jpg b/docs/loog-engineering.jpg deleted file mode 100644 index ce07c88..0000000 Binary files a/docs/loog-engineering.jpg and /dev/null differ diff --git a/docs/loog-engineering.txt b/docs/loog-engineering.txt deleted file mode 100644 index 41f58ae..0000000 --- a/docs/loog-engineering.txt +++ /dev/null @@ -1,52 +0,0 @@ -Loop engineering is replacing yourself as the person who prompts the agent. You design the system that does it instead. A loop here can be thought of a recursive goal where you define a purpose and the AI iterates until complete. It's roughly five building blocks and Claude Code and Codex both have all five now. -I believe this may be the future of how we work with coding agents. However, its still early, I'm skeptical and you absolutely have to be careful about token costs (usage patterns can vary wildly if you are token rich or poor). You also still need some way to ensure quality doesn't drop and concerns re: slop are valid. That said, let's explore what this is all about. -@steipete recently said: “You shouldn't be prompting coding agents anymore. You should be designing loops that prompt your agents.” Similarly, @bcherny, head of Claude Code at Anthropic, said “I don't prompt Claude anymore. I have loops running that prompt Claude and figuring out what to do. My job is to write loops”. -Okay, so what does any of that mean? -For like two years the way you got something out of a coding agent was you wrote a good prompt and shared enough context. You type a thing, you read what came back, you type the next thing. The agent is a tool and you are holding it the entire time, one turn after the other. That part is kind of over, or at least some think it's going to be. -Now you build a small system that finds the work, hands it out, checks it, writes down what is done and then decides the next thing, and you let that system poke the agents instead of you. I wrote before about the cousin of this, agent harness engineering, which is making the environment one single agent runs inside and the factory model - the system that builds the software. Loop engineering sits one floor above the harness. The harness but it runs on a timer, it spawns little helpers, and it feeds itself. -The thing that surprised me is this is not really a tool thing anymore. A year ago if you wanted a loop you wrote a pile of bash and you maintained that pile forever and it was yours and only yours. Now the pieces just ship inside the products. Steinberger's list maps almost exactly onto the Codex app, and then almost the same onto Claude Code. And once you notice the shape is the same you stop arguing about which tool, you just design a loop that still works no matter which one you happen to be sitting in. -The five pieces, and then notes -A loop needs five things and then one place to remember stuff. Let me list it first and then map it. -Automations that go off on a schedule and do discovery and triage by themselves. -Worktrees so two agents working in parallel dont step on each other. -Skills to write down the project knowledge the agent would otherwise just guess. -Plugins and connectors to plug the agent into the tools you already use. -Sub-agents so one of them has the idea and a different one checks it. -Then the sixth thing, the memory. A markdown file, or a Linear board, anything that lives outside the single conversation and holds what's done and what is next. Sounds too dumb to matter. But it's the same trick every long running agent depends on and I went into it in long-running agents, the model forgets everything between runs so the memory has to be on disk and not in the context. The agent forgets, the repo doesnt. -Both products have all five now. -The names are a bit different here and there but the capability is the same thing. Let me go one by one because honestly the details are where a loop either holds together or quietly leaks everywhere. -Automations, this is the heartbeat -Automations are what make a loop an actual loop and not just one run you did once. In the Codex app you make one in the Automations tab and you pick the project, the prompt it will run, how often, and if it runs on your local checkout or on a background worktree. The runs that find something go to a Triage inbox, and the runs that find nothing just archive themselves which is nice. OpenAI uses them internally for boring stuff like daily issue triage, summarizing CI failures, writing commit briefings, hunting bugs somebody added last week. And an automation can call a skill, so you keep the recurring thing maintainable, you fire $skill-name instead of pasting a giant wall of instructions into a schedule that nobody will ever update. -Claude Code gets to the same place but through scheduling and hooks. You can run a prompt or a command on a interval with /loop, you can schedule a cron task, you can fire shell commands at certain points in the agent lifecycle with hooks, or you push the whole thing to GitHub Actions if you want it to keep running after you close the laptop. Same idea exactly, you define an autonomous task, you give it a cadence, and the findings come to you so you are not the one going around checking. -There is a second in-session primitive worth knowing, and it's the one closer to what this whole post is about. /loop re-runs on a cadence. /goal keeps going until a condition you wrote is actually true, and after every turn a separate small model checks whether you are done, so the agent that wrote the code isnt the one grading it. You give it something like "all tests in test/auth pass and lint is clean" and walk away. Codex has the same thing, also called /goal, it keeps working across turns until a verifiable stopping condition holds, with pause and resume and clear. Same primitive, both tools, wich is kind of the pattern for this whole article. -So this is the part that surfaces the work. The rest of the loop is what acts on it. -Worktrees so parallel doesn't turn into chaos -The second you run more than one agent the files start colliding, that becomes the failure. Two agents writing the same file is the exact same headache as two engineers committing to the same lines and nobody talked to each other first. A git worktree fixes it, its a separate working directory on its own branch sharing the same repo history, so one agent's edits literally can not touch the other one's checkout. -Codex builds the worktree support right in so several threads hit the same repo at once and dont bump into each other. Claude Code gives you the same isolation with git worktree, a --worktree flag to open a session in its own checkout, and a isolation: worktree setting you stick on a subagent so each helper gets a fresh checkout that cleans itself up after. I wrote about the human side of all this in the orchestration tax, the worktrees take away the mechanical collision but YOU are still the ceiling, your review bandwith decides how many you can actually run, not the tool. -Skills, so you stop explaining your project every single time -A skill is how you stop re-explaining the same project context every session like a goldfish. Both tools use the same format, a folder with a SKILL.md inside holding instructions and metadata, and then optional scripts, references, assets. Codex runs a skill when you call it with $ or /skills, or by itself when your task matches the skill description, which is the reason a tight boring description beats a clever one. Claude Code does it the same way and I wrote the pattern up in agent skills. -Skills are also where intent stops costing you over and over. I argued in the intent debt that an agent starts every session cold and it will fill any hole in your intent with a confident guess. A skill is that intent written down on the outside, the conventions, the build steps, the “we dont do it like this because of that one incident”, written one time where the agent reads it every run. Without skills the loop re-derives your whole project from zero every cycle, with skills it kind of compounds. -One thing to keep straight, the skill is the authoring format and a plugin is how you ship it. When you want to share a skill across repos or bundle a few together you package them as a plugin. True in Codex, true in Claude Code. -Plugins and connectors, the loop touches your real tools -A loop that can only see the filesystem is a tiny loop. Connectors, which are built on MCP, let the agent read your issue tracker, query a database, hit a staging api, drop a message in Slack. Codex and Claude Code both speak MCP so the connector you wrote for one usually just works in the other. And plugins bundle connectors and skills together so your teammate installs your setup in one go instead of rebuilding the whole thing from memory. -This is the difference between an agent that says “here is the fix” and a loop that opens the PR, links the Linear ticket and pings the channel once CI is green by itself. The connectors are the reason the loop can act inside your actual environment instead of just telling you what it would do if it could. -Sub-agents, keep the maker away from the checker -The most useful structural thing in a loop, by far, is splitting the one who writes from the one who checks. The model that wrote the code is way too nice grading its own homework. A second agent with different instructions and sometimes a different model catches the stuff the first one talked itself into. -Codex only spawns subagents when you ask, runs them at the same time and then folds the results back into one answer. You define your own agents as TOML files in .codex/agents/, each with a name, a description, instructions and optional model and reasoning effort, so your security reviewer can be a strong model on high effort while your explorer is some fast read-only thing. Claude Code does the same with subagents in .claude/agents/ and agent teams that pass work between them. The usual split in both is one agent explores, one implements, one verifies against the spec. -I made this case twice already, once as the code agent orchestra and once as adversarial code review. The reason it matters specifically inside a loop is the loop runs while you are not watching, so a verifier you actually trust is the only reason you can walk away. Subagents do burn more tokens since each one does its own model and tool work, so spend them where a second opinion is worth paying for. This is also basically what Claude Code's /goal does under the hood, a fresh model decides if the loop is done instead of the one that did the work, the maker and checker split applied to the stop condition itself. -What one loop looks like -Stick it together and a single thread turns into a little control panel. Here is one shape I keep using. -An automation runs every morning on the repo. Its prompt calls a triage skill that reads yesterdays CI failures, the open issues, the recent commits, and writes the findings into a markdown file or a Linear board. For each finding that is worth doing the thread opens an isolated worktree and sends a sub-agent to draft the fix, and a second sub-agent reviews that draft against the project skills and the existing tests. -Connectors let the loop open the PR and update the ticket. Anything the loop can not handle lands in the triage inbox for me. The state file is the spine of the whole thing, it remembers what got tried, what passed, what is still open, so tomorrow morning the run picks up where today stopped. -And look at what you actually did there. You designed it one time. You did not prompt any of those steps. Thats Steinberger's whole point made real, and its the same loop in Codex or in Claude Code because the pieces are the same pieces. -What the loop still does not do for you -The loop changes the work, it does not delete you from it. And three problems actually get sharper as the loop gets better, not easier. -Verification is still on you. A loop running unattended is also a loop making mistakes unattended. The whole reason you split the verifier sub-agent from the maker is to make the loop's “its done” mean something, and even then “done” is a claim and not a proof. I keep saying the same line from code review in the age of AI, your job is to ship code you confirmed works. -Your understanding still rots if you allow it. The faster the loop ships code you did not write, the bigger the gap between what exists and what you actually get. Thats comprehension debt and a smooth loop just makes it grow faster unless you read what the loop made. -And yeah, the comfortable posture is probably the risky one. When the loop runs itself its very tempting to stop having an opinion and just take whatever it gives back. I called that cognitive surrender. Designing the loop is the cure when you do it with judgement and the accelerant when you do it to avoid thinking, same action, opposite result. -Build the loop. Stay the engineer. -I think this is a preview of how our work is going to evolve. That said, If I weren't reviewing the code myself or if I relied entirely on automated loops to fix it my product’s quality would suffer. I'd likely end up stuck in a downward spiral, continuously digging myself into a deeper hole. -That said, go ahead and set up your loops, but don't forget that prompting your agents directly is still effective. It's all about finding the right balance. -Loops can also result in different outcomes depending on you. Two people can build the exact same loop and get completely opposite results. One uses it to move faster on work they understand deeply. The other uses it to avoid understanding the work at all. The loop doesn't know the difference. You do. -That's what makes loop design harder than prompt engineering, not easier. Cherny's point isn’t that the work got easier. It's that the leverage point moved. -Build the loop. But build it like someone who intends to stay the engineer, not just the person who presses go. diff --git a/legionaries.yaml b/legionaries.yaml index d7d199c..4bfaf11 100644 --- a/legionaries.yaml +++ b/legionaries.yaml @@ -32,3 +32,6 @@ custom_agents: effort: high domains: coding: true +# Optional post-execution verification. Defaults to on when omitted. +# verification: +# default_check: true diff --git a/package-lock.json b/package-lock.json deleted file mode 100644 index f095bd5..0000000 --- a/package-lock.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "name": "omo-clone", - "lockfileVersion": 3, - "requires": true, - "packages": { - "": { - "name": "omo-clone", - "dependencies": { - "yaml": "^2.8.4" - }, - "version": "0.1.6" - }, - "node_modules/yaml": { - "version": "2.8.4", - "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.8.4.tgz", - "integrity": "sha512-ml/JPOj9fOQK8RNnWojA67GbZ0ApXAUlN2UQclwv2eVgTgn7O9gg9o7paZWKMp4g0H3nTLtS9LVzhkpOFIKzog==", - "license": "ISC", - "bin": { - "yaml": "bin.mjs" - }, - "engines": { - "node": ">= 14.6" - }, - "funding": { - "url": "https://github.com/sponsors/eemeli" - } - } - }, - "version": "0.1.6" -} diff --git a/package.json b/package.json index 6b2fa16..48cda9e 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "url": "git+https://github.com/whchi/your-legion.git" }, "license": "ISC", - "author": "", + "author": "whchi ", "type": "module", "exports": { ".": "./dist/server.js", @@ -21,10 +21,6 @@ "bin": { "your-legion": "./dist/cli.js" }, - "directories": { - "doc": "docs", - "test": "tests" - }, "files": [ "dist", "README.md" diff --git a/scripts/extract-benchmark-metrics.ts b/scripts/extract-benchmark-metrics.ts new file mode 100644 index 0000000..830a95b --- /dev/null +++ b/scripts/extract-benchmark-metrics.ts @@ -0,0 +1,123 @@ +#!/usr/bin/env bun +/** + * Extract paired benchmark sessions from a local OpenCode DB into the metrics JSONL + * that `your-legion benchmark-summarize --metrics ` consumes. + * + * Bun-only dev tool (uses bun:sqlite); it is not part of the node CLI build. + * + * Usage: + * bun scripts/extract-benchmark-metrics.ts --marker \ + * [--db ] [--profile same-provider|mixed-provider] [--tasks coding-001,finance-001] > metrics.jsonl + * + * Note: pass/fail is a human rubric and is NOT stored in the DB. Every row is emitted + * passed=true; edit the output to match the recorded rubric before summarizing. + * See docs/adr/0004-orchestration-value-evidence.md (D3). + */ +import { Database } from 'bun:sqlite'; + +type TokenRow = { + id?: string; + agent?: string; + title?: string; + tokens_input: number; + tokens_output: number; + tokens_reasoning: number; + tokens_cache_read: number; + tokens_cache_write: number; +}; + +function option(name: string): string | undefined { + const index = process.argv.indexOf(name); + return index === -1 ? undefined : process.argv[index + 1]; +} + +const dbPath = option('--db') ?? `${process.env.HOME}/.local/share/opencode/opencode.db`; +const marker = option('--marker'); +const profile = option('--profile'); +const taskFilter = option('--tasks') + ?.split(',') + .map(value => value.trim()) + .filter(Boolean); + +if (!marker) { + console.error('extract-benchmark-metrics requires --marker '); + process.exit(1); +} +if (profile && profile !== 'same-provider' && profile !== 'mixed-provider') { + console.error('--profile must be same-provider or mixed-provider'); + process.exit(1); +} + +const TOKEN_COLUMNS = 'tokens_input, tokens_output, tokens_reasoning, tokens_cache_read, tokens_cache_write'; +const db = new Database(dbPath, { readonly: true }); +const like = (value: string) => `%${value}%`; + +function tokensOf(row: TokenRow) { + return { + tokensInput: row.tokens_input ?? 0, + tokensOutput: row.tokens_output ?? 0, + tokensReasoning: row.tokens_reasoning ?? 0, + tokensCacheRead: row.tokens_cache_read ?? 0, + tokensCacheWrite: row.tokens_cache_write ?? 0, + }; +} + +function taskIDFromTitle(title: string): string | undefined { + // Multi-segment ids like exec-coding-001 must not truncate to coding-001, + // which would collide with the routing-baseline task ids. + return title.match(/\b([a-z]+(?:-[a-z]+)*-\d{3})\b/)?.[1]; +} + +function emit(metric: Record) { + process.stdout.write(`${JSON.stringify(metric)}\n`); +} + +const roots = db + .query(`select agent, title from session where title like ? and (agent='build' or agent='orchestrator')`) + .all(like(marker)) as TokenRow[]; + +const tasks = new Map(); +for (const root of roots) { + const taskID = root.title ? taskIDFromTitle(root.title) : undefined; + if (taskID) { + tasks.set(taskID, taskID.replace(/-\d{3}$/, '')); + } +} + +for (const [taskID, taskType] of [...tasks].sort(([left], [right]) => left.localeCompare(right))) { + if (taskFilter && !taskFilter.includes(taskID)) { + continue; + } + + const base = { benchmarkID: marker, taskID, taskType, messages: 1, cost: 0, reworkTurns: 0, traceWarnings: 0, passed: true }; + const profileFields = profile ? { providerProfile: profile } : {}; + + const native = db + .query(`select ${TOKEN_COLUMNS} from session where title like ? and title like ? and agent='build'`) + .get(like(marker), like(taskID)) as TokenRow | null; + if (native) { + emit({ ...base, variant: 'native-builder', agent: 'builder', ...tokensOf(native) }); + } + + const root = db + .query(`select id, ${TOKEN_COLUMNS} from session where title like ? and title like ? and agent='orchestrator'`) + .get(like(marker), like(taskID)) as TokenRow | null; + if (root) { + emit({ ...base, variant: 'your-legion-orchestrated', agent: 'orchestrator', ...profileFields, ...tokensOf(root) }); + + const children = db + .query(`select agent, ${TOKEN_COLUMNS} from session where parent_id=?`) + .all(root.id) as TokenRow[]; + for (const child of children) { + emit({ + ...base, + variant: 'your-legion-orchestrated', + agent: child.agent ?? 'specialist', + ...profileFields, + ...tokensOf(child), + }); + } + } +} + +console.error('note: pass/fail is a human rubric, not stored in the DB. Rows are emitted passed=true; edit to match the recorded rubric before summarizing.'); diff --git a/src/agents/builder.ts b/src/agents/builder.ts index 4fd7a1c..0a7af20 100644 --- a/src/agents/builder.ts +++ b/src/agents/builder.ts @@ -4,55 +4,41 @@ const MODE = 'subagent' as const; const PROMPT = `# Builder -You are the execution specialist for approved tasks and tightly scoped deliverables, including code, tests, frontend/UI work, analysis, copy, and code-coupled documentation. +You are the execution specialist for approved, tightly scoped deliverables, including code, tests, analysis, copy, and code-coupled documentation. -Play the role of a deep worker: make the change, keep it small, verify it, and report the real result. - -## Focus Areas - -- backend and application logic -- frontend, UI, styling, interaction, responsiveness, and accessibility -- non-code deliverables such as concise analysis, copy, summaries, and structured reviews -- configuration and wiring -- tests and verification -- refactors and bug fixes -- documentation directly coupled to the change - -## Execution Workflow - -- For behavior changes, write or update a focused failing test first. Run it, confirm it fails for the expected reason, then implement the smallest change that makes it pass. -- Choose the narrowest test level that gives confidence: unit for pure behavior, integration for component boundaries, and end-to-end only when the user or system flow matters. -- Decide dependencies deliberately: keep real code when practical, fake internal boundaries when persistence or IO is not under test, and mock third-party, slow, expensive, or nondeterministic services. -- When the root cause is not known, write the symptom and expected behavior, then split hypotheses into environment, data, and logic before editing. Gather evidence, minimize the reproduction, and fix only the confirmed cause. -- For build, type, or compile failures, identify the project build command, group errors by file, fix one build or type error at a time, and rerun verification after each fix. -- For frontend work, preserve existing visual language, check responsiveness and accessibility, cover loading/empty/error states when behavior changes, and avoid broad backend changes unless explicitly required. +This prompt defines a work mode, not a profession: you make the change, keep it small, verify it, and report the real result. You are domain-neutral. The professional capability for the task comes from the Domain skills named in the Task Context Envelope, not from this prompt. ## Working Style -- Read the Task Context Envelope first. Follow its Active domains and Context refs before using broader Domain Pack context. -- If the envelope names a Loop, read the loop contract and referenced inbox before changing code, then report what you used as Loop evidence. +- Read the Task Context Envelope first. Follow its Active domains, Domain skills, and Context refs before using broader Domain Pack context. +- Treat the named Domain skills as your capability boundary: read them and apply their procedure for this domain. If the deliverable needs domain expertise that no active domain provides, say so instead of improvising it. - Provider specialization: Trust your specialist responsibility and configured tool boundary; do not split the task into an imagined team. +- If the envelope names a Loop, read the loop contract and referenced inbox before acting, then report what you used as Loop evidence. - If you read Domain refs or Domain skills, report them under Domain evidence; list the exact catalog ids or paths you actually read. -- Read the plan or task carefully before changing code. -- Follow existing project patterns. +- Follow existing project conventions. - Prefer the smallest correct change. -- Add or update tests when behavior changes. -- Run verification before claiming success. +- Verify before claiming success; if you cannot verify, say so rather than asserting it. - For loop work, include a Loop Run Report with the same Loop run id from the envelope. Use \`maker-complete\` only when the completion claim is verified, \`blocked\` when human input is needed, and \`failed\` when the attempt did not satisfy the objective. +- At the end of loop or domain work, report Domain update candidates. Include only reusable knowledge that changes future domain judgment, workflow, verification, correction, or decisions. Use "none" when nothing should enter a domain update inbox. - If the envelope lacks correctness-critical context, ask instead of guessing. ## Boundaries -- If the plan is unclear or unsafe, stop and ask instead of guessing. -- Do not invent extra architecture that was not requested. +- If the task is unclear or unsafe, stop and ask instead of guessing. +- Do not invent scope, abstractions, or architecture that was not requested. - You are a leaf specialist. Do not delegate to other subagents yourself. ## Output Expectations Return: -- files changed -- what was implemented +- files or artifacts changed +- what was delivered +- Verification status: + - Files changed: yes | no + - Verifier pass: ran | not-run + - Status: verified | unverified | verification-skipped + - Reason (required when Status is unverified or verification-skipped) - Loop Run Report: - Loop: - Loop run: @@ -62,6 +48,7 @@ Return: - Verification outcome: passed | failed | not-run | unknown - Loop evidence: loop id and inbox/context refs actually read, or none - Domain evidence: domain refs and domain skills actually read, or none +- Domain update candidates: candidate entries with domain, type, title, promote target, and evidence, or none - verification commands run and outcomes - any remaining follow-up or risk`; diff --git a/src/agents/librarian.ts b/src/agents/librarian.ts index 655981c..3217476 100644 --- a/src/agents/librarian.ts +++ b/src/agents/librarian.ts @@ -4,15 +4,15 @@ const MODE = 'subagent' as const; const PROMPT = `# Librarian -You are the read-only documentation and API reference specialist for this workspace. +You are the read-only documentation and reference-lookup specialist for this workspace. -Use this role when the requested deliverable is third-party or external documentation, especially information unknown outside the current repo. +This prompt defines the external-lookup work mode, not a profession: find authoritative third-party or external documentation, especially information unknown outside the current repo, and report it. The kind of source depends on the active domain; for software libraries and frameworks, Context7 is the primary tool. ## Focus Areas -- official documentation lookup -- Context7 MCP documentation lookup for libraries and frameworks -- library and framework API references +- official documentation and primary-source lookup +- Context7 MCP lookup for library and framework docs +- external API references, standards, and specifications - external implementation patterns - integration notes and compatibility concerns diff --git a/src/agents/orchestrator.ts b/src/agents/orchestrator.ts index 1ba57e9..7addcd2 100644 --- a/src/agents/orchestrator.ts +++ b/src/agents/orchestrator.ts @@ -15,11 +15,12 @@ Before routing, classify the request into one primary intent: - \`review\`: audit, bug-finding, regression analysis, PR review; handled by the \`/code-review\` command, not a Your Legion runtime agent - \`plan\`: specs, architecture notes, acceptance criteria, execution plans - \`implement\`: approved execution with clear scope, including code, tests, config, analysis, copy, or code-coupled docs -- \`frontend\`: UI, styling, interaction, accessibility, or client-side behavior handled by \`builder\` - \`explore\`: codebase discovery, impact tracing, pattern lookup, unknown-file search - \`docs-research\`: library docs, framework APIs, external references, version behavior - \`orchestrate\`: multi-step work that needs a plan before execution +Intents are work modes (how to work), not domains (what expertise). UI/frontend, finance, marketing, and similar are domains, not work modes: express them through Active domains and Domain skills, never as a separate agent or intent. For example, UI work is the \`implement\` work mode with the coding domain active and \`Domain skills: coding/frontend-change\`. + Route based on the dominant intent. If two intents are both important and sequencing is unclear, favor \`planner\` before implementation. If you cannot choose exactly one specialist because the user's intent, objective, constraints, expected output, or verification is missing, ask one concise clarifying question before using the \`task\` tool. Do not delegate with a guessed agent. Do not invent missing objective, constraints, expected output, or verification. Ignore benchmark metadata such as "Benchmark", "Task", "Variant", titles, or harness labels when routing; route by the actual user task body. @@ -48,11 +49,15 @@ The configured model mapping is an operator-configured capability boundary, not - New feature with unclear scope: route through \`planner\`, then send the approved execution work to \`builder\` - Large feature with multiple tracks: \`planner\` first, then \`builder\` for execution; use \`explorer\`/\`librarian\` only when the next deliverable is discovery or external docs research - Review-only request: direct the user to \`/code-review\`; review is command-owned by default -- UI-only request: \`builder\` +- UI or frontend request: \`builder\` with the coding domain active and \`Domain skills: coding/frontend-change\` - General execution with clear scope: if the task is clear, delegate directly to \`builder\` - Requested repo/local-file discovery as the deliverable: \`explorer\` - Requested third-party documentation research as the deliverable: \`librarian\` +## Post-Execution Verification + +When a \`builder\` deliverable reports Verification status with "Files changed: yes" and "Status: unverified", route a \`verifier\` pass before reporting completion to the user. Pass the completion claim, the changed files, and the verification commands to \`verifier\` in the envelope. If the runtime cannot support another delegation, or the user opted out, surface the unverified status to the user instead of presenting the work as done. Do not treat unverified file changes as complete on the maker's word alone. + ## Delegation Standard When invoking a subagent, provide a compact Task Context Envelope. Every \`task\` tool prompt must start with \`Task Context Envelope:\`. Keep it around 120-180 tokens unless the task is genuinely multi-step. @@ -75,7 +80,7 @@ Task Context Envelope: - Verification outcome: Use Scenario only for a benchmark or validation marker already present in the user request; otherwise write "none". Scenario is a marker only when present, not a place for task prose. -Use Loop only when the user request or Loop Catalog clearly names a configured loop; otherwise write "none". A loop id is a routing and evidence contract, not a place for prose. +Loop engineering is explicit-only. Use Loop only when the user names a configured loop id or provides a loop-prompt Task Context Envelope. If the user asks to use loop engineering but does not name a configured loop id, ask one concise clarifying question before delegation. For all other requests, write "Loop: none". Do not set Loop from task intent, topic, similarity, or the Loop Catalog alone. A loop id is a routing and evidence contract, not a place for prose. Use Loop run only for loop work. Keep the same run id across maker and verifier delegations for one loop attempt; otherwise write "none". Use Loop status as \`started\` for the first maker delegation, \`maker-complete\` when sending the maker result to the verifier, and \`none\` when no loop applies. Use Completion claim, Verification commands, and Verification outcome when passing maker results to verifier. Otherwise write "none". Before delegating, compare the task with the Domain Catalog. Activate every domain whose description materially applies to the delegated work. Active domains must use \`domain-id: responsibility\`, for example "coding: implement UI" and "marketing: write launch copy". For mixed-domain work, name each responsibility directly. Do not blend domain assumptions across responsibilities. diff --git a/src/agents/planner.ts b/src/agents/planner.ts index 03e075e..4009db0 100644 --- a/src/agents/planner.ts +++ b/src/agents/planner.ts @@ -6,7 +6,7 @@ const PROMPT = `# Planner You are the planning specialist for this workspace. -Turn ambiguous requests into implementation-ready design and execution documents without modifying application code. +This prompt defines the planning work mode, not a profession: turn ambiguous requests into implementation-ready design and execution documents without modifying application code. You are domain-neutral. The professional capability for the plan (engineering architecture, campaign structure, financial method, and similar) comes from the Domain skills named in the Task Context Envelope, not from this prompt. ## Enforced File Boundary @@ -26,28 +26,20 @@ Turn ambiguous requests into implementation-ready design and execution documents - Specs and design docs: \`docs/your-legion/specs/YYYY-MM-DD--design.md\` - Implementation plans: \`docs/your-legion/plans/YYYY-MM-DD-.md\` -## Structure And Boundary Heuristics - -- Keep structure proportional to project scale and ownership. Small projects usually need simple, discoverable feature or MVC-style boundaries; larger teams may benefit from grouping by business capability. -- Place user-facing IO close to routes, controllers, views, validators, DTOs, and use cases. Keep database and vendor IO in clients, repositories, DAOs, and adapters. Put business logic between user intent and data access. -- Do not split folders mechanically. Split when it matches how the team changes the code and makes the next feature easier to place. -- Keep repository code focused on persistence access: queries, persistence mapping, aggregate retrieval, and DB-specific details hidden from higher layers. -- Put workflow decisions, authorization policy, pagination for user workflows, user-facing errors, and state transitions in use cases or application services rather than repositories. -- Before introducing domain-based folders, aggregates, repositories, domain services, or clean architecture layers, make an explicit DDD fit call: strong, partial, or weak. Prefer simple structure when the project is early, CRUD-heavy, or lacks shared domain language. - ## Planning Standard When planning work: -1. Read the Task Context Envelope first. Follow its Active domains and Context refs before using broader Domain Pack context. -2. Provider specialization: Trust your specialist responsibility and configured tool boundary; do not split the task into an imagined team. -3. If you read Domain refs or Domain skills, report them under Domain evidence; list the exact catalog ids or paths you actually read. -4. Inspect the relevant code and docs first. -5. Clarify scope with the smallest useful question when requirements are still fuzzy. -6. Define what is in scope, out of scope, and what success looks like. -7. Identify files that will likely change. -8. Break implementation into ordered, verifiable steps. -9. Call out risks, edge cases, and decisions that should not be guessed during execution. +1. Read the Task Context Envelope first. Follow its Active domains, Domain skills, and Context refs before using broader Domain Pack context. +2. Treat the named Domain skills as your capability boundary: read them and apply their structure and decision heuristics for this domain. Do not improvise domain-specific structure that the active domains do not provide. +3. Provider specialization: Trust your specialist responsibility and configured tool boundary; do not split the task into an imagined team. +4. If you read Domain refs or Domain skills, report them under Domain evidence; list the exact catalog ids or paths you actually read. +5. Inspect the relevant code and docs first. +6. Clarify scope with the smallest useful question when requirements are still fuzzy. +7. Define what is in scope, out of scope, and what success looks like. +8. Identify files that will likely change. +9. Break implementation into ordered, verifiable steps. +10. Call out risks, edge cases, and decisions that should not be guessed during execution. If the envelope lacks correctness-critical context, ask instead of guessing. diff --git a/src/agents/verifier.ts b/src/agents/verifier.ts index 72f4b8b..beac332 100644 --- a/src/agents/verifier.ts +++ b/src/agents/verifier.ts @@ -8,12 +8,14 @@ You are the verification specialist for completed or nearly completed work. Your job is to make completion claims meaningful. Keep the maker/checker split clear: the agent that made the change should not be the only one judging whether it is done. +This prompt defines a work mode, not a profession: you are domain-neutral. The specific quality bar comes from the verification commands and the Domain skills declared in the envelope; apply their checks for the work's domain. + ## Review Focus - correctness bugs, behavioral regressions, and missing requirements - whether the stated verification commands actually prove the completion claim - whether Domain evidence and Loop evidence were read when declared -- whether tests protect the rule, not just a convenient return value +- whether the declared verification and domain-skill checks actually hold; for code, whether tests protect the rule, not just a convenient return value - whether a human review inbox needs unresolved items before the loop can continue ## Workflow @@ -22,6 +24,7 @@ Your job is to make completion claims meaningful. Keep the maker/checker split c - If a Loop is named, inspect the loop contract and any referenced inbox or Context refs before judging completion. - If Domain refs or Domain skills are declared, check whether the maker used them or whether missing evidence changes the risk. - Review the actual diff, relevant tests, command output, and completion claim. +- Check Domain update candidates for loop or domain work. Flag missing candidates when the work produced reusable domain judgment, workflow, verification, correction, or decisions; flag noisy candidates when the knowledge is one-off task state. - For loop work, include a Loop Run Report with the same Loop run id from the envelope. Use \`verifier-complete\` only when the completion claim is acceptable, \`blocked\` when human input is needed, and \`failed\` when the claim is rejected. - Do not edit files and do not delegate. @@ -40,7 +43,8 @@ Use this order: - Verification outcome: passed | failed | not-run | unknown 4. Loop evidence: loop id and inbox/context refs inspected, or none 5. Domain evidence: domain refs and domain skills inspected, or none -6. Residual risk or test gaps +6. Domain update candidates: accepted candidates, rejected noisy candidates, missing candidates, or none +7. Residual risk or test gaps For each finding include severity, file and line reference when available, why it matters, and the smallest useful fix. If there are no findings, say that directly and still mention any remaining verification gaps.`; diff --git a/src/cli.ts b/src/cli.ts index 8dd4b6b..786ea28 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -15,25 +15,43 @@ import { analyzeDomainUsageTraceEvents, DOMAIN_USAGE_SCENARIOS, evaluateDomainUsageScenarios, + findUnverifiedFileChangeLedgers, readDomainUsageTraceEvents, } from './runtime/domain-usage-contract'; -import { runYourLegionDoctor } from './runtime/doctor'; +import { diagnoseCustomAgentDomainNeutrality, runYourLegionDoctor } from './runtime/doctor'; +import { + type OrchestrationBenchmarkReport, + parseBenchmarkMetrics, + summarizeOrchestrationBenchmark, +} from './runtime/orchestration-benchmark'; import { AGENT_NAME_PATTERN, resolveLegionariesConfigPath } from './config/legionaries'; function printUsage() { console.log(`Usage: + +Setup: bunx @whchi/your-legion install [--config-dir ] [--domains ] [--add-domains ] + +Domains: bunx @whchi/your-legion create-domain [--config-dir ] [--components ] [--enable] + bunx @whchi/your-legion domain-update --type --title [--config-dir ] [--promote-to ] + +Loops: bunx @whchi/your-legion create-loop [--preset ] [--worktree ] [--config-dir ] [--description ] [--objective ] [--verification ] bunx @whchi/your-legion loops [--config-dir ] bunx @whchi/your-legion loop-presets bunx @whchi/your-legion loop-prompt [--worktree ] [--config-dir ] [--run-id ] bunx @whchi/your-legion loop-runs [--worktree ] [--config-dir ] [--loop ] + +Evidence: bunx @whchi/your-legion doctor [--worktree ] [--config-dir ] [--scenarios] bunx @whchi/your-legion trace [--worktree ] [--config-dir ] [--limit ] [--summary] - bunx @whchi/your-legion trace-check [--worktree ] [--config-dir ] + bunx @whchi/your-legion trace-check [--worktree ] [--config-dir ] [--require-evidence] bunx @whchi/your-legion domain-scenarios - bunx @whchi/your-legion domain-scenario-check [--worktree ] [--config-dir ]`); + bunx @whchi/your-legion domain-scenario-check [--worktree ] [--config-dir ] + +Benchmark: + bunx @whchi/your-legion benchmark-summarize --metrics `); } function optionValue(name: string) { @@ -52,6 +70,8 @@ function hasFlag(name: string) { return process.argv.includes(name); } +const DOMAIN_UPDATE_TYPES = ['decision', 'correction', 'workflow', 'verification', 'reusable'] as const; + function configPathFor(worktree: string) { return resolveLegionariesConfigPath({ rootDir: worktree, @@ -212,6 +232,7 @@ if (command === 'create-domain') { console.log(`Created domain ${result.domainID} at ${result.domainRootPath}`); console.log(`Edit DOMAIN.md: ${result.descriptionPath}`); + console.log(`Domain update inbox: ${result.updatesPath}`); console.log( `Components: ${ result.componentPaths.length === 0 @@ -232,6 +253,60 @@ if (command === 'create-domain') { process.exit(0); } +if (command === 'domain-update') { + const domainID = process.argv[3]; + if (!domainID) { + printUsage(); + process.exit(1); + } + + const type = optionValue('--type'); + if (!type || !DOMAIN_UPDATE_TYPES.includes(type as (typeof DOMAIN_UPDATE_TYPES)[number])) { + console.error(`unknown domain update type: ${type ?? 'none'}`); + console.error(`Available types: ${DOMAIN_UPDATE_TYPES.join(', ')}`); + process.exit(1); + } + + const title = optionValue('--title')?.trim(); + if (!title) { + console.error('missing --title for domain update candidate'); + process.exit(1); + } + + const configDir = optionValue('--config-dir') ?? dirname(configPathFor(process.cwd())); + const domainRootPath = join(configDir, 'your-legion', 'domains', domainID); + const updatesPath = join(domainRootPath, 'updates.md'); + if (!existsSync(join(domainRootPath, 'DOMAIN.md'))) { + console.error(`unknown domain: ${domainID}`); + process.exit(1); + } + if (!existsSync(updatesPath)) { + writeFileSync( + updatesPath, + '# Domain Updates\n\n## Pending\n\n- none\n\n## Promoted\n\n- none\n', + ); + } + + const body = readFileSync(0, 'utf8').trim(); + if (!body) { + console.error('domain update candidate body must be provided on stdin'); + process.exit(1); + } + + const promoteTo = optionValue('--promote-to')?.trim() || 'none'; + const date = new Date().toISOString().slice(0, 10); + const entry = `### ${date} - ${title}\n\nType: ${type}\nPromote to: ${promoteTo}\nStatus: pending\n\n${body}\n`; + const currentUpdates = readFileSync(updatesPath, 'utf8'); + const pendingUpdates = currentUpdates.replace(/## Pending\n\n- none\n/, '## Pending\n\n'); + const nextUpdates = pendingUpdates.includes('\n## Promoted') + ? pendingUpdates.replace('\n## Promoted', `\n${entry}\n## Promoted`) + : `${pendingUpdates.trimEnd()}\n\n${entry}`; + writeFileSync(updatesPath, nextUpdates); + + console.log(`Recorded domain update candidate for ${domainID}: ${updatesPath}`); + process.exit(0); +} + if (command === 'create-loop') { const loopID = process.argv[3]; if (!loopID) { @@ -467,7 +542,7 @@ Domain refs: ${formatList(loop.domain_refs ?? [])} Domain skills: ${formatList(loop.domain_skills ?? [])} Context refs: ${loop.inbox_path} Constraints: Use ${loop.worktree?.isolation ?? 'required'} worktree isolation when the loop touches code or long-running state. -Expected output: Patch or findings, updated loop inbox, verification results, and a Loop Run Report. +Expected output: Patch or findings, updated loop inbox, verification results, Domain update candidates: list reusable domain knowledge candidates or none, and a Loop Run Report. Verification: ${formatList(verificationCommands)} Completion claim: none Verification commands: none @@ -739,9 +814,17 @@ if (command === 'doctor') { configDir: optionValue('--config-dir'), includeScenarios: hasFlag('--scenarios'), }); + const customAgentSection = await diagnoseCustomAgentDomainNeutrality({ + rootDir: worktree, + configDir: optionValue('--config-dir'), + }); + const merged = { + passed: result.passed && customAgentSection.status !== 'FAIL', + sections: [...result.sections, customAgentSection], + }; - printDoctorResult(result); - process.exit(result.passed ? 0 : 1); + printDoctorResult(merged); + process.exit(merged.passed ? 0 : 1); } if (command === 'trace-check') { @@ -752,6 +835,22 @@ if (command === 'trace-check') { }); const warnings = analyzeDomainUsageTraceEvents(events); + if (hasFlag('--require-evidence')) { + const evidenceEvents = events.filter(event => event.event === 'delegation' || event.event === 'loop-run-report'); + if (evidenceEvents.length === 0) { + console.error('trace-check --require-evidence: no delegation or loop-run-report evidence found for this worktree'); + process.exit(1); + } + } + + // Ledger integrity is blocking wherever a ledger exists (ADR 0003 D4), not gated + // behind --require-evidence. + const ledgerFailures = findUnverifiedFileChangeLedgers(events); + if (ledgerFailures.length > 0) { + console.error(ledgerFailures.join('\n')); + process.exit(1); + } + if (warnings.length > 0) { console.error(warnings.join('\n')); process.exit(1); @@ -792,5 +891,54 @@ if (command === 'domain-scenario-check') { process.exit(0); } +function printBenchmarkReport(report: OrchestrationBenchmarkReport) { + const pct = (value: number | null) => (value === null ? 'n/a' : `${(value * 100).toFixed(2)}%`); + + console.log('Per task:'); + for (const task of report.tasks) { + console.log( + ` ${task.taskID} (${task.taskType}): native=${task.nativeTotalTokens} your-legion=${task.yourLegionTotalTokens} delta=${pct(task.netDeltaPct)} -> ${task.outcome}`, + ); + } + + console.log('Outcome summary:'); + for (const outcome of report.byOutcome) { + console.log(` ${outcome.outcome}: ${outcome.tasks}`); + } + + console.log('By variant / task type:'); + for (const group of report.byVariantAndTaskType) { + console.log( + ` ${group.variant} ${group.taskType}: tasks=${group.tasks} passed=${group.passedTasks} tokens=${group.totalTokens} tokens/pass=${group.tokensPerPassedTask ?? 'n/a'}`, + ); + } + + if (report.byProviderProfile) { + console.log('By provider profile (native vs profile):'); + for (const profile of report.byProviderProfile) { + const summary = profile.byOutcome.map(outcome => `${outcome.outcome}=${outcome.tasks}`).join(', '); + console.log(` ${profile.providerProfile}: ${summary || 'no paired tasks'}`); + } + } +} + +if (command === 'benchmark-summarize') { + const metricsPath = optionValue('--metrics'); + if (!metricsPath) { + console.error('benchmark-summarize requires --metrics (JSON array or JSONL of session metrics)'); + process.exit(1); + } + + const resolvedPath = resolve(metricsPath); + if (!existsSync(resolvedPath)) { + console.error(`metrics file not found: ${resolvedPath}`); + process.exit(1); + } + + const report = summarizeOrchestrationBenchmark(parseBenchmarkMetrics(readFileSync(resolvedPath, 'utf8'))); + printBenchmarkReport(report); + process.exit(0); +} + printUsage(); process.exit(1); diff --git a/src/config/legionaries.ts b/src/config/legionaries.ts index 406126b..533541f 100644 --- a/src/config/legionaries.ts +++ b/src/config/legionaries.ts @@ -18,6 +18,7 @@ import { type ResolvedDomainConfigMap, type ResolvedLegionariesMap, type ResolvedLoopConfigMap, + type ResolvedVerificationConfig, type ReasoningEffort, type SystemAgentName, } from '../shared/agent-types'; @@ -276,6 +277,23 @@ function validateLoopConfigMap(loops: Record = {}): Resolved return resolvedLoops; } +function validateVerificationConfig(verification: unknown): ResolvedVerificationConfig { + if (verification === undefined) { + return { default_check: true }; + } + + if (!verification || typeof verification !== 'object' || Array.isArray(verification)) { + throw new Error('legionaries.yaml verification must be a map'); + } + + const defaultCheck = (verification as Record).default_check; + if (defaultCheck !== undefined && typeof defaultCheck !== 'boolean') { + throw new Error('legionaries.yaml verification.default_check must be a boolean'); + } + + return { default_check: defaultCheck ?? true }; +} + function resolveConfiguredMaps(parsed: LegionariesConfig | null) { if (!parsed || typeof parsed !== 'object') { throw new Error('legionaries.yaml missing system_agents map'); @@ -294,6 +312,7 @@ function resolveConfiguredMaps(parsed: LegionariesConfig | null) { customAgents: parsed.custom_agents ?? {}, domains: parsed.domains ?? {}, loops: parsed.loops ?? {}, + verification: parsed.verification, }; } @@ -308,6 +327,7 @@ export function loadLegionariesConfig(options: LoadLegionariesConfigOptions) { const customAgents = validateCustomModelMap(configuredMaps.customAgents); const domains = validateDomainConfigMap(configuredMaps.domains); const loops = validateLoopConfigMap(configuredMaps.loops); + const verification = validateVerificationConfig(configuredMaps.verification); return { filePath, @@ -315,5 +335,6 @@ export function loadLegionariesConfig(options: LoadLegionariesConfigOptions) { customAgents, domains, loops, + verification, }; } diff --git a/src/domains/coding/DOMAIN.md b/src/domains/coding/DOMAIN.md index 9b0424a..cf8e79a 100644 --- a/src/domains/coding/DOMAIN.md +++ b/src/domains/coding/DOMAIN.md @@ -1,6 +1,6 @@ # Coding Domain -Use this domain when the task involves code implementation, tests, debugging, refactoring, configuration changes, migrations, build fixes, developer tooling, or code-coupled documentation. +Use this domain when the task involves code implementation, frontend/UI work, tests, debugging, refactoring, configuration changes, migrations, build fixes, developer tooling, or code-coupled documentation. Do not use this domain for pure marketing copy, standalone financial analysis, or accounting treatment questions unless the task also requires code changes. @@ -15,3 +15,6 @@ Examples: Skills: - `skills/make-code-change/SKILL.md` +- `skills/frontend-change/SKILL.md` +- `skills/debug-fault/SKILL.md` +- `skills/plan-architecture/SKILL.md` diff --git a/src/domains/coding/skills/debug-fault/SKILL.md b/src/domains/coding/skills/debug-fault/SKILL.md new file mode 100644 index 0000000..33103b8 --- /dev/null +++ b/src/domains/coding/skills/debug-fault/SKILL.md @@ -0,0 +1,34 @@ +--- +name: debug-fault +description: Diagnose runtime bugs and build/type failures by isolating the confirmed cause before changing code. +when_to_use: Use when the root cause is unknown, a test fails for unclear reasons, or a build, type, or compile step fails. +signals: + - bug + - regression + - build failure + - type error +--- + +# Debug Fault + +Use this domain skill when the cause of a failure is not yet known. + +## Runtime And Logic Faults + +1. State the symptom and the expected behavior in one sentence each. +2. Split hypotheses into environment, data, and logic before editing. +3. Gather evidence for or against each hypothesis; do not guess. +4. Minimize the reproduction to the smallest failing case. +5. Fix only the confirmed cause, then re-run the reproduction to confirm it is resolved. + +## Build, Type, And Compile Failures + +1. Identify the project build or type-check command. +2. Group errors by file. +3. Fix one build or type error at a time. +4. Re-run verification after each fix so new errors are attributed correctly. + +## Boundaries + +- Do not apply speculative fixes that are not backed by evidence. +- Do not suppress errors to make a build pass without addressing the cause. diff --git a/src/domains/coding/skills/frontend-change/SKILL.md b/src/domains/coding/skills/frontend-change/SKILL.md new file mode 100644 index 0000000..c3a624d --- /dev/null +++ b/src/domains/coding/skills/frontend-change/SKILL.md @@ -0,0 +1,28 @@ +--- +name: frontend-change +description: Implement UI and client-side changes that preserve visual language, responsiveness, accessibility, and loading/empty/error states. +when_to_use: Use for UI, styling, interaction, layout, responsiveness, accessibility, or other client-side behavior changes. +signals: + - ui + - frontend + - styling + - accessibility +--- + +# Frontend Change + +Use this domain skill when the deliverable is UI or client-side behavior. + +## Workflow + +1. Preserve the existing visual language and component patterns; match the surrounding UI. +2. Check responsiveness across the supported breakpoints. +3. Check accessibility: semantic markup, labels, focus order, keyboard operation, and contrast. +4. When behavior changes, cover loading, empty, and error states, not only the happy path. +5. Keep the change client-side; avoid broad backend changes unless the task explicitly requires them. +6. Verify in the relevant component or view before claiming success. + +## Boundaries + +- Do not redesign unrelated screens or introduce a new design system. +- Do not broaden backend or data contracts to satisfy a UI change unless required. diff --git a/src/domains/coding/skills/make-code-change/SKILL.md b/src/domains/coding/skills/make-code-change/SKILL.md index 8f96250..397ad8a 100644 --- a/src/domains/coding/skills/make-code-change/SKILL.md +++ b/src/domains/coding/skills/make-code-change/SKILL.md @@ -21,6 +21,11 @@ Read the relevant coding workflow, inspect the existing public surface and calle 6. Run the focused test, then the relevant broader check. 7. Report changed files, verification results, skipped checks, assumptions, and remaining risk. +## Test And Dependency Choices + +- Choose the narrowest test level that gives confidence: unit for pure behavior, integration for component boundaries, and end-to-end only when the user or system flow matters. +- Keep real code when practical, fake internal boundaries when persistence or IO is not under test, and mock third-party, slow, expensive, or nondeterministic services. + ## Boundaries - Do not invent unrelated features or abstractions. diff --git a/src/domains/coding/skills/plan-architecture/SKILL.md b/src/domains/coding/skills/plan-architecture/SKILL.md new file mode 100644 index 0000000..70eed12 --- /dev/null +++ b/src/domains/coding/skills/plan-architecture/SKILL.md @@ -0,0 +1,26 @@ +--- +name: plan-architecture +description: Choose code structure, module boundaries, and layering proportional to project scale, with an explicit DDD fit call before heavy abstractions. +when_to_use: Use when a plan must decide folder structure, module boundaries, layering, or whether to adopt domain-driven patterns. +signals: + - architecture + - structure + - boundaries + - layering +--- + +# Plan Architecture + +Use this domain skill when a plan needs structural or boundary decisions for code. + +## Heuristics + +- Keep structure proportional to project scale and ownership. Small projects usually need simple, discoverable feature or MVC-style boundaries; larger teams may benefit from grouping by business capability. +- Place user-facing IO close to routes, controllers, views, validators, DTOs, and use cases. Keep database and vendor IO in clients, repositories, DAOs, and adapters. Put business logic between user intent and data access. +- Keep repository code focused on persistence access: queries, persistence mapping, aggregate retrieval, and DB-specific details hidden from higher layers. +- Put workflow decisions, authorization policy, pagination for user workflows, user-facing errors, and state transitions in use cases or application services rather than repositories. +- Do not split folders mechanically. Split when it matches how the team changes the code and makes the next feature easier to place. + +## DDD Fit Call + +Before introducing domain-based folders, aggregates, repositories, domain services, or clean architecture layers, make an explicit DDD fit call: strong, partial, or weak. Prefer simple structure when the project is early, CRUD-heavy, or lacks shared domain language. diff --git a/src/index.ts b/src/index.ts index b75023c..83f42f8 100644 --- a/src/index.ts +++ b/src/index.ts @@ -56,8 +56,10 @@ export { createDomainUsageTraceHooks, DOMAIN_USAGE_SCENARIOS, evaluateDomainUsageScenarios, + findUnverifiedFileChangeLedgers, getDomainUsageTracePath, parseTaskContextEnvelope, + parseVerificationStatusReport, readDomainUsageTraceEvents, validateDomainUsageContract, } from './runtime/domain-usage-contract'; diff --git a/src/install.ts b/src/install.ts index 413189c..2753b8d 100644 --- a/src/install.ts +++ b/src/install.ts @@ -43,6 +43,7 @@ export type CreateDomainPackResult = { domainRootPath: string; componentPaths: string[]; descriptionPath: string; + updatesPath: string; enabled: boolean; enablementSnippet: string; }; @@ -125,6 +126,21 @@ ${sections.join('\n\n')} `; } +function domainUpdatesTemplate() { + return `# Domain Updates + +This file is a candidate inbox for reusable domain knowledge. It is not loaded into runtime until a candidate is promoted into DOMAIN.md-listed components. + +## Pending + +- none + +## Promoted + +- none +`; +} + function scaffoldDomainComponentFiles(domainRootPath: string, components: DomainComponentDir[]) { for (const component of components) { switch (component) { @@ -371,6 +387,7 @@ export function createDomainPack({ const componentPaths = selectedComponents.map(component => join(domainRootPath, component)); const descriptionPath = join(domainRootPath, 'DOMAIN.md'); + const updatesPath = join(domainRootPath, 'updates.md'); mkdirSync(domainRootPath, { recursive: true }); for (const componentPath of componentPaths) { @@ -381,6 +398,9 @@ export function createDomainPack({ if (!existsSync(descriptionPath)) { writeFileSync(descriptionPath, domainDescriptionTemplate(domainID, selectedComponents)); } + if (!existsSync(updatesPath)) { + writeFileSync(updatesPath, domainUpdatesTemplate()); + } if (enable) { enableDomainInLegionariesConfig(configDir, domainID); } @@ -391,6 +411,7 @@ export function createDomainPack({ domainRootPath, componentPaths, descriptionPath, + updatesPath, enabled: enable, enablementSnippet: `domains:\n ${domainID}: true\n`, }; diff --git a/src/runtime/agent-definition-provider.ts b/src/runtime/agent-definition-provider.ts index e832330..6f066d4 100644 --- a/src/runtime/agent-definition-provider.ts +++ b/src/runtime/agent-definition-provider.ts @@ -4,6 +4,7 @@ import { fileURLToPath } from 'node:url'; import YAML from 'yaml'; import { AGENT_FACTORIES } from '../agents/index'; +import { findDomainCapabilityLeaks } from '../shared/domain-neutrality'; import { AGENT_NAMES, type AgentFactory, @@ -149,7 +150,15 @@ function loadCustomAgentFactory(name: string, filePath: string) { throw new Error(`custom agent cannot replace system agent: ${name}`); } - return asAgentFactory(name, loadCustomAgentDefinition(name, filePath)); + const definition = loadCustomAgentDefinition(name, filePath); + const leaks = findDomainCapabilityLeaks(`${definition.description}\n${definition.prompt}`); + if (leaks.length > 0) { + console.warn( + `your-legion: custom agent ${name} is not domain-neutral (leaks: ${leaks.join(', ')}); move professional capability into domain skills (ADR 0003 D3)`, + ); + } + + return asAgentFactory(name, definition); } export async function loadAgentDefinitionProviders( diff --git a/src/runtime/build-agent-config.ts b/src/runtime/build-agent-config.ts index 6c90f56..916bc2f 100644 --- a/src/runtime/build-agent-config.ts +++ b/src/runtime/build-agent-config.ts @@ -135,6 +135,7 @@ export async function buildEffectiveAgentConfig(options: LoadLegionariesConfigOp customAgents: configuredCustomAgents, domains: configuredDomains, loops: configuredLoops, + verification: configuredVerification, filePath: configPath, } = loadLegionariesConfig(options); const providers = await loadAgentDefinitionProviders(options); @@ -170,6 +171,17 @@ export async function buildEffectiveAgentConfig(options: LoadLegionariesConfigOp agent.orchestrator = augmentOrchestratorForCustomAgents(agent.orchestrator, customAgentDefinitions); } + if (agent.orchestrator && configuredVerification.default_check === false) { + agent.orchestrator = { + ...agent.orchestrator, + prompt: `${agent.orchestrator.prompt} + +## Verification Override + +Default post-execution verification is disabled for this workspace. Do not route a verifier pass for builder file changes unless the user explicitly asks. Still surface any reported unverified status to the user.`, + }; + } + const domainPacks = resolveDomainPacks({ configDir: options.configDir ? toPath(options.configDir) : undefined, configPath, diff --git a/src/runtime/doctor.ts b/src/runtime/doctor.ts index 4a367a2..c87f5f9 100644 --- a/src/runtime/doctor.ts +++ b/src/runtime/doctor.ts @@ -12,6 +12,8 @@ import { readDomainUsageTraceEvents, } from './domain-usage-contract'; import { resolveDomainPacks, type DomainPack } from './domain-packs'; +import { loadAgentDefinitionProviders } from './agent-definition-provider'; +import { findDomainCapabilityLeaks } from '../shared/domain-neutrality'; type DomainComponentKind = 'workflows' | 'decisions' | 'examples' | 'skills'; @@ -453,10 +455,12 @@ function loopCatalogSection({ rootDir, loops, domainPacks, + agentModels, }: { rootDir: string | URL; loops: ResolvedLoopConfigMap; domainPacks: DomainPack[]; + agentModels: Record; }): DoctorSection { const failures: string[] = []; const warnings: string[] = []; @@ -475,6 +479,14 @@ function loopCatalogSection({ } if (maker === verifier) { failures.push(`loop maker and verifier must be separate: ${loopID}`); + } else { + const makerModel = agentModels[maker]; + const verifierModel = agentModels[verifier]; + if (makerModel && verifierModel && makerModel === verifierModel) { + warnings.push( + `loop maker (${maker}) and verifier (${verifier}) share model ${makerModel}; a different model for the checker avoids correlated blind spots: ${loopID}`, + ); + } } if (loop.verification.commands.length === 0) { failures.push(`loop has no verification commands: ${loopID}`); @@ -639,6 +651,10 @@ export function runYourLegionDoctor(options: RunYourLegionDoctorOptions): YourLe configPath: config.filePath, domains: config.domains, }); + const agentModels: Record = { + ...Object.fromEntries(Object.entries(config.systemAgents).map(([name, entry]) => [name, entry.model])), + ...Object.fromEntries(Object.entries(config.customAgents).map(([name, entry]) => [name, entry.model])), + }; const sections = [ diagnoseStaticDomainCatalog({ configDir: options.configDir, @@ -650,6 +666,7 @@ export function runYourLegionDoctor(options: RunYourLegionDoctorOptions): YourLe rootDir: options.rootDir, loops: config.loops, domainPacks, + agentModels, }), loopRuntimeSection(options, config.loops), scenarioSection(options), @@ -664,3 +681,38 @@ export function runYourLegionDoctor(options: RunYourLegionDoctorOptions): YourLe export function doctorResultHash(result: YourLegionDoctorResult) { return createHash('sha256').update(JSON.stringify(result)).digest('hex').slice(0, 16); } + +export async function diagnoseCustomAgentDomainNeutrality( + options: LoadLegionariesConfigOptions, +): Promise { + const providers = await loadAgentDefinitionProviders({ + rootDir: options.rootDir, + configDir: options.configDir, + }); + const warnings: string[] = []; + const details: string[] = []; + const customNames = Object.keys(providers.custom).sort(); + + for (const name of customNames) { + const prompt = providers.custom[name]('').prompt; + const leaks = findDomainCapabilityLeaks(prompt); + if (leaks.length > 0) { + warnings.push( + `custom agent ${name} mixes domain capability into a work-mode agent (move to a domain skill): ${leaks.join(', ')}`, + ); + } + details.push(`${name}: ${leaks.length === 0 ? 'domain-neutral' : `leaks ${leaks.join(', ')}`}`); + } + + if (customNames.length === 0) { + details.push('no custom agents discovered'); + } + + return { + name: 'Custom agent neutrality', + status: 'PASS', + failures: [], + warnings, + details, + }; +} diff --git a/src/runtime/domain-usage-contract.ts b/src/runtime/domain-usage-contract.ts index 8952507..66f3b24 100644 --- a/src/runtime/domain-usage-contract.ts +++ b/src/runtime/domain-usage-contract.ts @@ -46,6 +46,14 @@ const FIELD_NAMES: Record = { export type LoopRunStatus = 'started' | 'maker-complete' | 'verifier-complete' | 'blocked' | 'failed'; export type VerificationOutcome = 'passed' | 'failed' | 'not-run' | 'unknown'; +export type VerificationLedgerStatus = 'verified' | 'unverified' | 'verification-skipped'; + +export type VerificationStatusReport = { + filesChanged?: 'yes' | 'no'; + verifierPass?: 'ran' | 'not-run'; + verificationStatus?: VerificationLedgerStatus; + verificationReason?: string; +}; export type ActiveDomainUsage = { id: string; @@ -90,7 +98,11 @@ export type DomainUsageTraceEvent = { completionClaim?: string; verificationCommands?: string[]; verificationOutcome?: VerificationOutcome; - event: 'delegation' | 'domain-read' | 'loop-run-report'; + event: 'delegation' | 'domain-read' | 'loop-run-report' | 'verification-report'; + filesChanged?: 'yes' | 'no'; + verifierPass?: 'ran' | 'not-run'; + verificationStatus?: VerificationLedgerStatus; + verificationReason?: string; domainCatalogHash?: string; domainCatalogSize?: number; toolName?: string; @@ -425,8 +437,20 @@ export function validateDomainUsageContract( return { envelope, warnings }; } +/** + * Resolve a worktree to a stable trace key. Guards the degenerate `/` (and empty) + * input that OpenCode can hand the plugin: hashing it would key traces under a path + * `trace-check --worktree .` never reproduces, so the trace file reads as empty and + * falsely passes. Falling back to the process cwd keeps write-time and check-time in + * the same real workspace. See docs/adr/0004-orchestration-value-evidence.md (D4). + */ +export function normalizeWorktree(worktree: string) { + const resolved = resolve(worktree || '.'); + return resolved === '/' ? resolve(process.cwd()) : resolved; +} + export function getDomainUsageTracePath({ configDir = getOpenCodeConfigDir(), worktree }: DomainUsageTraceOptions) { - const hash = createHash('sha256').update(resolve(worktree)).digest('hex').slice(0, TRACE_HASH_LENGTH); + const hash = createHash('sha256').update(normalizeWorktree(worktree)).digest('hex').slice(0, TRACE_HASH_LENGTH); return join(configDir, 'your-legion', 'traces', `${hash}.jsonl`); } @@ -618,6 +642,83 @@ function extractLoopRunReportText(input: unknown) { return text.slice(markerIndex); } +const FILES_CHANGED_VALUES = new Set(['yes', 'no']); +const VERIFIER_PASS_VALUES = new Set(['ran', 'not-run']); +const VERIFICATION_LEDGER_STATUSES = new Set([ + 'verified', + 'unverified', + 'verification-skipped', +]); + +/** + * Parse the maker's `Verification status:` ledger block (see builder Output Expectations) + * from task output. Deliberately separate from the envelope parser: the block uses generic + * field names like `Status:` that must not become global envelope fields. + */ +export function parseVerificationStatusReport(input: unknown): VerificationStatusReport { + const text = extractText(input); + const markerIndex = text.toLowerCase().lastIndexOf('verification status:'); + if (markerIndex === -1) { + return {}; + } + + const report: VerificationStatusReport = {}; + for (const line of text.slice(markerIndex).split(/\r?\n/).slice(1)) { + const fieldMatch = cleanLine(line).match(/^([A-Za-z][A-Za-z ]+):\s*(.*)$/); + if (!fieldMatch) { + continue; + } + + const field = fieldMatch[1].trim().toLowerCase(); + const value = cleanToken(fieldMatch[2]).toLowerCase(); + if (field === 'files changed' && FILES_CHANGED_VALUES.has(value)) { + report.filesChanged = value as 'yes' | 'no'; + } else if (field === 'verifier pass' && VERIFIER_PASS_VALUES.has(value)) { + report.verifierPass = value as 'ran' | 'not-run'; + } else if (field === 'status' && VERIFICATION_LEDGER_STATUSES.has(value as VerificationLedgerStatus)) { + report.verificationStatus = value as VerificationLedgerStatus; + } else if (field === 'reason') { + report.verificationReason = cleanToken(fieldMatch[2]) || undefined; + } else if (!field.startsWith('files') && !field.startsWith('verifier') && field !== 'status' && field !== 'reason') { + break; + } + } + + return report; +} + +/** + * Blocking ledger check for the non-loop maker/checker path (ADR 0003 D2/D4): + * a maker that reported file changes with status `unverified` must be followed by a + * `verifier` delegation in the same session. `verification-skipped` with a reason is an + * operator opt-out and stays out of this check. + */ +export function findUnverifiedFileChangeLedgers(events: DomainUsageTraceEvent[]): string[] { + const failures: string[] = []; + + events.forEach((event, index) => { + if (event.event !== 'verification-report' || event.filesChanged !== 'yes' || event.verificationStatus !== 'unverified') { + return; + } + + const checkedByVerifier = events + .slice(index + 1) + .some( + candidate => + candidate.event === 'delegation' && + candidate.targetAgent === 'verifier' && + (!event.sessionID || !candidate.sessionID || candidate.sessionID === event.sessionID), + ); + if (!checkedByVerifier) { + failures.push( + `${event.timestamp} verification-report [unverified-file-changes]: maker reported file changes with status unverified and no verifier delegation followed`, + ); + } + }); + + return failures; +} + function createTraceEvent( options: CreateDomainUsageTraceHooksOptions, event: Omit, @@ -626,7 +727,7 @@ function createTraceEvent( version: TRACE_VERSION, contractVersion: `domain-usage-v${TRACE_VERSION}`, timestamp: new Date().toISOString(), - worktree: options.worktree, + worktree: normalizeWorktree(options.worktree), domainCatalogHash: domainCatalogHash(options.domainPacks), domainCatalogSize: options.domainPacks.length, ...event, @@ -862,7 +963,32 @@ export function createDomainUsageTraceHooks(options: CreateDomainUsageTraceHooks if (toolName === 'task') { const envelope = parseTaskContextEnvelope(extractLoopRunReportText(output)); - if (!envelope.loopID || !envelope.loopRunID || !envelope.loopStatus) { + if (envelope.loopID && envelope.loopRunID && envelope.loopStatus) { + write( + createTraceEvent(options, { + sessionID, + delegationID: sessionID ? latestDelegationBySession.get(sessionID) : undefined, + loopID: envelope.loopID, + loopRunID: envelope.loopRunID, + loopStatus: envelope.loopStatus, + completionClaim: envelope.completionClaim, + verificationCommands: envelope.verificationCommands, + verificationOutcome: envelope.verificationOutcome, + event: 'loop-run-report', + toolName, + activeDomains: [], + domainRefs: [], + domainSkills: [], + warnings: [], + }), + ); + return; + } + + // Non-loop maker/checker ledger (ADR 0003 D2): loop runs are covered by the + // loop-run-report path above; everything else with a ledger block lands here. + const report = parseVerificationStatusReport(output); + if (!report.filesChanged || !report.verificationStatus) { return; } @@ -870,14 +996,12 @@ export function createDomainUsageTraceHooks(options: CreateDomainUsageTraceHooks createTraceEvent(options, { sessionID, delegationID: sessionID ? latestDelegationBySession.get(sessionID) : undefined, - loopID: envelope.loopID, - loopRunID: envelope.loopRunID, - loopStatus: envelope.loopStatus, - completionClaim: envelope.completionClaim, - verificationCommands: envelope.verificationCommands, - verificationOutcome: envelope.verificationOutcome, - event: 'loop-run-report', + event: 'verification-report', toolName, + filesChanged: report.filesChanged, + verifierPass: report.verifierPass, + verificationStatus: report.verificationStatus, + verificationReason: report.verificationReason, activeDomains: [], domainRefs: [], domainSkills: [], diff --git a/src/runtime/orchestration-benchmark.ts b/src/runtime/orchestration-benchmark.ts index 2152b8d..855b65c 100644 --- a/src/runtime/orchestration-benchmark.ts +++ b/src/runtime/orchestration-benchmark.ts @@ -1,10 +1,14 @@ export type BenchmarkVariant = 'native-builder' | 'your-legion-orchestrated'; +export type BenchmarkProviderProfile = 'same-provider' | 'mixed-provider'; + export type BenchmarkSessionMetric = { benchmarkID: string; taskID: string; taskType: string; variant: BenchmarkVariant; + /** For orchestrated runs: whether every agent used one provider or the configured per-agent map. */ + providerProfile?: BenchmarkProviderProfile; agent: string; messages: number; tokensInput: number; @@ -65,10 +69,18 @@ export type BenchmarkOutcomeSummary = { tasks: number; }; +export type BenchmarkProviderProfileComparison = { + providerProfile: BenchmarkProviderProfile; + tasks: BenchmarkTaskComparison[]; + byOutcome: BenchmarkOutcomeSummary[]; +}; + export type OrchestrationBenchmarkReport = { tasks: BenchmarkTaskComparison[]; byVariantAndTaskType: BenchmarkVariantTaskTypeSummary[]; byOutcome: BenchmarkOutcomeSummary[]; + /** Present when orchestrated metrics carry a providerProfile: native vs each profile. */ + byProviderProfile?: BenchmarkProviderProfileComparison[]; }; const OUTCOME_ORDER: BenchmarkTaskOutcome[] = [ @@ -177,13 +189,17 @@ function groupBy(values: T[], keyFor: (value: T) => string) { return groups; } -export function summarizeOrchestrationBenchmark(metrics: BenchmarkSessionMetric[]): OrchestrationBenchmarkReport { - const taskGroups = groupBy(metrics, metric => metric.taskID); - const tasks = [...taskGroups.entries()] +function taskComparisons( + metrics: BenchmarkSessionMetric[], + orchestratedFilter?: (metric: BenchmarkSessionMetric) => boolean, +): BenchmarkTaskComparison[] { + return [...groupBy(metrics, metric => metric.taskID).entries()] .map(([taskID, taskMetrics]) => { const taskType = taskMetrics[0]?.taskType ?? 'unknown'; const nativeMetrics = taskMetrics.filter(metric => metric.variant === 'native-builder'); - const orchestratedMetrics = taskMetrics.filter(metric => metric.variant === 'your-legion-orchestrated'); + const orchestratedMetrics = taskMetrics.filter( + metric => metric.variant === 'your-legion-orchestrated' && (!orchestratedFilter || orchestratedFilter(metric)), + ); const nativeTotalTokens = nativeMetrics.reduce((total, metric) => total + totalTokens(metric), 0); const orchestratorTokens = orchestratedMetrics .filter(metric => metric.agent === 'orchestrator') @@ -216,6 +232,16 @@ export function summarizeOrchestrationBenchmark(metrics: BenchmarkSessionMetric[ }; }) .sort((left, right) => left.taskID.localeCompare(right.taskID)); +} + +function outcomeSummary(tasks: BenchmarkTaskComparison[]): BenchmarkOutcomeSummary[] { + return [...groupBy(tasks, task => task.outcome).entries()] + .map(([outcome, group]) => ({ outcome: outcome as BenchmarkTaskOutcome, tasks: group.length })) + .sort((left, right) => outcomeOrder(left.outcome) - outcomeOrder(right.outcome)); +} + +export function summarizeOrchestrationBenchmark(metrics: BenchmarkSessionMetric[]): OrchestrationBenchmarkReport { + const tasks = taskComparisons(metrics); const variantTaskTypeGroups = groupBy(metrics, metric => `${metric.variant}\u0000${metric.taskType}`); const byVariantAndTaskType = [...variantTaskTypeGroups.entries()] @@ -243,16 +269,83 @@ export function summarizeOrchestrationBenchmark(metrics: BenchmarkSessionMetric[ variantOrder(left.variant) - variantOrder(right.variant) || left.taskType.localeCompare(right.taskType), ); - const byOutcome = [...groupBy(tasks, task => task.outcome).entries()] - .map(([outcome, group]) => ({ - outcome: outcome as BenchmarkTaskOutcome, - tasks: group.length, - })) - .sort((left, right) => outcomeOrder(left.outcome) - outcomeOrder(right.outcome)); + const byOutcome = outcomeSummary(tasks); + + const profiles = [ + ...new Set( + metrics + .filter(metric => metric.variant === 'your-legion-orchestrated' && metric.providerProfile) + .map(metric => metric.providerProfile as BenchmarkProviderProfile), + ), + ].sort(); + + const report: OrchestrationBenchmarkReport = { tasks, byVariantAndTaskType, byOutcome }; + + if (profiles.length > 0) { + report.byProviderProfile = profiles.map(profile => { + const profileTasks = taskComparisons(metrics, metric => metric.providerProfile === profile); + return { providerProfile: profile, tasks: profileTasks, byOutcome: outcomeSummary(profileTasks) }; + }); + } + + return report; +} + +/** + * Parse a committed metrics export (JSON array or JSONL, one metric per line) into + * benchmark session metrics. Pure so the CLI (which reads the file) and tests (which + * read a fixture) share the same shape. See docs/adr/0004-orchestration-value-evidence.md (D3). + */ +export function parseBenchmarkMetrics(contents: string): BenchmarkSessionMetric[] { + const trimmed = contents.trim(); + if (!trimmed) { + return []; + } + + const raw: unknown[] = trimmed.startsWith('[') + ? (JSON.parse(trimmed) as unknown[]) + : trimmed.split(/\r?\n/).filter(Boolean).map(line => JSON.parse(line) as unknown); + + return raw.map((entry, index) => { + if (!entry || typeof entry !== 'object') { + throw new Error(`benchmark metric ${index} is not an object`); + } + + const metric = entry as Record; + for (const field of ['taskID', 'taskType', 'agent'] as const) { + if (typeof metric[field] !== 'string') { + throw new Error(`benchmark metric ${index} missing string field: ${field}`); + } + } + if (metric.variant !== 'native-builder' && metric.variant !== 'your-legion-orchestrated') { + throw new Error(`benchmark metric ${index} has invalid variant: ${String(metric.variant)}`); + } + if ( + metric.providerProfile !== undefined && + metric.providerProfile !== 'same-provider' && + metric.providerProfile !== 'mixed-provider' + ) { + throw new Error(`benchmark metric ${index} has invalid providerProfile: ${String(metric.providerProfile)}`); + } - return { - tasks, - byVariantAndTaskType, - byOutcome, - }; + const num = (field: string) => (typeof metric[field] === 'number' ? (metric[field] as number) : 0); + return { + benchmarkID: typeof metric.benchmarkID === 'string' ? metric.benchmarkID : 'unknown', + taskID: metric.taskID as string, + taskType: metric.taskType as string, + variant: metric.variant, + ...(metric.providerProfile ? { providerProfile: metric.providerProfile as BenchmarkProviderProfile } : {}), + agent: metric.agent as string, + messages: num('messages'), + tokensInput: num('tokensInput'), + tokensOutput: num('tokensOutput'), + tokensReasoning: num('tokensReasoning'), + tokensCacheRead: num('tokensCacheRead'), + tokensCacheWrite: num('tokensCacheWrite'), + cost: num('cost'), + passed: metric.passed === true, + reworkTurns: num('reworkTurns'), + traceWarnings: num('traceWarnings'), + }; + }); } diff --git a/src/shared/agent-types.ts b/src/shared/agent-types.ts index bab6028..fad5d05 100644 --- a/src/shared/agent-types.ts +++ b/src/shared/agent-types.ts @@ -112,11 +112,21 @@ export type LoopConfig = { }; }; +export type VerificationConfig = { + /** When true (default), the orchestrator routes an independent verifier pass for unverified builder file changes. */ + default_check?: boolean; +}; + +export type ResolvedVerificationConfig = { + default_check: boolean; +}; + export type LegionariesConfig = { system_agents?: Partial>; custom_agents?: Record; domains?: Record; loops?: Record; + verification?: VerificationConfig; }; export type ResolvedLegionaryEntry = { diff --git a/src/shared/domain-neutrality.ts b/src/shared/domain-neutrality.ts new file mode 100644 index 0000000..af4753e --- /dev/null +++ b/src/shared/domain-neutrality.ts @@ -0,0 +1,20 @@ +/** + * Work-mode agents (bundled and custom) must stay domain-neutral; professional capability + * belongs in domain skills. Single source of truth for the two-axis rule, shared by the + * bundled-agent test and the custom-agent doctor check. + * See docs/adr/0003-enforced-verification-and-maker-checker-integrity.md (D3) and the README Routing Model. + */ +export const DOMAIN_CAPABILITY_VOCAB = [ + 'failing test', + 'TDD', + 'narrowest test level', + 'DDD', + 'persistence access', + 'visual language', + 'responsiveness', + 'brand voice', +]; + +export function findDomainCapabilityLeaks(text: string): string[] { + return DOMAIN_CAPABILITY_VOCAB.filter(term => new RegExp(`\\b${term}\\b`, 'i').test(text)); +} diff --git a/tests/agent-config.test.ts b/tests/agent-config.test.ts index 6080d00..d6263cf 100644 --- a/tests/agent-config.test.ts +++ b/tests/agent-config.test.ts @@ -106,6 +106,21 @@ test('orchestrator prompt keeps every envelope ref field structured', async () = assert.match(orchestrator.prompt, /Do not put prose, explanations, summaries, or parenthetical notes in any refs field/i); }); +test('orchestrator prompt treats loop engineering as explicit-only', async () => { + const { BASE_AGENT_DEFINITIONS } = await import('../src/agents/index'); + const orchestrator = BASE_AGENT_DEFINITIONS.orchestrator; + + assert.match(orchestrator.prompt, /Loop engineering is explicit-only/i); + assert.match(orchestrator.prompt, /user names a configured loop id/i); + assert.match(orchestrator.prompt, /provides a loop-prompt Task Context Envelope/i); + assert.match(orchestrator.prompt, /For all other requests, write "Loop: none"/i); + assert.match(orchestrator.prompt, /Do not set Loop from task intent, topic, similarity, or the Loop Catalog alone/i); + assert.match(orchestrator.prompt, /ask one concise clarifying question/i); + assert.doesNotMatch(orchestrator.prompt, /implicit recognition/i); + assert.doesNotMatch(orchestrator.prompt, /implicitly recognize/i); + assert.doesNotMatch(orchestrator.prompt, /infer a loop/i); +}); + test('orchestrator prompt treats builder as the execution specialist', async () => { const { BASE_AGENT_DEFINITIONS } = await import('../src/agents/index'); const orchestrator = BASE_AGENT_DEFINITIONS.orchestrator; @@ -243,26 +258,47 @@ test('planner is docs-only and cannot modify application code paths', async () = assert.doesNotMatch(planner.prompt, /write markdown planning documents only by convention/i); }); -test('builder prompt carries debugging, testing, and verification workflow cues', async () => { +test('builder prompt stays a domain-neutral execution work mode', async () => { const { BASE_AGENT_DEFINITIONS } = await import('../src/agents/index'); const builder = BASE_AGENT_DEFINITIONS.builder; - assert.match(builder.prompt, /frontend|UI|accessibility/i); - assert.match(builder.prompt, /failing test/i); - assert.match(builder.prompt, /narrowest test level/i); - assert.match(builder.prompt, /environment, data, and logic/i); - assert.match(builder.prompt, /one build or type error at a time/i); + // Work-mode contract stays in the agent. + assert.match(builder.prompt, /execution specialist/i); + assert.match(builder.prompt, /work mode, not a profession/i); + assert.match(builder.prompt, /smallest correct change/i); + assert.match(builder.prompt, /Verify before claiming success/i); assert.match(builder.prompt, /Loop evidence/i); assert.match(builder.prompt, /inbox/i); + // Capability is deferred to Domain skills, not baked into the agent. + assert.match(builder.prompt, /Domain skills as your capability boundary/i); + assert.doesNotMatch(builder.prompt, /failing test/i); + assert.doesNotMatch(builder.prompt, /narrowest test level/i); + assert.doesNotMatch(builder.prompt, /one build or type error at a time/i); }); -test('planner prompt carries structure and boundary review workflow cues', async () => { +test('planner prompt stays a domain-neutral planning work mode', async () => { const { BASE_AGENT_DEFINITIONS } = await import('../src/agents/index'); const planner = BASE_AGENT_DEFINITIONS.planner; - assert.match(planner.prompt, /project scale and ownership/i); - assert.match(planner.prompt, /persistence access/i); - assert.match(planner.prompt, /DDD fit/i); + // Work-mode contract stays in the agent. + assert.match(planner.prompt, /planning work mode/i); + assert.match(planner.prompt, /in scope, out of scope/i); + assert.match(planner.prompt, /ordered, verifiable steps/i); + // Capability is deferred to Domain skills, not baked into the agent. + assert.match(planner.prompt, /Domain skills as your capability boundary/i); + assert.doesNotMatch(planner.prompt, /persistence access/i); + assert.doesNotMatch(planner.prompt, /DDD fit/i); +}); + +test('work-mode agent prompts stay domain-neutral and defer capability to domain skills', async () => { + const { BASE_AGENT_DEFINITIONS } = await import('../src/agents/index'); + // Same rule and word list as the custom-agent doctor check (single source of truth). + const { findDomainCapabilityLeaks } = await import('../src/shared/domain-neutrality'); + + for (const agentName of ['builder', 'planner', 'explorer', 'verifier'] satisfies AgentName[]) { + const leaks = findDomainCapabilityLeaks(BASE_AGENT_DEFINITIONS[agentName].prompt); + assert.deepEqual(leaks, [], `${agentName} should defer domain capability to a domain skill: ${leaks.join(', ')}`); + } }); test('verifier is a protected read-only checker for loop completion claims', async () => { @@ -367,3 +403,16 @@ test('buildAgentDefinition produces valid config from factory', async () => { 'docs/**/*.md': 'allow', }); }); + +test('builder reports a verification-status ledger and orchestrator routes unverified changes', async () => { + const { BASE_AGENT_DEFINITIONS } = await import('../src/agents/index'); + const builder = BASE_AGENT_DEFINITIONS.builder; + const orchestrator = BASE_AGENT_DEFINITIONS.orchestrator; + + assert.match(builder.prompt, /Verification status/i); + assert.match(builder.prompt, /verified \| unverified \| verification-skipped/i); + + assert.match(orchestrator.prompt, /Post-Execution Verification/i); + assert.match(orchestrator.prompt, /Files changed: yes.*Status: unverified/is); + assert.match(orchestrator.prompt, /route a `verifier` pass before reporting completion/i); +}); diff --git a/tests/custom-agent-provider.test.ts b/tests/custom-agent-provider.test.ts index a48d77b..3b51a35 100644 --- a/tests/custom-agent-provider.test.ts +++ b/tests/custom-agent-provider.test.ts @@ -144,6 +144,43 @@ test('repo code-reviewer custom agent is injected from custom-agents example', a assert.equal((result.agent.orchestrator.permission.task as Record)['code-reviewer'], 'allow'); }); +test('custom agent provider warns at load time when a custom agent leaks domain capability', async t => { + const projectDir = makeTempDir(t, 'custom-agent-neutrality-warning'); + const configPath = path.join(projectDir, 'legionaries.yaml'); + const original = YAML.parse(fs.readFileSync(legionariesConfigPath, 'utf8')); + const systemAgents = systemAgentsFrom(original); + + writeCustomAgent(projectDir, 'tdd-coach', 'Coaches TDD discipline and failing test first workflows'); + fs.writeFileSync( + configPath, + YAML.stringify({ + system_agents: systemAgents, + custom_agents: { + 'tdd-coach': 'openai/gpt-5.5', + }, + }), + ); + + const warnings: string[] = []; + const originalWarn = console.warn; + console.warn = (...args: unknown[]) => { + warnings.push(args.join(' ')); + }; + t.after(() => { + console.warn = originalWarn; + }); + + const { buildEffectiveAgentConfig } = await import('../src/runtime/build-agent-config'); + await buildEffectiveAgentConfig({ + rootDir: projectDir, + configPath, + }); + + assert.match(warnings.join('\n'), /custom agent tdd-coach is not domain-neutral/); + assert.match(warnings.join('\n'), /TDD/); + assert.match(warnings.join('\n'), /failing test/); +}); + test('custom agent provider rejects attempts to replace system agents', async t => { const projectDir = makeTempDir(t, 'custom-agent-system-collision'); const configPath = path.join(projectDir, 'legionaries.yaml'); diff --git a/tests/doctor.test.ts b/tests/doctor.test.ts index 3884f80..db59e0a 100644 --- a/tests/doctor.test.ts +++ b/tests/doctor.test.ts @@ -25,17 +25,19 @@ function writeConfig({ configDir, domains, loops, + systemAgents, }: { configDir: string; domains: Record; loops?: Record; + systemAgents?: Record; }) { const original = YAML.parse(fs.readFileSync(legionariesConfigPath, 'utf8')); fs.mkdirSync(configDir, { recursive: true }); fs.writeFileSync( path.join(configDir, 'legionaries.yaml'), YAML.stringify({ - system_agents: original.system_agents, + system_agents: systemAgents ?? original.system_agents, custom_agents: original.custom_agents, domains, ...(loops ? { loops } : {}), @@ -82,7 +84,7 @@ Skills: assert.notEqual(result.status, 0); assert.match(result.stdout + result.stderr, /Your Legion doctor/); assert.match(result.stdout + result.stderr, /Summary:/); - assert.match(result.stdout + result.stderr, /- Sections: 4 passed, 1 failed, 1 skipped/); + assert.match(result.stdout + result.stderr, /- Sections: 5 passed, 1 failed, 1 skipped/); assert.match(result.stdout + result.stderr, /Loop catalog: PASS/); assert.match(result.stdout + result.stderr, /- Findings: 3 failures, 2 warnings/); assert.match(result.stdout + result.stderr, /Static domain catalog: FAIL/); @@ -140,7 +142,7 @@ description: Review product operations constraints. assert.match(output, /Your Legion doctor/); assert.match(output, /Static domain catalog: PASS/); assert.match(output, /Summary:/); - assert.match(output, /- Sections: 5 passed, 0 failed, 1 skipped/); + assert.match(output, /- Sections: 6 passed, 0 failed, 1 skipped/); assert.match(output, /- Findings: 0 failures, 1 warning/); assert.match(output, /Runtime trace diagnostics: PASS/); assert.match(output, /Domain usage stats: PASS/); @@ -186,6 +188,67 @@ test('doctor CLI fails loop catalog validation for missing inbox and unsafe chec assert.match(result.stdout + result.stderr, /loop maker and verifier must be separate: daily-ci-triage/); }); +test('doctor warns when a loop maker and verifier share the same model', async t => { + const configDir = makeTempDir(t, 'your-legion-doctor-same-model-config'); + const worktree = makeTempDir(t, 'your-legion-doctor-same-model-worktree'); + + writeConfig({ + configDir, + domains: {}, + systemAgents: { + orchestrator: { model: 'openai/gpt-5.5' }, + explorer: { model: 'openai/gpt-5.5' }, + librarian: { model: 'openai/gpt-5.5' }, + planner: { model: 'openai/gpt-5.5' }, + builder: { model: 'openai/gpt-5.5' }, + verifier: { model: 'openai/gpt-5.5' }, + }, + loops: { + 'daily-ci-triage': { + description: 'Daily CI triage', + objective: 'Find and fix CI failures', + trigger: { type: 'scheduled', cadence: 'daily' }, + inbox_path: 'docs/legion-loops/daily-ci-triage.md', + agents: { maker: 'builder', verifier: 'verifier' }, + verification: { commands: ['bun test'], completion: 'Tests pass' }, + }, + }, + }); + writeFile(path.join(worktree, 'docs', 'legion-loops', 'daily-ci-triage.md'), '# Daily CI triage\n'); + + const { runYourLegionDoctor } = await import('../src/runtime/doctor'); + const result = runYourLegionDoctor({ rootDir: worktree, configDir }); + const loopCatalog = result.sections.find(section => section.name === 'Loop catalog'); + + assert.ok(loopCatalog); + // Same-model checking is a warning, not a failure: separate agents are still valid config. + assert.equal(loopCatalog.status, 'PASS'); + assert.ok( + loopCatalog.warnings.some(warning => /share model openai\/gpt-5\.5/.test(warning)), + loopCatalog.warnings.join('; '), + ); +}); + +test('doctor flags custom agents that mix domain capability into work-mode agents', async t => { + const worktree = makeTempDir(t, 'your-legion-doctor-custom-agent-worktree'); + writeFile( + path.join(worktree, 'src', 'custom-agents', 'leaky-agent.yaml'), + 'name: leaky-agent\ndescription: Example leaky agent\nprompt: |\n You review UI work and check responsiveness and DDD boundaries.\n', + ); + + const { diagnoseCustomAgentDomainNeutrality } = await import('../src/runtime/doctor'); + const section = await diagnoseCustomAgentDomainNeutrality({ rootDir: worktree }); + + assert.equal(section.name, 'Custom agent neutrality'); + assert.ok( + section.warnings.some( + warning => + /leaky-agent mixes domain capability/.test(warning) && /responsiveness/.test(warning) && /DDD/.test(warning), + ), + section.warnings.join('; '), + ); +}); + test('doctor CLI fails loop runtime evidence when maker has no verifier delegation', async t => { const configDir = makeTempDir(t, 'your-legion-doctor-loop-runtime-config'); const worktree = makeTempDir(t, 'your-legion-doctor-loop-runtime-worktree'); @@ -570,6 +633,7 @@ test('loop-prompt CLI prints a ready task context envelope for a configured loop assert.match(output, /Domain refs: coding\/implementation-loop/); assert.match(output, /Domain skills: coding\/make-code-change/); assert.match(output, /Context refs: docs\/legion-loops\/daily-ci-triage\.md/); + assert.match(output, /Domain update candidates: list reusable domain knowledge candidates or none/); assert.match(output, /Verification: bun test, bun run build/); assert.match(output, /Completion claim: none/); assert.match(output, /Verification outcome: none/); diff --git a/tests/domain-packs.test.ts b/tests/domain-packs.test.ts index 01ec476..8d6d6ac 100644 --- a/tests/domain-packs.test.ts +++ b/tests/domain-packs.test.ts @@ -450,11 +450,15 @@ Skills: ); }); -test('default coding domain resolves from bundled domain files', async () => { +test('default coding domain resolves from bundled domain files', async t => { + // Force bundled resolution: an empty config dir has no global coding domain, + // so the runtime falls back to the bundled src/domains/coding files. + const configDir = makeTempDir(t, 'bundled-coding-config'); const { buildEffectiveAgentConfig } = await import('../src/runtime/build-agent-config'); const result = await buildEffectiveAgentConfig({ rootDir, configPath: legionariesConfigPath, + configDir, }); assert.match(result.agent.orchestrator.prompt, /## Domain Catalog/); @@ -464,6 +468,9 @@ test('default coding domain resolves from bundled domain files', async () => { assert.match(result.agent.orchestrator.prompt, /coding\/engineering-guardrails/); assert.match(result.agent.orchestrator.prompt, /coding\/change-report/); assert.match(result.agent.orchestrator.prompt, /coding\/make-code-change/); + assert.match(result.agent.orchestrator.prompt, /coding\/frontend-change/); + assert.match(result.agent.orchestrator.prompt, /coding\/debug-fault/); + assert.match(result.agent.orchestrator.prompt, /coding\/plan-architecture/); assert.match(result.agent.builder.prompt, /target specialists should read the exact paths/i); }); @@ -551,3 +558,28 @@ test('enabled domains without DOMAIN.md are not injected into the domain catalog assert.doesNotMatch(result.agent.orchestrator.prompt, /empty-domain/); assert.doesNotMatch(result.agent.orchestrator.prompt, /No domain components discovered/); }); + +test('orchestrator gets a verification override only when default_check is disabled', async t => { + const projectDir = makeTempDir(t, 'verification-override-project'); + const original = YAML.parse(fs.readFileSync(legionariesConfigPath, 'utf8')); + const { buildEffectiveAgentConfig } = await import('../src/runtime/build-agent-config'); + + const onPath = path.join(projectDir, 'legionaries.on.yaml'); + fs.writeFileSync(onPath, YAML.stringify({ system_agents: systemAgentsFrom(original), domains: { coding: true } })); + const onResult = await buildEffectiveAgentConfig({ rootDir: projectDir, configPath: onPath }); + assert.match(onResult.agent.orchestrator.prompt, /Post-Execution Verification/); + assert.doesNotMatch(onResult.agent.orchestrator.prompt, /Verification Override/); + + const offPath = path.join(projectDir, 'legionaries.off.yaml'); + fs.writeFileSync( + offPath, + YAML.stringify({ + system_agents: systemAgentsFrom(original), + domains: { coding: true }, + verification: { default_check: false }, + }), + ); + const offResult = await buildEffectiveAgentConfig({ rootDir: projectDir, configPath: offPath }); + assert.match(offResult.agent.orchestrator.prompt, /Verification Override/); + assert.match(offResult.agent.orchestrator.prompt, /Default post-execution verification is disabled/); +}); diff --git a/tests/domain-usage-contract.test.ts b/tests/domain-usage-contract.test.ts index 33ed540..b97f217 100644 --- a/tests/domain-usage-contract.test.ts +++ b/tests/domain-usage-contract.test.ts @@ -914,3 +914,161 @@ test('built server resolves bundled coding domain from dist artifacts', async t assert.match(result.agent.orchestrator.prompt, /coding\/make-code-change/); assert.match(result.agent.orchestrator.prompt, /dist\/domains\/coding/); }); + +test('trace worktree keying normalizes the degenerate root path', async () => { + const { getDomainUsageTracePath, normalizeWorktree } = await import('../src/runtime/domain-usage-contract'); + const configDir = path.join(rootDir, 'temp', 'trace-key-check'); + + assert.equal(normalizeWorktree('/'), path.resolve(process.cwd())); + assert.equal(normalizeWorktree(''), path.resolve(process.cwd())); + + // "/" and the real cwd resolve to the same trace file, so `trace-check --worktree .` + // reads the events written at runtime instead of an empty file that falsely passes. + assert.equal( + getDomainUsageTracePath({ configDir, worktree: '/' }), + getDomainUsageTracePath({ configDir, worktree: process.cwd() }), + ); + assert.notEqual( + getDomainUsageTracePath({ configDir, worktree: '/some/other/project' }), + getDomainUsageTracePath({ configDir, worktree: process.cwd() }), + ); +}); + +test('trace-check --require-evidence fails when no delegation evidence exists', t => { + const configDir = makeTempDir(t, 'trace-check-require-evidence-config'); + const worktree = makeTempDir(t, 'trace-check-require-evidence-worktree'); + + const withoutFlag = spawnSync('bun', ['src/cli.ts', 'trace-check', '--worktree', worktree, '--config-dir', configDir], { + cwd: rootDir, + encoding: 'utf8', + }); + assert.equal(withoutFlag.status, 0); + + const withFlag = spawnSync( + 'bun', + ['src/cli.ts', 'trace-check', '--worktree', worktree, '--config-dir', configDir, '--require-evidence'], + { cwd: rootDir, encoding: 'utf8' }, + ); + assert.notEqual(withFlag.status, 0); + assert.match(withFlag.stdout + withFlag.stderr, /no delegation or loop-run-report evidence/); +}); + +test('domain trace hooks record non-loop verification status reports', async t => { + const { createDomainUsageTraceHooks, readDomainUsageTraceEvents } = await import( + '../src/runtime/domain-usage-contract' + ); + const configDir = makeTempDir(t, 'verification-report-hooks'); + const worktree = path.join(configDir, 'project'); + const hooks = createDomainUsageTraceHooks({ + configDir, + worktree, + domainPacks: [], + }); + + await hooks['tool.execute.after']( + { tool: 'task', sessionID: 'ses_verification_report' }, + { + result: `Changed src/app.ts to fix the crash. +Verification status: +- Files changed: yes +- Verifier pass: not-run +- Status: unverified +- Reason: test runner unavailable in this environment`, + }, + ); + + const events = readDomainUsageTraceEvents({ configDir, worktree }); + + assert.equal(events.length, 1); + assert.equal(events[0].event, 'verification-report'); + assert.equal(events[0].filesChanged, 'yes'); + assert.equal(events[0].verifierPass, 'not-run'); + assert.equal(events[0].verificationStatus, 'unverified'); + assert.equal(events[0].verificationReason, 'test runner unavailable in this environment'); +}); + +test('domain trace hooks skip verification reports without a ledger block', async t => { + const { createDomainUsageTraceHooks, readDomainUsageTraceEvents } = await import( + '../src/runtime/domain-usage-contract' + ); + const configDir = makeTempDir(t, 'verification-report-absent'); + const worktree = path.join(configDir, 'project'); + const hooks = createDomainUsageTraceHooks({ + configDir, + worktree, + domainPacks: [], + }); + + await hooks['tool.execute.after']( + { tool: 'task', sessionID: 'ses_no_ledger' }, + { result: 'Here is the summary you asked for. No files were touched.' }, + ); + + assert.deepEqual(readDomainUsageTraceEvents({ configDir, worktree }), []); +}); + +test('unverified file-change ledgers fail unless a verifier delegation follows', async () => { + const { findUnverifiedFileChangeLedgers } = await import('../src/runtime/domain-usage-contract'); + const base = { + version: 1, + worktree: '/project', + sessionID: 'ses_ledger', + activeDomains: [], + domainRefs: [], + domainSkills: [], + warnings: [], + }; + const report = { + ...base, + timestamp: '2026-07-04T00:00:00.000Z', + event: 'verification-report' as const, + filesChanged: 'yes' as const, + verificationStatus: 'unverified' as const, + }; + const verifierDelegation = { + ...base, + timestamp: '2026-07-04T00:00:01.000Z', + event: 'delegation' as const, + targetAgent: 'verifier', + }; + + assert.match(findUnverifiedFileChangeLedgers([report]).join('\n'), /unverified-file-changes/); + assert.deepEqual(findUnverifiedFileChangeLedgers([report, verifierDelegation]), []); + // A verifier delegation that happened before the maker report does not vouch for it. + assert.match(findUnverifiedFileChangeLedgers([verifierDelegation, report]).join('\n'), /unverified-file-changes/); + + const skipped = { ...report, verificationStatus: 'verification-skipped' as const }; + assert.deepEqual(findUnverifiedFileChangeLedgers([skipped]), []); +}); + +test('trace-check fails when an unverified file-change ledger has no verifier pass', async t => { + const { appendDomainUsageTraceEvent } = await import('../src/runtime/domain-usage-contract'); + const configDir = makeTempDir(t, 'trace-check-unverified-ledger-config'); + const worktree = makeTempDir(t, 'trace-check-unverified-ledger-worktree'); + + appendDomainUsageTraceEvent({ + configDir, + worktree, + event: { + version: 1, + timestamp: '2026-07-04T00:00:00.000Z', + worktree, + sessionID: 'ses_ledger_cli', + event: 'verification-report', + filesChanged: 'yes', + verificationStatus: 'unverified', + activeDomains: [], + domainRefs: [], + domainSkills: [], + warnings: [], + }, + }); + + const check = spawnSync('bun', ['src/cli.ts', 'trace-check', '--worktree', worktree, '--config-dir', configDir], { + cwd: rootDir, + encoding: 'utf8', + }); + + assert.notEqual(check.status, 0); + assert.match(check.stdout + check.stderr, /unverified-file-changes/); +}); diff --git a/tests/fixtures/orchestration-benchmark/2026-05-23-deepseek-v4-pro.jsonl b/tests/fixtures/orchestration-benchmark/2026-05-23-deepseek-v4-pro.jsonl new file mode 100644 index 0000000..fb62bde --- /dev/null +++ b/tests/fixtures/orchestration-benchmark/2026-05-23-deepseek-v4-pro.jsonl @@ -0,0 +1,12 @@ +{"benchmarkID":"yl-orchestrator-vs-native-202605231350pro","taskID":"coding-001","taskType":"coding","variant":"native-builder","agent":"builder","messages":1,"tokensInput":207458,"tokensOutput":0,"tokensReasoning":0,"tokensCacheRead":0,"tokensCacheWrite":0,"cost":0,"passed":true,"reworkTurns":0,"traceWarnings":0} +{"benchmarkID":"yl-orchestrator-vs-native-202605231350pro","taskID":"coding-001","taskType":"coding","variant":"your-legion-orchestrated","agent":"orchestrator","messages":1,"tokensInput":36594,"tokensOutput":0,"tokensReasoning":0,"tokensCacheRead":0,"tokensCacheWrite":0,"cost":0,"passed":true,"reworkTurns":0,"traceWarnings":1} +{"benchmarkID":"yl-orchestrator-vs-native-202605231350pro","taskID":"coding-001","taskType":"coding","variant":"your-legion-orchestrated","agent":"explorer","messages":1,"tokensInput":452826,"tokensOutput":0,"tokensReasoning":0,"tokensCacheRead":0,"tokensCacheWrite":0,"cost":0,"passed":true,"reworkTurns":0,"traceWarnings":0} +{"benchmarkID":"yl-orchestrator-vs-native-202605231350pro","taskID":"marketing-001","taskType":"marketing","variant":"native-builder","agent":"builder","messages":1,"tokensInput":24023,"tokensOutput":0,"tokensReasoning":0,"tokensCacheRead":0,"tokensCacheWrite":0,"cost":0,"passed":true,"reworkTurns":0,"traceWarnings":0} +{"benchmarkID":"yl-orchestrator-vs-native-202605231350pro","taskID":"marketing-001","taskType":"marketing","variant":"your-legion-orchestrated","agent":"orchestrator","messages":1,"tokensInput":35655,"tokensOutput":0,"tokensReasoning":0,"tokensCacheRead":0,"tokensCacheWrite":0,"cost":0,"passed":true,"reworkTurns":0,"traceWarnings":1} +{"benchmarkID":"yl-orchestrator-vs-native-202605231350pro","taskID":"marketing-001","taskType":"marketing","variant":"your-legion-orchestrated","agent":"builder","messages":1,"tokensInput":83305,"tokensOutput":0,"tokensReasoning":0,"tokensCacheRead":0,"tokensCacheWrite":0,"cost":0,"passed":true,"reworkTurns":0,"traceWarnings":0} +{"benchmarkID":"yl-orchestrator-vs-native-202605231350pro","taskID":"finance-001","taskType":"finance","variant":"native-builder","agent":"builder","messages":1,"tokensInput":24509,"tokensOutput":0,"tokensReasoning":0,"tokensCacheRead":0,"tokensCacheWrite":0,"cost":0,"passed":true,"reworkTurns":0,"traceWarnings":0} +{"benchmarkID":"yl-orchestrator-vs-native-202605231350pro","taskID":"finance-001","taskType":"finance","variant":"your-legion-orchestrated","agent":"orchestrator","messages":1,"tokensInput":35954,"tokensOutput":0,"tokensReasoning":0,"tokensCacheRead":0,"tokensCacheWrite":0,"cost":0,"passed":true,"reworkTurns":0,"traceWarnings":1} +{"benchmarkID":"yl-orchestrator-vs-native-202605231350pro","taskID":"finance-001","taskType":"finance","variant":"your-legion-orchestrated","agent":"builder","messages":1,"tokensInput":21050,"tokensOutput":0,"tokensReasoning":0,"tokensCacheRead":0,"tokensCacheWrite":0,"cost":0,"passed":true,"reworkTurns":0,"traceWarnings":0} +{"benchmarkID":"yl-orchestrator-vs-native-202605231350pro","taskID":"accounting-001","taskType":"accounting","variant":"native-builder","agent":"builder","messages":1,"tokensInput":103325,"tokensOutput":0,"tokensReasoning":0,"tokensCacheRead":0,"tokensCacheWrite":0,"cost":0,"passed":true,"reworkTurns":0,"traceWarnings":0} +{"benchmarkID":"yl-orchestrator-vs-native-202605231350pro","taskID":"accounting-001","taskType":"accounting","variant":"your-legion-orchestrated","agent":"orchestrator","messages":1,"tokensInput":39815,"tokensOutput":0,"tokensReasoning":0,"tokensCacheRead":0,"tokensCacheWrite":0,"cost":0,"passed":true,"reworkTurns":0,"traceWarnings":1} +{"benchmarkID":"yl-orchestrator-vs-native-202605231350pro","taskID":"accounting-001","taskType":"accounting","variant":"your-legion-orchestrated","agent":"builder","messages":1,"tokensInput":72050,"tokensOutput":0,"tokensReasoning":0,"tokensCacheRead":0,"tokensCacheWrite":0,"cost":0,"passed":true,"reworkTurns":0,"traceWarnings":0} diff --git a/tests/fixtures/orchestration-benchmark/2026-07-04-exec-quality-3way.jsonl b/tests/fixtures/orchestration-benchmark/2026-07-04-exec-quality-3way.jsonl new file mode 100644 index 0000000..faf3470 --- /dev/null +++ b/tests/fixtures/orchestration-benchmark/2026-07-04-exec-quality-3way.jsonl @@ -0,0 +1,10 @@ +{"benchmarkID":"yl-exec-quality-20260704","taskID":"exec-coding-001","taskType":"exec-coding","messages":1,"cost":0,"reworkTurns":0,"traceWarnings":0,"passed":true,"variant":"native-builder","agent":"builder","tokensInput":15320,"tokensOutput":601,"tokensReasoning":139,"tokensCacheRead":45824,"tokensCacheWrite":0} +{"benchmarkID":"yl-exec-quality-20260704","taskID":"exec-coding-002","taskType":"exec-coding","messages":1,"cost":0,"reworkTurns":0,"traceWarnings":0,"passed":true,"variant":"native-builder","agent":"builder","tokensInput":14982,"tokensOutput":249,"tokensReasoning":86,"tokensCacheRead":14976,"tokensCacheWrite":0} +{"benchmarkID":"yl-exec-quality-20260704sp","taskID":"exec-coding-001","taskType":"exec-coding","messages":1,"cost":0,"reworkTurns":0,"traceWarnings":0,"passed":true,"variant":"your-legion-orchestrated","agent":"orchestrator","providerProfile":"same-provider","tokensInput":6389,"tokensOutput":685,"tokensReasoning":189,"tokensCacheRead":6400,"tokensCacheWrite":0} +{"benchmarkID":"yl-exec-quality-20260704sp","taskID":"exec-coding-001","taskType":"exec-coding","messages":1,"cost":0,"reworkTurns":0,"traceWarnings":0,"passed":true,"variant":"your-legion-orchestrated","agent":"builder","providerProfile":"same-provider","tokensInput":14522,"tokensOutput":1477,"tokensReasoning":431,"tokensCacheRead":84480,"tokensCacheWrite":0} +{"benchmarkID":"yl-exec-quality-20260704sp","taskID":"exec-coding-002","taskType":"exec-coding","messages":1,"cost":0,"reworkTurns":0,"traceWarnings":0,"passed":true,"variant":"your-legion-orchestrated","agent":"orchestrator","providerProfile":"same-provider","tokensInput":6299,"tokensOutput":769,"tokensReasoning":563,"tokensCacheRead":6784,"tokensCacheWrite":0} +{"benchmarkID":"yl-exec-quality-20260704sp","taskID":"exec-coding-002","taskType":"exec-coding","messages":1,"cost":0,"reworkTurns":0,"traceWarnings":0,"passed":true,"variant":"your-legion-orchestrated","agent":"builder","providerProfile":"same-provider","tokensInput":15137,"tokensOutput":927,"tokensReasoning":530,"tokensCacheRead":43392,"tokensCacheWrite":0} +{"benchmarkID":"yl-exec-quality-20260704mp","taskID":"exec-coding-001","taskType":"exec-coding","messages":1,"cost":0,"reworkTurns":0,"traceWarnings":0,"passed":true,"variant":"your-legion-orchestrated","agent":"orchestrator","providerProfile":"mixed-provider","tokensInput":6351,"tokensOutput":822,"tokensReasoning":238,"tokensCacheRead":6400,"tokensCacheWrite":0} +{"benchmarkID":"yl-exec-quality-20260704mp","taskID":"exec-coding-001","taskType":"exec-coding","messages":1,"cost":0,"reworkTurns":0,"traceWarnings":0,"passed":true,"variant":"your-legion-orchestrated","agent":"builder","providerProfile":"mixed-provider","tokensInput":11664,"tokensOutput":1695,"tokensReasoning":0,"tokensCacheRead":33112,"tokensCacheWrite":0} +{"benchmarkID":"yl-exec-quality-20260704mp","taskID":"exec-coding-002","taskType":"exec-coding","messages":1,"cost":0,"reworkTurns":0,"traceWarnings":0,"passed":true,"variant":"your-legion-orchestrated","agent":"orchestrator","providerProfile":"mixed-provider","tokensInput":6315,"tokensOutput":674,"tokensReasoning":365,"tokensCacheRead":6528,"tokensCacheWrite":0} +{"benchmarkID":"yl-exec-quality-20260704mp","taskID":"exec-coding-002","taskType":"exec-coding","messages":1,"cost":0,"reworkTurns":0,"traceWarnings":0,"passed":true,"variant":"your-legion-orchestrated","agent":"builder","providerProfile":"mixed-provider","tokensInput":11170,"tokensOutput":1029,"tokensReasoning":0,"tokensCacheRead":32170,"tokensCacheWrite":0} diff --git a/tests/installer.test.ts b/tests/installer.test.ts index 69971cd..157f7ed 100644 --- a/tests/installer.test.ts +++ b/tests/installer.test.ts @@ -447,10 +447,13 @@ test('create-domain cli scaffolds a domain under an explicit config dir', t => { assert.match(output, /Created domain marketing-ops/); assert.match(output, /Edit DOMAIN\.md:/); assert.match(output, new RegExp(path.join(configDir, 'your-legion', 'domains', 'marketing-ops', 'DOMAIN.md').replace(/[.*+?^${}()|[\]\\]/g, '\\$&'))); + assert.match(output, /Domain update inbox:/); + assert.match(output, new RegExp(path.join(configDir, 'your-legion', 'domains', 'marketing-ops', 'updates.md').replace(/[.*+?^${}()|[\]\\]/g, '\\$&'))); assert.match(output, /Authoring guide: docs\/DOMAIN_PACK_AUTHORING\.md/); assert.match(output, /Verify after use:/); assert.match(output, /bunx @whchi\/your-legion doctor --worktree \./); assert.equal(fs.existsSync(path.join(configDir, 'your-legion', 'domains', 'marketing-ops', 'DOMAIN.md')), true); + assert.equal(fs.existsSync(path.join(configDir, 'your-legion', 'domains', 'marketing-ops', 'updates.md')), true); assert.equal(fs.existsSync(path.join(configDir, 'your-legion', 'domains', 'marketing-ops', 'README.md')), false); }); @@ -487,6 +490,77 @@ test('create-domain cli rejects an existing domain', t => { assert.match(result.stderr + result.stdout, /domain already exists: product-ops/); }); +test('domain-update cli appends a pending domain knowledge candidate', t => { + const configDir = makeTempDir(t, 'your-legion-domain-update-cli'); + execFileSync('bun', ['src/cli.ts', 'create-domain', 'product-ops', '--config-dir', configDir], { + cwd: rootDir, + encoding: 'utf8', + }); + + const output = execFileSync( + 'bun', + [ + 'src/cli.ts', + 'domain-update', + 'product-ops', + '--config-dir', + configDir, + '--type', + 'decision', + '--title', + 'Plugin loading source', + '--promote-to', + 'decisions/plugin-loading.md', + ], + { + cwd: rootDir, + encoding: 'utf8', + input: 'OpenCode may load the published plugin instead of local dist.\n', + }, + ); + const updates = fs.readFileSync(path.join(configDir, 'your-legion', 'domains', 'product-ops', 'updates.md'), 'utf8'); + + assert.match(output, /Recorded domain update candidate for product-ops/); + assert.match(updates, /## Pending/); + assert.match(updates, /### .* - Plugin loading source/); + assert.match(updates, /Type: decision/); + assert.match(updates, /Promote to: decisions\/plugin-loading\.md/); + assert.match(updates, /OpenCode may load the published plugin instead of local dist\./); + assert.match(updates, /Status: pending/); + assert.ok(updates.indexOf('Plugin loading source') < updates.indexOf('## Promoted')); +}); + +test('domain-update cli rejects unknown candidate types', t => { + const configDir = makeTempDir(t, 'your-legion-domain-update-cli-invalid-type'); + execFileSync('bun', ['src/cli.ts', 'create-domain', 'product-ops', '--config-dir', configDir], { + cwd: rootDir, + encoding: 'utf8', + }); + + const result = spawnSync( + 'bun', + [ + 'src/cli.ts', + 'domain-update', + 'product-ops', + '--config-dir', + configDir, + '--type', + 'note', + '--title', + 'Nope', + ], + { + cwd: rootDir, + encoding: 'utf8', + input: 'not reusable\n', + }, + ); + + assert.notEqual(result.status, 0); + assert.match(result.stderr + result.stdout, /unknown domain update type: note/); +}); + test('create-domain cli can enable the domain in legionaries.yaml', t => { const configDir = makeTempDir(t, 'your-legion-domain-cli-enable'); execFileSync('bun', ['src/cli.ts', 'install', '--config-dir', configDir], { diff --git a/tests/legionaries.test.ts b/tests/legionaries.test.ts index 9ba2112..1f5c7cb 100644 --- a/tests/legionaries.test.ts +++ b/tests/legionaries.test.ts @@ -51,13 +51,14 @@ test('legionaries template explains model choice by agent responsibility', () => assert.doesNotMatch(text, /\bprofile:|\bpreset:|\brole_type:/); }); -test('public docs position provider mapping before diagnostics', () => { +test('public docs position the verification ledger before provider mapping', () => { const readme = fs.readFileSync(path.join(rootDir, 'README.md'), 'utf8'); const configuration = fs.readFileSync(path.join(rootDir, 'docs', 'CONFIGURATION.md'), 'utf8'); + // README leads with the maker/checker ledger story; provider mapping supports it. assert.ok( - readme.indexOf('per-agent provider/model mapping') !== -1 && - readme.indexOf('per-agent provider/model mapping') < readme.indexOf('doctor'), + readme.indexOf('Verification status') !== -1 && + readme.indexOf('Verification status') < readme.indexOf('per-agent provider/model mapping'), ); assert.match(configuration, /How To Choose Models/i); assert.ok(configuration.indexOf('How To Choose Models') < configuration.indexOf('Domain Packs')); @@ -160,6 +161,25 @@ test('legionaries loader supports per-agent overrides via config override', asyn assert.equal(result.systemAgents.builder.model, 'github-copilot/gemini-3.1-pro-preview'); }); +test('legionaries loader defaults verification.default_check to true and validates overrides', async () => { + fs.mkdirSync(tempDir, { recursive: true }); + const original = YAML.parse(fs.readFileSync(legionariesConfigPath, 'utf8')); + const systemAgents = systemAgentsFrom(original); + const { loadLegionariesConfig } = await import('../src/config/legionaries'); + + const defaultPath = path.join(tempDir, 'legionaries.verification-default.yaml'); + fs.writeFileSync(defaultPath, YAML.stringify({ system_agents: systemAgents })); + assert.equal(loadLegionariesConfig({ rootDir, configPath: defaultPath }).verification.default_check, true); + + const offPath = path.join(tempDir, 'legionaries.verification-off.yaml'); + fs.writeFileSync(offPath, YAML.stringify({ system_agents: systemAgents, verification: { default_check: false } })); + assert.equal(loadLegionariesConfig({ rootDir, configPath: offPath }).verification.default_check, false); + + const badPath = path.join(tempDir, 'legionaries.verification-bad.yaml'); + fs.writeFileSync(badPath, YAML.stringify({ system_agents: systemAgents, verification: { default_check: 'yes' } })); + assert.throws(() => loadLegionariesConfig({ rootDir, configPath: badPath }), /default_check must be a boolean/); +}); + test('legionaries loader accepts custom_agents with the same entry shape', async () => { fs.mkdirSync(tempDir, { recursive: true }); const tempConfigPath = path.join(tempDir, 'legionaries.custom-agents.yaml'); diff --git a/tests/orchestration-benchmark.test.ts b/tests/orchestration-benchmark.test.ts index 8d9e62b..e44902c 100644 --- a/tests/orchestration-benchmark.test.ts +++ b/tests/orchestration-benchmark.test.ts @@ -4,7 +4,11 @@ import path from 'node:path'; import test from 'node:test'; import { fileURLToPath } from 'node:url'; -import { summarizeOrchestrationBenchmark, type BenchmarkSessionMetric } from '../src/runtime/orchestration-benchmark'; +import { + parseBenchmarkMetrics, + summarizeOrchestrationBenchmark, + type BenchmarkSessionMetric, +} from '../src/runtime/orchestration-benchmark'; const __filename = fileURLToPath(import.meta.url); const __dirname = path.dirname(__filename); @@ -271,3 +275,94 @@ test('benchmark protocol distinguishes same-provider and mixed-provider orchestr assert.match(benchmarkDoc, /pass rate/i); assert.match(benchmarkDoc, /cost, speed, or quality/i); }); + +test('summarizes native against same-provider and mixed-provider orchestration separately', () => { + const report = summarizeOrchestrationBenchmark([ + metric({ taskID: 'coding-001', variant: 'native-builder', agent: 'builder', tokensInput: 100, passed: true }), + metric({ + taskID: 'coding-001', + variant: 'your-legion-orchestrated', + providerProfile: 'same-provider', + agent: 'orchestrator', + tokensInput: 120, + passed: true, + }), + metric({ + taskID: 'coding-001', + variant: 'your-legion-orchestrated', + providerProfile: 'mixed-provider', + agent: 'orchestrator', + tokensInput: 300, + passed: true, + }), + ]); + + assert.ok(report.byProviderProfile); + assert.deepEqual( + report.byProviderProfile.map(profile => profile.providerProfile), + ['mixed-provider', 'same-provider'], + ); + + const same = report.byProviderProfile.find(profile => profile.providerProfile === 'same-provider'); + const mixed = report.byProviderProfile.find(profile => profile.providerProfile === 'mixed-provider'); + // Each profile is compared against native independently, not summed together. + assert.equal(same?.tasks[0].yourLegionTotalTokens, 120); + assert.equal(mixed?.tasks[0].yourLegionTotalTokens, 300); + assert.equal(same?.tasks[0].outcome, 'more-expensive-not-better'); + assert.equal(mixed?.tasks[0].outcome, 'more-expensive-not-better'); +}); + +test('parseBenchmarkMetrics accepts JSON array and rejects invalid variants', () => { + const rows = parseBenchmarkMetrics( + '[{"taskID":"t","taskType":"coding","variant":"native-builder","agent":"builder","tokensInput":10,"passed":true}]', + ); + + assert.equal(rows.length, 1); + assert.equal(rows[0].variant, 'native-builder'); + assert.equal(rows[0].tokensInput, 10); + assert.throws(() => parseBenchmarkMetrics('[{"taskID":"t","taskType":"coding","variant":"bogus","agent":"x"}]')); +}); + +test('benchmark-summarize reproduces the committed 2026-05-23 run outcome', () => { + const fixture = fs.readFileSync( + path.join(rootDir, 'tests', 'fixtures', 'orchestration-benchmark', '2026-05-23-deepseek-v4-pro.jsonl'), + 'utf8', + ); + const report = summarizeOrchestrationBenchmark(parseBenchmarkMetrics(fixture)); + + assert.equal(report.tasks.length, 4); + for (const task of report.tasks) { + assert.equal(task.outcome, 'more-expensive-not-better', `${task.taskID} outcome`); + } + assert.deepEqual(report.byOutcome, [{ outcome: 'more-expensive-not-better', tasks: 4 }]); + + const nativeTotal = report.byVariantAndTaskType + .filter(group => group.variant === 'native-builder') + .reduce((sum, group) => sum + group.totalTokens, 0); + const yourLegionTotal = report.byVariantAndTaskType + .filter(group => group.variant === 'your-legion-orchestrated') + .reduce((sum, group) => sum + group.totalTokens, 0); + + assert.equal(nativeTotal, 359315); + assert.equal(yourLegionTotal, 777249); +}); + +test('benchmark-summarize reproduces the committed 2026-07-04 exec-quality 3-way outcome', () => { + const fixture = fs.readFileSync( + path.join(rootDir, 'tests', 'fixtures', 'orchestration-benchmark', '2026-07-04-exec-quality-3way.jsonl'), + 'utf8', + ); + const report = summarizeOrchestrationBenchmark(parseBenchmarkMetrics(fixture)); + + assert.equal(report.tasks.length, 2); + assert.ok(report.byProviderProfile); + const profiles = Object.fromEntries(report.byProviderProfile.map(profile => [profile.providerProfile, profile])); + + // Same-provider orchestration only added routing overhead on this suite. + assert.deepEqual(profiles['same-provider'].byOutcome, [{ outcome: 'more-expensive-not-better', tasks: 2 }]); + // Mixed-provider matched native cost on exec-coding-001 and stayed same-quality. + assert.deepEqual(profiles['mixed-provider'].byOutcome, [ + { outcome: 'cheaper-same-quality', tasks: 1 }, + { outcome: 'more-expensive-not-better', tasks: 1 }, + ]); +});