diff --git a/AGENTS.md b/AGENTS.md index 420215e..68f5898 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -32,7 +32,7 @@ - Custom profiles may set `backend: pi` (default), `backend: codex`, or `backend: claude`. Codex-backed profiles run external `codex exec --json --dangerously-bypass-approvals-and-sandbox --ephemeral -- -` for one-shot calls, omit `--ephemeral` for keyed first calls, and use `codex exec resume --json ... -` for keyed continuation; they send the task prompt on stdin, pass the profile body as `developer_instructions`, pass profile `model`/`thinking` through Codex CLI, parse `thread.started.thread_id`, token usage from Codex JSONL events, and estimate cost for listed models. Claude-backed profiles run external `claude -p --output-format stream-json --verbose --dangerously-skip-permissions --no-session-persistence` for one-shot calls, omit `--no-session-persistence` for keyed first calls, add `--resume ` for keyed continuation, send the task prompt on stdin, pass the profile body as `--append-system-prompt`, pass profile `model`/`thinking` through Claude Code, parse `system/init.session_id`, parse token usage from stream JSON, and use Claude Code's reported `total_cost_usd` when available. External CLI backends intentionally run in yolo/no-approval mode; only use them in trusted repositories. - `tools` frontmatter is a pi-backend child-session allowlist only. External CLI profiles use their CLI's own tool and permission surface. - There is no pi-flow permissions system in v1. Profiles are ordinary agents with optional prompts and tool allow-lists; external backends are explicit user dependencies. -- Pi-backed child sessions cannot launch other pi subagents. Do not give pi child sessions the `Agent` or `workflow` tool, or the coordinator prompt. +- Pi-backed child sessions cannot launch other pi subagents. Do not give pi child sessions the `Agent` or `workflow` tool, or the flow prompt (`buildFlowPrompt`). - External CLI backends are not given pi `Agent`/`workflow` tools, but their own CLIs may expose nested/delegation features; do not try to block that from this extension. - Parallel delegation is allowed and bounded by a global `maxConcurrentSubagents` limit (default `12`), which caps how many subagents run concurrently across the whole agent run. A slot is taken on launch and released on completion/failure/abort. In v2 this same cap is shared with the `workflow` tool. - Do not put exact concurrency values in the model-facing coordinator prompt. The prompt should say parallel delegation is bounded and queued. @@ -74,7 +74,7 @@ ## CI and release workflow - CI lives in `.github/workflows/ci.yml` and runs on pull requests plus pushes to `main`. It installs with `npm ci` and runs `npm run check` on Node 22.x and 24.x. -- E2E scripts are intentionally not part of required CI because they use real models and can be slow or inconclusive. Run them manually before risky releases: `npm run e2e -- --timeout-ms 300000`, `npm run e2e:workflow-features`, and `npm run e2e:session-key-resume -- --backend all` when session continuation changes. +- E2E scripts are intentionally not part of required CI because they use real models and can be slow or inconclusive. Run them manually before risky releases: `npm run e2e -- --timeout-ms 300000`, `npm run e2e:workflow-features`, `npm run e2e:prompt-routing` when prompt or routing guidance changes, and `npm run e2e:session-key-resume -- --backend all` when session continuation changes. - Every real-model E2E driver must install the guard from `scripts/e2e/lib/deepseek-claude-env.mjs` so any Claude Code process routes through DeepSeek's Anthropic-compatible endpoint with isolated settings. Drivers must fail fast without `DEEPSEEK_API_KEY`/`DEEPSEEK_API_TOKEN` (or `--deepseek-api-key-env`) and must not fall back to Anthropic login or another Claude Code provider. - There is intentionally no automated npm publish workflow right now; do not create tags expecting GitHub Actions to publish, and do not add an `NPM_TOKEN`-based workflow unless the user asks. - Normal version-prep steps for agents: @@ -98,12 +98,16 @@ Interactive tmux TUI runs use `deepseek/deepseek-v4-flash` with high thinking and isolated `--no-*` resource flags. - `width`: validates eight parallel foreground delegations. -- `proactive-multirepo-v3`: validates proactive parallel delegation for a two-repo auth comparison. -- `proactive-fanout-v3`: validates proactive multi-lane delegation for TODO/FIXME/skipped-test search. -- `proactive-migration-v2`: validates proactive second-opinion delegation for a risky migration review. +- `proactive-multirepo-v3`: validated two-repo parallel Agent fan-out under the previous routing contract. +- `proactive-fanout-v3`: validated three-lane TODO/FIXME/skipped-test Agent fan-out under the previous routing contract. +- `proactive-migration-v2`: validated second-opinion Agent delegation under the previous routing contract. - `max-concurrent-queue`: validates `--max-concurrent-subagents 1` with two parallel normal `Agent` calls; both completed (`FIRST_OK`, `SECOND_OK`) and no max-concurrency rejection was emitted. -Do not count `proactive-ship-v3` as proactive-pass evidence: the model handled that tiny ship-readiness fixture directly. This is acceptable as a behavioral limitation, but future prompt/tool tuning should continue improving this case. +Root-direct handling of narrow or small fixtures is intentional: `DIRECT_WORK_POLICY` explicitly permits staying in the root when delegation adds no value. + +### Prompt routing behavior (current contract) + +`scripts/e2e/prompt-routing-evidence.md` records the before/after comparison for the prompt consolidation. It covers the accepted routing boundary (root-direct, flat Agent fan-out, workflow for staged/structured/replay/large), privacy and isolation guarantees, prompt-size measurements, real-model routing outcomes, and variance limitations. The committed deterministic suite in `test/agent-contract.test.ts` and `test/delegation-scenarios.test.ts` covers the active-tool-gated sections, profile-roster deduplication, and session-key continuation contract without a real model. ### Workflow tool (v2) diff --git a/README.md b/README.md index bdaed13..eef9b6b 100644 --- a/README.md +++ b/README.md @@ -20,8 +20,8 @@ Pi stays in control while `pi-flow` gives it two new ways to delegate: | Primitive | Best for | | --- | --- | -| **`Agent`** | One focused task: explore a repository, review a diff, investigate a bug, or draft a solution. | -| **`workflow`** | Several independent tasks: parallel research, multi-model review, staged pipelines, and synthesized results. | +| **`Agent`** | One focused delegated task or a small flat fan-out of independent work. | +| **`workflow`** | Saved workflows, dependent stages, control flow, structured results or branching, replay, or larger fan-out. | Each subagent can run through a different harness and model. Route fast searches to Codex, UI review to Claude Code, local analysis to Pi—or define your own mix. @@ -58,10 +58,10 @@ Then ask Pi to delegate naturally: Use a subagent to map this repository without changing files, then summarize the important entry points. ``` -Or ask for a wider workflow: +Or ask for staged orchestration: ```text -Review this PR from three independent angles—correctness, security, and test coverage—and synthesize the findings. +Use a workflow to classify each changed file by risk, run a risk-specific review, and return structured results. ``` Pi decides how to invoke `Agent` or `workflow`, shows live progress, and returns the combined result in the same conversation. @@ -102,7 +102,7 @@ The lower-level `@kky42/pi-flow/runtime` export remains available for consumers ## Define your agent team -`pi-flow` includes one profile, `general-purpose`. Add specialists as Markdown files under: +`pi-flow` includes a minimal built-in profile and loads arbitrary custom profiles from Markdown files under: ```text ~/.pi/agent/subagents/.md @@ -130,7 +130,6 @@ Map the repository without modifying files. Return concise findings with paths a --- description: Reviews implementation changes for correctness and missed edge cases. backend: codex -model: gpt-5.5 thinking: high --- @@ -143,7 +142,6 @@ Review the current diff. Lead with concrete findings and identify missing tests. --- description: Reviews frontend work for UX, accessibility, and visual quality. backend: claude -model: sonnet thinking: high --- @@ -168,7 +166,6 @@ A fresh call starts a clean child conversation in the same working directory. Pa ```ts Agent({ description: "Map the authentication flow", - subagent_type: "explorer", prompt: "Trace login from the HTTP entry point to session creation. Do not edit files.", }); ``` @@ -178,14 +175,12 @@ For follow-up work, reuse a stable `session_key`: ```ts Agent({ description: "Draft the migration", - subagent_type: "implementation-expert", session_key: "auth-migration", prompt: "Propose a migration plan based on the current implementation.", }); Agent({ description: "Revise the migration", - subagent_type: "implementation-expert", session_key: "auth-migration", prompt: "Revise the plan using the review feedback. Address rollback and compatibility.", }); @@ -193,15 +188,15 @@ Agent({ `pi-flow` maps that key to the backend-native session or thread and keeps the continuation explicit. -## Fan out with workflows +## Choose Agent or workflow by task shape -Use `workflow` when several lanes can run independently or when work benefits from multiple perspectives. +Use parallel `Agent` calls for a small flat set of independent investigations. Use `workflow` when orchestration matches a saved workflow or needs dependent stages, control flow, structured results or decisions, replay, or larger fan-out. ```text -Run a workflow that asks one agent to inspect the API, one to inspect persistence, and one to inspect tests. Synthesize the highest-risk gaps. +Use a workflow to classify each module with a strict risk result, then run the matching follow-up review for each classification. ``` -You usually do not need to write workflow code yourself. Pi can generate and run a small trusted JavaScript workflow, then present the result. +You usually do not need to write workflow code yourself. Pi can generate and run a trusted JavaScript workflow, then present the result. Ask Pi to save repeatable orchestration: diff --git a/package.json b/package.json index 5372f92..153aeaf 100644 --- a/package.json +++ b/package.json @@ -44,6 +44,7 @@ "smoke:runtime": "npm run build:dist && node scripts/smoke-runtime.mjs", "e2e": "node scripts/e2e/basic-metrics.mjs", "e2e:workflow-features": "node scripts/e2e/workflow-features.mjs", + "e2e:prompt-routing": "node scripts/e2e/prompt-routing.mjs", "e2e:session-key-resume": "node scripts/e2e/session-key-resume.mjs", "test": "vitest run", "prepack": "npm run build:dist" diff --git a/scripts/e2e/prompt-routing-evidence.md b/scripts/e2e/prompt-routing-evidence.md new file mode 100644 index 0000000..33ca44c --- /dev/null +++ b/scripts/e2e/prompt-routing-evidence.md @@ -0,0 +1,123 @@ +# Prompt-routing behavior evidence + +This document records the bounded real-model comparison for the prompt consolidation based on commit `8954ab8` (`v2.1.5`). It is maintainer evidence, not a claim that stochastic routing is deterministic. + +## Accepted behavior boundary + +- Narrow local work may stay in the root. +- One focused delegation or a small flat fan-out uses direct `Agent` calls. +- Saved workflows, dependent stages, control flow, structured results or branching, replay, and larger fan-out use `workflow`. +- One logical child stream may reuse a `session_key`; independent work stays fresh. +- Routing expectations do not name or assert any profile, child model, specialist, or default. + +## Privacy and isolation + +`scripts/e2e/prompt-routing.mjs` creates a five-file report-CLI fixture and an isolated Pi agent directory. It disables discovered extensions, skills, templates, themes, context files, and session persistence, then loads only the extension under test. It checks that every fixture file remains byte-for-byte unchanged. + +The driver does not read normal Pi session history or copy local profiles. The Pi process receives a minimal environment containing basic process settings plus one resolved DeepSeek credential; unrelated and alternate credential variables are not forwarded. Only terminal JSON events needed to assess tool calls, results, and usage are retained, high-volume streaming updates are discarded, and every discovered DeepSeek credential value plus local root paths are redacted before artifacts are analyzed or written. The driver always removes its isolated child-session directory and Claude runtime directory, even when other sanitized artifacts are retained. + +## Reproduction + +Prerequisites for the recorded run: + +- Node `v22.23.1` +- npm `10.9.8` +- Pi `0.83.0` +- `DEEPSEEK_API_KEY` or `DEEPSEEK_API_TOKEN` + +The driver intentionally requires the root model and thinking level instead of encoding defaults: + +```bash +npm run e2e:prompt-routing -- \ + --model deepseek/deepseek-v4-flash \ + --thinking high \ + --repetitions 2 +``` + +Useful scoped form: + +```bash +npm run e2e:prompt-routing -- \ + --model deepseek/deepseek-v4-flash \ + --thinking high \ + --repetitions 2 \ + --only flat,continuation,staged +``` + +The run installs the repository's DeepSeek Claude-provider guard even though these scenarios use Pi-backed children. `--extension ` can compare another checkout or archived baseline. `--run-root` must identify a new or empty real directory; the driver marks ownership before writing and refuses to remove an unmarked root. An explicit `--agent-dir` must stay outside that root, while omitting it uses a driver-owned isolated directory. `--keep` retains sanitized fixture artifacts and `report.json`, while driver-owned session/runtime directories are always removed. The default removes the marked artifact root after a passing run. + +## Prompt size + +The same built-in-only profile map and the same counting method were used before and after. “Native” is the concatenated tool snippets/guidelines; “appended” is the `before_agent_start` pi-flow section. + +| Prompt contribution | Before | After | +| --- | ---: | ---: | +| Tool-native metadata | 5,438 chars | 734 chars | +| Appended contract | 5,224 chars | 3,893 chars | +| Combined | 10,662 chars / 1,587 words | 4,627 chars / 675 words | +| Roster entry occurrences | 2 | 1 | + +The combined contract shrank 56.6% by characters and 57.5% by words. The detailed appended contract remains because custom Pi system prompts omit normal tool-native snippets and guidelines. + +## Real-model comparison + +Setup for both sides: + +- Root model: `deepseek/deepseek-v4-flash` +- Thinking: `high` +- Two repetitions per scenario +- Active tools: `read`, `bash`, `Agent`, `workflow` +- Ephemeral root session and isolated agent directory +- Same generated fixture, prompts, driver, process timeout, and checks +- Baseline extension: archived `8954ab8` +- Modified extension: this change + +### Routing outcomes + +| Scenario | Baseline | Modified | Accepted result | +| --- | --- | --- | --- | +| Narrow package lookup | root direct in 2/2 | root direct in 2/2 | yes | +| Focused repository map | root direct in 2/2 | root direct in 2/2 | yes; focused delegation is deliberately soft | +| Small flat three-lane review | three parallel Agent calls in 2/2 | three parallel Agent calls in 2/2 | yes; all calls fresh and completed | +| Same-child follow-up | two sequential Agent calls sharing one non-empty key in 2/2 | same in 2/2 | yes | +| Structured classify-then-follow-up | a pipeline workflow with at least two agent expressions and six completed children in 2/2 | same in 2/2 | yes | + +All hard checks passed on both sides. The focused-map delegation observation was inconclusive in all four runs because the root handled the tiny fixture directly without workflow. Under the committed driver, the modified run's duplicate successful staged execution is also reported as inconclusive rather than hidden or treated as deterministic proof. + +### Usage and cost + +Usage sums root assistant usage plus nested usage from `Agent` and `workflow` tool results. Tokens include cache reads/writes as reported by Pi; costs are provider-reported. + +| Side | Runs | Reported tokens | Reported cost | Summed wall time | +| --- | ---: | ---: | ---: | ---: | +| Baseline | 10 | 622,059 | $0.014483 | 267.3s | +| Modified | 10 | 710,465 | $0.014233 | 256.0s | + +Modified-run tokens were 14.2% higher, while reported cost was 1.7% lower and summed wall time was 4.2% lower. These small mixed differences are not evidence of an execution-cost improvement: child output and root search behavior dominated this stochastic sample despite the much smaller static prompt. + +## Variance and limitations + +- Two repetitions expose obvious route variance but cannot prove deterministic behavior or generalize to other root models. +- The focused scenario is intentionally soft. Direct investigation of this tiny fixture is acceptable. +- Both prompt versions sometimes inspected files in the root before delegating. The modified flat scenario did this in one repetition; prompt wording does not reliably eliminate duplicate search. +- An intermediate modified-prompt trial exposed a staged-authoring regression: one run compressed classification and follow-up into three children. Restoring the `pipeline()` stage argument contract and a concise dependent-stage example produced six-child pipeline workflows in the final 2/2 rerun. +- In the final modified sample, one staged run retried an invalid workflow before succeeding, while the other ran two successful six-child workflows despite guidance not to repeat completed branches. The driver reports duplicate successful execution as inconclusive; it accepts the route only when the terminal script contains a schema-bearing pipeline with at least two agent expressions and completes at least six children. +- The driver tests entry-point and continuation behavior only. It neither selects nor grades profile names, child models, or specialist roles. +- Saved-name and replay behavior remain covered by `npm run e2e:workflow-features`; this driver focuses on root prompt routing. +- After artifact-safety hardening, the final driver was rerun once each for flat fan-out, continuation, and staged orchestration: all three passed with 294,296 reported tokens, $0.007482 reported cost, and 107.2s summed wall time. + +## Deterministic validation + +The committed suite separately covers: + +- active Agent/workflow prompt sections for both, either, and neither tool; +- detailed contract retention with a custom Pi base system prompt; +- one dynamic profile roster occurrence; +- same-stream continuation and fresh parallel Agent execution; +- dynamic-schema and saved-workflow filename guidance matching runtime behavior. + +Run all required checks with: + +```bash +npm run check +``` diff --git a/scripts/e2e/prompt-routing.mjs b/scripts/e2e/prompt-routing.mjs new file mode 100644 index 0000000..5cf899c --- /dev/null +++ b/scripts/e2e/prompt-routing.mjs @@ -0,0 +1,673 @@ +#!/usr/bin/env node + +import { spawn } from "node:child_process"; +import { + existsSync, + lstatSync, + mkdirSync, + readFileSync, + readdirSync, + realpathSync, + rmSync, + statSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import process from "node:process"; +import { fileURLToPath } from "node:url"; +import { + DEEPSEEK_ANTHROPIC_BASE_URL, + deepseekCredentialEnvNames, + loadDotEnv, + prepareDeepseekClaudeE2EEnv, + resolveDeepseekApiKey, +} from "./lib/deepseek-claude-env.mjs"; + +const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", ".."); +const extensionPath = path.join(repoRoot, "index.ts"); +const MAX_CAPTURE_CHARS = 16 * 1024 * 1024; +const MAX_STDOUT_LINE_CHARS = 8 * 1024 * 1024; +const SCENARIO_KEYS = ["direct", "focused", "flat", "continuation", "staged"]; +const RUN_ROOT_MARKER = ".pi-flow-prompt-routing-owned"; +const SAFE_ENV_NAMES = [ + "PATH", + "HOME", + "USER", + "LOGNAME", + "SHELL", + "TMPDIR", + "TMP", + "TEMP", + "LANG", + "LC_ALL", + "LC_CTYPE", + "TERM", + "COLORTERM", + "CI", + "NO_COLOR", + "FORCE_COLOR", + "SSL_CERT_FILE", + "NODE_EXTRA_CA_CERTS", +]; + +loadDotEnv(path.join(repoRoot, ".env")); + +function parseArgs(argv) { + const options = { + model: undefined, + thinking: undefined, + repetitions: 2, + timeoutMs: 300_000, + deepseekApiKeyEnv: "DEEPSEEK_API_KEY", + runRoot: path.join(tmpdir(), `pi-flow-prompt-routing-${Date.now()}`), + agentDir: undefined, + piCommand: process.env.PI_E2E_COMMAND || "pi", + extension: extensionPath, + only: undefined, + keep: false, + help: false, + }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + const value = () => { + const next = argv[index + 1]; + if (next === undefined) throw new Error(`${arg} requires a value`); + index += 1; + return next; + }; + if (arg === "--model") options.model = value(); + else if (arg === "--thinking") options.thinking = value(); + else if (arg === "--repetitions") options.repetitions = Number(value()); + else if (arg === "--timeout-ms") options.timeoutMs = Number(value()); + else if (arg === "--deepseek-api-key-env") options.deepseekApiKeyEnv = value(); + else if (arg === "--run-root") options.runRoot = path.resolve(value()); + else if (arg === "--agent-dir") options.agentDir = path.resolve(value()); + else if (arg === "--pi-command") options.piCommand = value(); + else if (arg === "--extension") options.extension = path.resolve(value()); + else if (arg === "--only") options.only = value().split(",").map((item) => item.trim()).filter(Boolean); + else if (arg === "--keep") options.keep = true; + else if (arg === "--help" || arg === "-h") options.help = true; + else throw new Error(`Unknown option: ${arg}`); + } + if (!options.help && (!options.model || !options.thinking)) { + throw new Error("--model and --thinking are required; this E2E intentionally has no encoded model defaults"); + } + if (!Number.isInteger(options.repetitions) || options.repetitions < 1 || options.repetitions > 5) { + throw new Error("--repetitions must be an integer from 1 to 5"); + } + if (!Number.isFinite(options.timeoutMs) || options.timeoutMs <= 0) { + throw new Error("--timeout-ms must be positive"); + } + const unknown = (options.only ?? []).filter((key) => !SCENARIO_KEYS.includes(key)); + if (unknown.length > 0) throw new Error(`Unknown scenario(s): ${unknown.join(", ")}`); + options.ownsAgentDir = options.agentDir === undefined; + options.agentDir ??= path.join(options.runRoot, "agent"); + return options; +} + +function printHelp() { + console.log(`Usage: node scripts/e2e/prompt-routing.mjs --model --thinking [options] + +Runs privacy-safe real-Pi routing scenarios against the current extension prompt. +No profile name, child model, or specialist choice is asserted. + +Options: + --model required root model + --thinking required root thinking level + --repetitions <1-5> repetitions per scenario (default: 2) + --only direct, focused, flat, continuation, staged + --timeout-ms timeout per root run (default: 300000) + --deepseek-api-key-env preferred DeepSeek credential variable + --run-root artifact root + --agent-dir isolated Pi agent directory (default: under run root) + --pi-command Pi executable (default: PI_E2E_COMMAND or pi) + --extension extension entry to evaluate (default: current worktree) + --keep keep artifacts after a passing run +`); +} + +function ensureDir(directory) { + mkdirSync(directory, { recursive: true }); +} + +function canonicalPotentialPath(target) { + let existing = target; + const missing = []; + while (!existsSync(existing)) { + const parent = path.dirname(existing); + if (parent === existing) break; + missing.unshift(path.basename(existing)); + existing = parent; + } + return path.join(realpathSync(existing), ...missing); +} + +function isInside(parent, candidate) { + const relativePath = path.relative(parent, candidate); + return relativePath === "" || (!relativePath.startsWith("..") && !path.isAbsolute(relativePath)); +} + +function assertAgentDirOwnership(options) { + if (options.ownsAgentDir) return; + const runRoot = canonicalPotentialPath(options.runRoot); + const agentDir = canonicalPotentialPath(options.agentDir); + if (isInside(runRoot, agentDir)) { + throw new Error("an explicit --agent-dir must be outside --run-root; omit it to use the isolated default"); + } +} + +function findPackageRoot(entry) { + let directory = path.dirname(realpathSync(entry)); + while (true) { + if (existsSync(path.join(directory, "package.json"))) return directory; + const parent = path.dirname(directory); + if (parent === directory) return path.dirname(realpathSync(entry)); + directory = parent; + } +} + +function prepareRunRoot(directory) { + if (existsSync(directory) && (lstatSync(directory).isSymbolicLink() || readdirSync(directory).length > 0)) { + throw new Error("--run-root must not exist or must be an empty real directory"); + } + ensureDir(directory); + writeFileSync(path.join(directory, RUN_ROOT_MARKER), "pi-flow prompt-routing E2E\n", "utf8"); +} + +function removeOwnedRunRoot(directory) { + if (!existsSync(path.join(directory, RUN_ROOT_MARKER))) { + throw new Error("refusing to remove an artifact root without the pi-flow ownership marker"); + } + rmSync(directory, { recursive: true, force: true }); +} + +function minimalE2EEnvironment(baseEnv, credential) { + const environment = {}; + for (const name of SAFE_ENV_NAMES) { + if (typeof baseEnv[name] === "string") environment[name] = baseEnv[name]; + } + environment.DEEPSEEK_API_KEY = credential; + return environment; +} + +function redactSecrets(text, secrets) { + let redacted = text; + for (const secret of secrets) { + if (secret) redacted = redacted.replaceAll(secret, "[REDACTED]"); + } + return redacted; +} + +function createFixture(root) { + const fixture = path.join(root, "fixture"); + ensureDir(path.join(fixture, "src")); + ensureDir(path.join(fixture, "test")); + const files = { + "package.json": `${JSON.stringify({ name: "routing-fixture", type: "module", scripts: { test: "node --test" } }, null, 2)}\n`, + "README.md": "# Routing fixture\n\nA tiny command-line report formatter.\n", + "src/cli.js": "import { formatReport } from './report.js';\nprocess.stdout.write(formatReport(process.argv[2] ?? 'sample'));\n", + "src/report.js": "export function formatReport(name) { return `Report: ${name}\\n`; }\n", + "test/report.test.js": "import test from 'node:test';\nimport assert from 'node:assert/strict';\nimport { formatReport } from '../src/report.js';\ntest('formats', () => assert.equal(formatReport('a'), 'Report: a\\n'));\n", + }; + for (const [relativePath, content] of Object.entries(files)) { + writeFileSync(path.join(fixture, relativePath), content, "utf8"); + } + return fixture; +} + +function snapshotFiles(root) { + const snapshot = new Map(); + const walk = (directory) => { + for (const entry of readdirSync(directory)) { + const filePath = path.join(directory, entry); + if (statSync(filePath).isDirectory()) walk(filePath); + else snapshot.set(path.relative(root, filePath), readFileSync(filePath, "utf8")); + } + }; + walk(root); + return snapshot; +} + +function changedFiles(before, after) { + return [...new Set([...before.keys(), ...after.keys()])] + .filter((key) => before.get(key) !== after.get(key)) + .sort(); +} + +function addUsage(total, usage) { + for (const field of ["input", "output", "cacheRead", "cacheWrite", "totalTokens"]) { + if (typeof usage?.[field] === "number") total[field] = (total[field] ?? 0) + usage[field]; + } + if (typeof usage?.cost?.total === "number") total.cost = (total.cost ?? 0) + usage.cost.total; +} + +function analyzeJsonl(text) { + const analysis = { + malformedLines: 0, + toolCounts: {}, + agentCalls: [], + workflowCalls: [], + agentResults: [], + workflowResults: [], + rootUsage: {}, + childUsage: {}, + finalStop: false, + }; + let group = 0; + for (const line of text.split("\n")) { + if (!line.trim()) continue; + let event; + try { + event = JSON.parse(line); + } catch { + analysis.malformedLines += 1; + continue; + } + if (event.type === "tool_execution_end") { + if (event.toolName === "Agent" || event.toolName === "workflow") { + addUsage(analysis.childUsage, event.result?.usage); + } + continue; + } + if (event.type !== "message_end") continue; + const message = event.message ?? {}; + if (message.role === "assistant") { + addUsage(analysis.rootUsage, message.usage); + const calls = (message.content ?? []).filter((item) => item?.type === "toolCall"); + if (calls.length > 0) group += 1; + for (const call of calls) { + analysis.toolCounts[call.name] = (analysis.toolCounts[call.name] ?? 0) + 1; + const input = call.arguments ?? {}; + if (call.name === "Agent") { + analysis.agentCalls.push({ group, sessionKey: typeof input.session_key === "string" ? input.session_key : "" }); + } else if (call.name === "workflow") { + const script = typeof input.script === "string" ? input.script : ""; + analysis.workflowCalls.push({ + group, + source: script ? "inline" : input.name ? "saved" : "path", + pipeline: script.includes("pipeline("), + parallel: script.includes("parallel("), + schema: /\bschema\s*:/.test(script), + agentExpressions: (script.match(/\bagent\s*\(/g) ?? []).length, + }); + } + } + if (message.stopReason === "stop") analysis.finalStop = true; + } else if (message.role === "toolResult") { + if (message.toolName !== "Agent" && message.toolName !== "workflow") continue; + const details = message.details ?? {}; + if (message.toolName === "Agent") { + analysis.agentResults.push({ status: details.status }); + } else { + analysis.workflowResults.push({ + status: details.status, + agentCount: details.agentCount, + childStatuses: Array.isArray(details.agents) ? details.agents.map((agent) => agent.status) : [], + }); + } + } + } + return analysis; +} + +function check(label, ok, info = "", soft = false) { + return { label, status: ok ? "PASS" : soft ? "INCONCLUSIVE" : "FAIL", info }; +} + +function commonChecks(run, analysis) { + return [ + check("Pi process completed", run.code === 0 && !run.timedOut, `code=${run.code} timedOut=${run.timedOut}`), + check("output stayed within capture bounds", run.captureError === undefined, run.captureError ?? ""), + check("JSON event stream parsed", analysis.malformedLines === 0, `malformed=${analysis.malformedLines}`), + check("root produced a final response", analysis.finalStop), + check("fixture stayed unchanged", run.changed.length === 0, run.changed.join(",")), + ]; +} + +function validateDirect(run, analysis) { + const delegated = analysis.agentCalls.length + analysis.workflowCalls.length; + const inspected = (analysis.toolCounts.read ?? 0) + (analysis.toolCounts.bash ?? 0); + return [ + ...commonChecks(run, analysis), + check("narrow lookup stayed in the root", delegated === 0, `Agent=${analysis.agentCalls.length} workflow=${analysis.workflowCalls.length}`), + check("root inspected the fixture", inspected > 0, `readOrBash=${inspected}`), + ]; +} + +function validateFocused(run, analysis) { + const routeAllowed = analysis.workflowCalls.length === 0 && analysis.agentCalls.length <= 1; + const didWork = analysis.agentCalls.length === 1 || (analysis.toolCounts.read ?? 0) + (analysis.toolCounts.bash ?? 0) > 0; + return [ + ...commonChecks(run, analysis), + check("focused map used root or one Agent, never workflow", routeAllowed, `Agent=${analysis.agentCalls.length} workflow=${analysis.workflowCalls.length}`), + check("focused map performed an investigation", didWork), + check("focused map delegated once", analysis.agentCalls.length === 1, `Agent=${analysis.agentCalls.length}`, true), + ]; +} + +function validateFlat(run, analysis) { + const callsByGroup = new Map(); + for (const call of analysis.agentCalls) callsByGroup.set(call.group, (callsByGroup.get(call.group) ?? 0) + 1); + const maxParallelGroup = Math.max(0, ...callsByGroup.values()); + const fresh = analysis.agentCalls.every((call) => call.sessionKey === ""); + const done = analysis.agentResults.length === analysis.agentCalls.length && analysis.agentResults.every((result) => result.status === "done"); + return [ + ...commonChecks(run, analysis), + check("small flat fan-out used direct Agent calls", analysis.agentCalls.length >= 2 && analysis.workflowCalls.length === 0, `Agent=${analysis.agentCalls.length} workflow=${analysis.workflowCalls.length}`), + check("independent Agent calls were issued together", maxParallelGroup >= 2, `maxGroup=${maxParallelGroup}`), + check("independent work stayed fresh", fresh), + check("all direct Agent calls completed", done), + ]; +} + +function validateContinuation(run, analysis) { + const keys = analysis.agentCalls.map((call) => call.sessionKey); + const groups = new Set(analysis.agentCalls.map((call) => call.group)); + const sameNonEmptyKey = keys.length === 2 && keys[0].length > 0 && keys[0] === keys[1]; + const done = analysis.agentResults.length === 2 && analysis.agentResults.every((result) => result.status === "done"); + return [ + ...commonChecks(run, analysis), + check("continuation used exactly two Agent calls", analysis.agentCalls.length === 2 && analysis.workflowCalls.length === 0, `Agent=${analysis.agentCalls.length} workflow=${analysis.workflowCalls.length}`), + check("same logical child reused one non-empty session key", sameNonEmptyKey), + check("continuation calls were sequential", groups.size === 2, `groups=${groups.size}`), + check("both continuation calls completed", done), + ]; +} + +function validateStaged(run, analysis) { + const workflow = analysis.workflowCalls.at(-1); + const result = analysis.workflowResults.at(-1); + const completedResults = analysis.workflowResults.filter((item) => item.status === "completed"); + const childrenDone = result?.childStatuses.length >= 6 && result.childStatuses.every((status) => status === "done"); + return [ + ...commonChecks(run, analysis), + check("staged structured task used workflow", analysis.workflowCalls.length >= 1 && analysis.agentCalls.length === 0, `workflow=${analysis.workflowCalls.length} Agent=${analysis.agentCalls.length}`), + check("successful workflow encoded dependent pipeline stages and schemas", Boolean(workflow?.pipeline && workflow?.schema && workflow?.agentExpressions >= 2), JSON.stringify(workflow ?? {})), + check("workflow completed at least six child calls", result?.status === "completed" && result?.agentCount >= 6, JSON.stringify(result ?? {})), + check("all staged children completed", childrenDone), + check("workflow avoided duplicate successful execution", completedResults.length === 1, `completed=${completedResults.length}`, true), + ]; +} + +const SCENARIOS = [ + { + key: "direct", + prompt: "Read package.json directly with the root file tools and report only the package name. This is a narrow local lookup; do not delegate it and do not modify files.", + validate: validateDirect, + }, + { + key: "focused", + prompt: "Map this repository's purpose, key files, runtime flow, and tests. Keep the investigation read-only and give a concise evidence-based summary.", + validate: validateFocused, + }, + { + key: "flat", + prompt: "Review this repository across three independent dimensions: source correctness, tests and coverage, and documentation and package configuration. Keep this as a small flat fan-out, make no file changes, and synthesize a concise final assessment.", + validate: validateFlat, + }, + { + key: "continuation", + prompt: "Use one subagent in two sequential turns. First ask it to read src/cli.js and src/report.js, remember the exported function and command-line argument flow, and reply with STEP1_DONE. Then ask that same continuing child, without re-reading files and without copying its first result into the second prompt, to recall the function and flow. Use pi-flow's child-conversation continuation feature, report the second result, and do not modify files.", + validate: validateContinuation, + }, + { + key: "staged", + prompt: "For each of package.json, src/cli.js, and test/report.test.js, first classify it as config, entry, or test using a strict machine-readable result. Then dispatch a classification-specific follow-up that states its role in one sentence. Different files may proceed concurrently, but each file's classify step must precede its follow-up. Return one object containing all classifications and follow-ups. Keep this read-only.", + validate: validateStaged, + }, +]; + +function runPi({ options, fixture, scenario, repetition, environment, redactions }) { + const runDir = path.join(options.runRoot, "runs", `${scenario.key}-${repetition}`); + ensureDir(runDir); + const args = [ + "-p", + "--mode", "json", + "--model", options.model, + "--thinking", options.thinking, + "--no-session", + "--no-extensions", + "--extension", options.extension, + "--no-skills", + "--no-prompt-templates", + "--no-themes", + "--no-context-files", + "--tools", "read,bash,Agent,workflow", + "--approve", + scenario.prompt, + ]; + return new Promise((resolve, reject) => { + const before = snapshotFiles(fixture); + const startedAt = Date.now(); + const child = spawn(options.piCommand, args, { + cwd: fixture, + env: { + ...environment, + PI_CODING_AGENT_DIR: options.agentDir, + PI_SKIP_VERSION_CHECK: "1", + }, + stdio: ["ignore", "pipe", "pipe"], + }); + let stdout = ""; + let stdoutPending = ""; + let stderr = ""; + let timedOut = false; + let captureError; + let killTimer; + const terminate = () => { + child.kill("SIGTERM"); + if (!killTimer) { + killTimer = setTimeout(() => child.kill("SIGKILL"), 3000); + killTimer.unref(); + } + }; + const append = (current, chunk, stream) => { + const next = current + String(chunk); + if (next.length > MAX_CAPTURE_CHARS && !captureError) { + captureError = `${stream} exceeded ${MAX_CAPTURE_CHARS} retained characters`; + terminate(); + } + return next.slice(0, MAX_CAPTURE_CHARS); + }; + const retainStdoutLine = (line) => { + if (!line.trim()) return; + if (line.length > MAX_STDOUT_LINE_CHARS) { + captureError ??= `stdout line exceeded ${MAX_STDOUT_LINE_CHARS} characters`; + terminate(); + return; + } + try { + const event = JSON.parse(line); + if (event.type === "message_end" || event.type === "tool_execution_end") { + stdout = append(stdout, `${line}\n`, "stdout"); + } + } catch { + stdout = append(stdout, `${line}\n`, "stdout"); + } + }; + const consumeStdout = (chunk) => { + stdoutPending += String(chunk); + let newline = stdoutPending.indexOf("\n"); + while (newline !== -1) { + retainStdoutLine(stdoutPending.slice(0, newline)); + stdoutPending = stdoutPending.slice(newline + 1); + newline = stdoutPending.indexOf("\n"); + } + if (stdoutPending.length > MAX_STDOUT_LINE_CHARS) { + captureError ??= `stdout line exceeded ${MAX_STDOUT_LINE_CHARS} characters`; + stdoutPending = ""; + terminate(); + } + }; + const timer = setTimeout(() => { + timedOut = true; + terminate(); + }, options.timeoutMs); + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + child.stdout.on("data", consumeStdout); + child.stderr.on("data", (chunk) => { stderr = append(stderr, chunk, "stderr"); }); + child.once("error", (error) => { + stderr = append(stderr, `\nspawn error: ${error.message}`, "stderr"); + }); + child.once("close", (code, signal) => { + clearTimeout(timer); + if (killTimer) clearTimeout(killTimer); + try { + if (stdoutPending && !captureError) retainStdoutLine(stdoutPending); + const after = snapshotFiles(fixture); + stdout = redactSecrets(stdout, redactions); + stderr = redactSecrets(stderr, redactions); + writeFileSync(path.join(runDir, "stdout.jsonl"), stdout, "utf8"); + writeFileSync(path.join(runDir, "stderr.log"), stderr, "utf8"); + resolve({ + code, + signal, + timedOut, + captureError, + stdout, + stderr, + durationMs: Date.now() - startedAt, + changed: changedFiles(before, after), + }); + } catch (error) { + reject(error); + } + }); + }); +} + +function summarizedAnalysis(analysis) { + const keys = analysis.agentCalls.map((call) => call.sessionKey).filter(Boolean); + return { + toolCounts: analysis.toolCounts, + agentCallCount: analysis.agentCalls.length, + agentCallGroups: analysis.agentCalls.map((call) => call.group), + sessionKeyState: keys.length === 0 ? "none" : new Set(keys).size === 1 ? "same" : "different", + workflowCalls: analysis.workflowCalls, + agentResults: analysis.agentResults, + workflowResults: analysis.workflowResults, + rootUsage: analysis.rootUsage, + childUsage: analysis.childUsage, + }; +} + +function printRun(result) { + const failed = result.checks.filter((item) => item.status === "FAIL").length; + const inconclusive = result.checks.filter((item) => item.status === "INCONCLUSIVE").length; + const status = failed > 0 ? "FAIL" : inconclusive > 0 ? "INCONCLUSIVE" : "PASS"; + console.log(`\n[${status}] ${result.scenario} repetition ${result.repetition} (${(result.durationMs / 1000).toFixed(1)}s)`); + for (const item of result.checks) { + const marker = item.status === "PASS" ? "✓" : item.status === "FAIL" ? "✗" : "•"; + console.log(` ${marker} ${item.label}${item.info ? ` (${item.info})` : ""}`); + } + console.log(` tools=${JSON.stringify(result.analysis.toolCounts)} usage=${JSON.stringify({ root: result.analysis.rootUsage, child: result.analysis.childUsage })}`); + return { failed, inconclusive }; +} + +async function main() { + const options = parseArgs(process.argv.slice(2)); + if (options.help) { + printHelp(); + return; + } + + const credential = resolveDeepseekApiKey(process.env, options.deepseekApiKeyEnv); + if (!credential) { + throw new Error(`missing DeepSeek credential in ${deepseekCredentialEnvNames(options.deepseekApiKeyEnv).join(", ")}`); + } + + let runRootPrepared = false; + let completedSuccessfully = false; + try { + prepareRunRoot(options.runRoot); + runRootPrepared = true; + assertAgentDirOwnership(options); + ensureDir(options.agentDir); + const fixture = createFixture(options.runRoot); + const environment = prepareDeepseekClaudeE2EEnv( + minimalE2EEnvironment(process.env, credential), + { + apiKeyEnv: "DEEPSEEK_API_KEY", + runtimeDir: path.join(options.runRoot, "claude-runtime"), + }, + ); + const credentialValues = deepseekCredentialEnvNames(options.deepseekApiKeyEnv) + .map((name) => process.env[name]) + .filter((value) => typeof value === "string" && value.length > 0); + const redactions = [...new Set([ + credential, + ...credentialValues, + options.runRoot, + canonicalPotentialPath(options.runRoot), + options.agentDir, + canonicalPotentialPath(options.agentDir), + options.extension, + realpathSync(options.extension), + findPackageRoot(options.extension), + repoRoot, + process.env.HOME, + ].filter((value) => typeof value === "string" && value.length > 0))]; + const selected = SCENARIOS.filter((scenario) => !options.only || options.only.includes(scenario.key)); + console.log("pi-flow prompt-routing E2E"); + console.log(` model: ${options.model}`); + console.log(` thinking: ${options.thinking}`); + console.log(` repetitions: ${options.repetitions}`); + console.log(` Claude Code provider guard: DeepSeek (${DEEPSEEK_ANTHROPIC_BASE_URL})`); + + const results = []; + for (const scenario of selected) { + for (let repetition = 1; repetition <= options.repetitions; repetition += 1) { + const run = await runPi({ + options, + fixture, + scenario, + repetition, + environment, + redactions, + }); + const analysis = analyzeJsonl(run.stdout); + const checks = scenario.validate(run, analysis); + const result = { + scenario: scenario.key, + repetition, + durationMs: run.durationMs, + checks, + analysis: summarizedAnalysis(analysis), + }; + results.push(result); + printRun(result); + } + } + const failedChecks = results.flatMap((result) => result.checks).filter((item) => item.status === "FAIL").length; + const inconclusiveChecks = results.flatMap((result) => result.checks).filter((item) => item.status === "INCONCLUSIVE").length; + const report = { + model: options.model, + thinking: options.thinking, + repetitions: options.repetitions, + scenarios: selected.map((scenario) => scenario.key), + failedChecks, + inconclusiveChecks, + results, + }; + writeFileSync(path.join(options.runRoot, "report.json"), `${JSON.stringify(report, null, 2)}\n`, "utf8"); + console.log(`\nSummary: ${results.length} run(s), ${failedChecks} failed check(s), ${inconclusiveChecks} inconclusive check(s).`); + if (failedChecks > 0) throw new Error("prompt-routing E2E failed"); + completedSuccessfully = true; + } finally { + if (runRootPrepared) { + rmSync(path.join(options.runRoot, "claude-runtime"), { recursive: true, force: true }); + if (options.ownsAgentDir) { + rmSync(options.agentDir, { recursive: true, force: true }); + } + if (completedSuccessfully && !options.keep) removeOwnedRunRoot(options.runRoot); + else if (existsSync(options.runRoot)) console.log("Artifacts kept under the requested run root."); + } + } +} + +main().catch((error) => { + console.error(`FAIL prompt-routing E2E: ${error instanceof Error ? error.message : String(error)}`); + process.exitCode = 1; +}); diff --git a/src/pi-subagent.ts b/src/pi-subagent.ts index e0feced..915b5a1 100644 --- a/src/pi-subagent.ts +++ b/src/pi-subagent.ts @@ -12,8 +12,7 @@ import { Type, type Static } from "typebox"; import { AGENT_PROMPT_GUIDELINES, AGENT_PROMPT_SNIPPET, - buildCoordinatorPrompt, - buildWorkflowPrompt, + buildFlowPrompt, } from "./prompts.ts"; import { getSubagentProfiles } from "./profiles.ts"; import { ConcurrencyLimiter } from "./core/concurrency.ts"; @@ -67,12 +66,12 @@ const agentToolParameters = Type.Object({ }), subagent_type: Type.Optional( Type.String({ - description: "The subagent profile to use. Defaults to general-purpose. Custom profiles are loaded from ~/.pi/agent/subagents/.md.", + description: "The subagent profile to use. Available profiles are loaded from the agent configuration directory.", }), ), session_key: Type.Optional( Type.String({ - description: "Caller-chosen key for a resumable subagent conversation. Omit for a fresh one-shot subagent; reuse the same key to continue that child context.", + description: "Caller-chosen key for one resumable child stream. Reuse it only to continue the same logical child stream; omit it for fresh or independent work.", }), ), }); @@ -374,7 +373,7 @@ function createAgentTool( return defineTool({ name: "Agent", label: "Agent", - description: "Launch a subagent. Omit session_key for a fresh one-shot context; pass a caller-chosen session_key to create or continue a resumable subagent conversation. Available agents include built-ins and custom profiles from ~/.pi/agent/subagents/*.md.", + description: "Launch one focused subagent task. Omit session_key for a fresh child; reuse the same caller-chosen key only to continue one logical child stream. subagent_type selects an available profile.", promptSnippet: AGENT_PROMPT_SNIPPET, promptGuidelines: AGENT_PROMPT_GUIDELINES, parameters: agentToolParameters, @@ -688,24 +687,27 @@ export function createSubagentExtension(options: SubagentExtensionOptions = {}): }); pi.on("before_agent_start", (event, ctx) => { - const tools = pi.getAllTools(); - if (!tools.some((tool) => tool.name === "Agent")) { + const activeTools = new Set(pi.getActiveTools()); + const agentActive = activeTools.has("Agent"); + const workflowActive = workflowEnabled && activeTools.has("workflow"); + if (!agentActive && !workflowActive) { return; } - // No per-turn counter reset: the shared ConcurrencyLimiter is acquired - // immediately before a child spawn and released in the matching finally, - // so the in-flight count stays accurate across turns without a reset. + const profiles = filterProfilesForModelRegistry(getSubagentProfiles(getAgentDir()), ctx.modelRegistry); - const sections = [event.systemPrompt, buildCoordinatorPrompt(profiles)]; - if (workflowEnabled && tools.some((tool) => tool.name === "workflow")) { - const savedWorkflows = listSavedWorkflows({ - agentDir: getAgentDir(), - cwd: ctx.cwd, - projectTrusted: isProjectTrusted(ctx), - }); - sections.push(buildWorkflowPrompt(profiles, savedWorkflows)); - } - return { systemPrompt: sections.join("\n\n") }; + const savedWorkflows = workflowActive + ? listSavedWorkflows({ + agentDir: getAgentDir(), + cwd: ctx.cwd, + projectTrusted: isProjectTrusted(ctx), + }) + : []; + return { + systemPrompt: [ + event.systemPrompt, + buildFlowPrompt(profiles, { agentActive, workflowActive, savedWorkflows }), + ].join("\n\n"), + }; }); }; } diff --git a/src/prompts.ts b/src/prompts.ts index 9f97c1e..f65b4e8 100644 --- a/src/prompts.ts +++ b/src/prompts.ts @@ -1,42 +1,35 @@ import type { SavedWorkflow } from "./workflow/registry.ts"; import type { SubagentProfile } from "./types.ts"; +export const DIRECT_WORK_POLICY = "Narrow, local work can stay in the root when delegation adds no value."; +export const AGENT_USE_POLICY = + "Use Agent for one focused delegated task or a small flat fan-out of independent work."; +export const WORKFLOW_USE_POLICY = + "Use workflow for a saved workflow, dependent stages or control flow, structured results or branching, replay, or larger fan-out."; + export const AGENT_PROMPT_SNIPPET = - "Launch a subagent when the task matches an available agent, can run independently, or would read across several files; pass session_key to create or continue a resumable subagent."; + "Delegate one focused task or a small flat fan-out; session_key continues one logical child stream."; export const AGENT_PROMPT_GUIDELINES = [ - "Reach for Agent when the task matches an available agent, when you have independent work to run in parallel, or when answering would mean reading across several files.", - "Use a specialized custom agent when its description matches the task.", - "Use general-purpose for repository reconnaissance, complex questions, broader multi-step investigations, or independent second opinions when no custom agent is a better match.", - "For a single-fact lookup where you already know the file, symbol, or value, search directly instead of spawning a subagent.", - "Once you delegate a search, do not also run the same search yourself; wait for the result and keep the conclusion, not raw file dumps.", - "If the user asks to explore or survey a repo, delegate a concise read-only map before doing detailed follow-up yourself.", - "If the user asks for parallel work, launch multiple Agent calls in the same assistant response.", - "Write self-contained subagent prompts: when session_key is omitted, subagents do not inherit parent conversation, tool results, or reasoning and cannot be resumed.", - "Pi-backed subagents do not receive Agent; external CLI backends use their own tool surface. Coordinate follow-up delegation from the main conversation after a result returns.", - "Clearly tell the subagent whether you expect read-only research or code changes.", - "Use session_key only when you intentionally want to continue that same subagent conversation later. The Agent final message is returned to you as the tool result and is not shown to the user; relay what matters.", + AGENT_USE_POLICY, + "Reuse the same Agent session_key only for the same logical child stream. Omit it for independent work, including parallel branches.", + "Give each fresh Agent call a self-contained task because it does not inherit parent messages, tool results, or reasoning.", ]; export const WORKFLOW_PROMPT_SNIPPET = - "Run a saved or ad-hoc trusted JavaScript workflow that fans subagents out and synthesizes their results, when the user asks for a workflow or multi-agent orchestration."; + "Run a saved or ad-hoc trusted workflow for staged, structured, replayable, or larger orchestration."; export const WORKFLOW_PROMPT_GUIDELINES = [ - "Use workflow only when the user explicitly asks for a workflow, fan-out, or multi-agent orchestration, when a saved workflow matches the user's request, or when a task decomposes into many independent subagent runs that you then synthesize.", - "Prefer `workflow({ name, args })` when an available saved workflow matches the request. Use `workflow({ scriptPath, resumeFromRunId, args })` to rerun or resume an edited persisted script. Use inline `script` only for ad-hoc orchestration.", - "If the user asks to save a reusable workflow, copy or write a `.js` file directly to `~/.pi/agent/workflows/` for global scope or `.pi/workflows/` for project scope. Project workflows are ignored unless the project is trusted. The file must start with `export const meta = { name, description }`; use a filename that exactly matches the workflow name. After saving, invoke it with `workflow({ name, args })`.", - "For inline scripts, pass one raw JavaScript string in the `script` parameter. No Markdown fences, no prose around it. Inline runs in persisted sessions return `scriptPath` and `runId` for later editing/resume; in-memory runs may only return `runId`.", - "The script's first statement must be `export const meta = { name: 'short_name', description: 'non-empty description' }`. meta must be a plain literal.", - "Available globals: agent(prompt, opts), parallel(thunks), pipeline(items, ...stages), phase(title), log(message), args, cwd. Every workflow must call agent() at least once and return a JSON-serializable value (use null if there is no synthesized result). Results are canonicalized to JSON; non-plain objects are rejected.", - "Write plain JavaScript only. Do not use TypeScript syntax, import/require, fs, Date APIs, or Math.random(). Simple Date/Math.random aliases and destructuring are rejected too. Scripts are trusted code; the determinism check is a cooperative lint, not a sandbox.", - "parallel() takes functions, not promises: `await parallel(items.map(item => () => agent('...', { label: '...' })))`. Results come back in input order.", - "pipeline(items, ...stages) runs each item through the stages in order while different items run concurrently; each stage receives (previousValue, originalItem, index). Prefer pipeline() for multi-stage work — there is no barrier between stages. Reach for parallel() only when you genuinely need all results together, e.g. dedup or a zero-count early exit.", - "Give each agent() a unique short `label` and pick a `subagent_type` (defaults to general-purpose) so it uses that profile's configured backend, model, thinking level, prompt, and pi-backend tool allowlist. Pass `session_key` only when you intentionally want to continue a prior subagent conversation.", - "Pass a portable strict JSON Schema as agent()'s `schema` option whenever the script must branch, route, filter, or aggregate on a result: every object must set `additionalProperties: false`, every property must be listed in `required`, and optional values must use a nullable type. Schemas must be static object literals or top-level consts so workflow preflight can validate all of them before any subagent starts. Omit `schema` for prose findings you only read or synthesize.", - "When `session_key` is omitted, subagents are fresh one-shot sessions with no parent context. Pi-backed subagents do not receive Agent/workflow; external CLI backends use their own tool surface. Include all needed context and paths in each fresh agent() prompt.", - "Failed agent()/parallel()/pipeline() branches resolve to null and are logged unless the workflow is aborted; check for nulls before synthesizing.", + WORKFLOW_USE_POLICY, + "Treat workflow scripts as trusted local code, not sandboxed input.", ]; +export interface FlowPromptOptions { + agentActive: boolean; + workflowActive: boolean; + savedWorkflows?: SavedWorkflow[]; +} + function formatAvailableAgents(profiles: Map): string { return [...profiles.values()] .map((profile) => `- ${profile.name}: ${profile.description}`) @@ -59,52 +52,62 @@ function formatSavedWorkflows(workflows: SavedWorkflow[], maxItems = 20): string return `\n\nSaved workflows:\n${lines.join("\n")}`; } -export function buildWorkflowPrompt(profiles: Map, savedWorkflows: SavedWorkflow[] = []): string { - return `# Dynamic Workflows +function buildAgentSection(): string { + return `## Agent + +Consider Agent for one focused task that can run independently or would keep substantial search output out of the root context. For a small flat fan-out, issue independent Agent calls in the same assistant response. -The \`workflow\` tool runs a saved or ad-hoc trusted JavaScript script that orchestrates many subagents and synthesizes their results. Reach for it when the user asks for a workflow or fan-out, when a saved workflow matches the request, or when a task splits into many independent subagent runs. +- Give each call a short description and a self-contained prompt. Select a \`subagent_type\` from the available-agent roster when its description fits. +- Omit \`session_key\` for a fresh one-shot child. Reuse the same caller-chosen key only to continue the same logical child stream; independent work, including parallel branches, stays fresh. +- Once a search is delegated, do not repeat the same search in the root. The child result is returned to you; relay the useful conclusion to the user.`; +} -Tool input: -- Use \`{ name: 'saved-workflow-name', args }\` for a saved workflow listed below. -- Use \`{ scriptPath, args }\` to run a persisted script file. Add \`resumeFromRunId\` to reuse cached agent results from a previous run's unchanged prefix. -- Use \`{ script, args }\` for ad-hoc orchestration. Inline runs in persisted sessions return \`scriptPath\` and \`runId\` for later editing/resume; in-memory runs may only return \`runId\`. Provide exactly one of \`name\`, \`scriptPath\`, or \`script\`. +function buildWorkflowSection(savedWorkflows: SavedWorkflow[]): string { + return `## Workflow -Inline script contract: -- First statement: \`export const meta = { name: 'short_name', description: 'non-empty' }\` (a plain literal; \`phases\` optional). -- Globals: agent(prompt, opts), parallel(thunks), pipeline(items, ...stages), phase(title), log(message), args, cwd. Call agent() at least once and return a JSON-serializable value (use \`null\` if there is no synthesized result). Results are canonicalized to JSON; non-plain objects are rejected. -- Plain JavaScript only; no imports, no Date APIs, no Math.random(). Simple Date/Math.random aliases and destructuring are rejected too. Scripts are trusted code; the determinism check is cooperative lint, not a sandbox. -- parallel() takes thunks: \`await parallel(items.map(i => () => agent('...', { label: '...' })))\`. pipeline(items, ...stages) pipelines each item through stages while items run concurrently — prefer it for multi-stage work (no barrier between stages); use parallel() only when you need all results together. +Use workflow when the request matches a saved workflow or needs dependent stages, control flow, structured results or decisions, replay, or larger fan-out. After a workflow completes, synthesize its result instead of repeating completed branches through another delegation path. -Each agent() spawns a fresh one-shot subagent unless you pass \`session_key\` to create or continue a resumable child conversation. Set \`subagent_type\` to use a profile's backend, model, thinking, prompt, and pi-backend tool allowlist: -${formatAvailableAgents(profiles)} +Source: +- Provide exactly one of \`name\`, \`scriptPath\`, or raw \`script\`. Use \`resumeFromRunId\` with \`scriptPath\` to replay the longest unchanged prefix. +- Reusable \`.js\` files live in the global workflow directory or a trusted project's workflow directory. The filename need not match \`meta.name\`; \`meta.name\` is the saved-workflow identity and must match \`[a-z0-9][a-z0-9_-]*\`. Saved files are parsed before each run and never run on discovery. -agent() options: \`label\` (short unique id), \`phase\` (progress group), \`subagent_type\` (profile above), \`session_key\` (caller-chosen key for a resumable child conversation), and \`schema\` (a portable strict JSON Schema). Pass \`schema\` when the script must branch, route, filter, or aggregate on the result: the subagent is forced to return one validated object and agent() resolves to that object instead of free text. Every object schema must set \`additionalProperties: false\`, list every property in \`required\`, and represent optional values with nullable types. Define schemas as static object literals or top-level consts so preflight can reject invalid schemas before any subagent starts. Omit \`schema\` for prose findings you only synthesize. Example — classify, then dispatch: -\`\`\` -const r = await agent("Classify " + file, { label: "classify", schema: { type: "object", additionalProperties: false, required: ["kind"], properties: { kind: { type: "string", enum: ["entry", "lib", "test"] } } } }); -if (r.kind === "entry") { /* ... */ } -\`\`\` +Script contract: +- Start with the plain literal \`export const meta = { name: 'short_name', description: 'non-empty' }\` (optional \`phases\`), call \`agent()\` at least once, and return a JSON-serializable value. +- Globals are \`agent(prompt, opts)\`, \`parallel(thunks)\`, \`pipeline(items, ...stages)\`, \`phase(title)\`, \`log(message)\`, \`args\`, and \`cwd\`. +- Write plain JavaScript without imports, filesystem APIs, Date APIs, or Math.random(). Scripts are trusted code; the determinism check is not a security sandbox. +- \`parallel()\` takes functions, not promises. Each \`pipeline(items, ...stages)\` stage receives \`(previousValue, originalItem, index)\`; stage order is preserved per item while different items progress concurrently. For dependent per-item work, use separate stages such as \`await pipeline(items, (item) => agent('classify ' + item, classifyOpts), (classification, item) => agent('follow up ' + item + ': ' + classification, followupOpts))\`. -Subagents do not inherit parent context unless you continue them with \`session_key\` — brief fresh agent() prompts fully. Pi-backed subagents do not receive Agent/workflow; external CLI backends use their own tool surface. Subagent fan-out is bounded by the same global concurrency cap as the Agent tool; the workflow queues excess agents and drains them as slots free.${formatSavedWorkflows(savedWorkflows)}`; +Workflow agent calls: +- Options are \`label\`, \`phase\`, \`subagent_type\`, \`session_key\`, and \`schema\`. Give each call a unique short label. Reuse a session key only within the same logical child stream; independent branches omit it. +- A schema forces one validated object. Every object schema sets \`additionalProperties: false\`, lists every property in \`required\`, and represents optional values with nullable types. +- Prefer a schema literal or top-level const: statically visible schemas are preflighted before any child starts. A schema supplied through dynamic options is validated immediately before that child launches, so earlier calls may already have run. +- Failed branches resolve to \`null\` unless the workflow is aborted; handle nulls before synthesis.${formatSavedWorkflows(savedWorkflows)}`; } -export function buildCoordinatorPrompt(profiles: Map): string { - return `# Subagent Delegation +export function buildFlowPrompt(profiles: Map, options: FlowPromptOptions): string { + const routing = [ + DIRECT_WORK_POLICY, + ...(options.agentActive ? [AGENT_USE_POLICY] : []), + ...(options.workflowActive ? [WORKFLOW_USE_POLICY] : []), + ]; + const sections = [ + `# Subagent Delegation Available agents: ${formatAvailableAgents(profiles)} -Use Agent when a specialized agent matches the task, the work can run independently, or delegating would keep large search/read output out of the main context. +Routing boundary: +${routing.map((policy) => `- ${policy}`).join("\n")} -Guidelines: -- Do not use subagents excessively; direct lookup is better when the target file, symbol, or value is already known. -- If the user asks for parallel work, launch independent Agent calls in the same assistant response. -- Subagents start fresh and do not inherit parent messages, tool results, or reasoning unless you pass the same caller-chosen session_key to continue that child conversation. Brief fresh subagents with all needed context. -- Pi-backed subagents do not receive Agent; external CLI backends use their own tool surface. Coordinate follow-up delegation from the main conversation after each result returns. -- Use session_key only when continuation is desired; otherwise omit it for a one-shot child. The Agent final message is returned to you as the tool result. Relay what matters to the user. +Children start with fresh context unless a caller-chosen session key continues the same logical stream. Pi-backed children cannot invoke pi-flow delegation tools; external backends use their own tool surface. All fan-out is bounded and queued.`, + ]; -Example usage: -- User asks "explore this repo": use Agent with subagent_type "general-purpose" and ask it to map the project purpose, key directories, important files, scripts, tests, and caveats without editing files. -- User asks for a second opinion on a risky change: use Agent with subagent_type "general-purpose" and give it enough context to review independently. + if (options.agentActive) { + sections.push(buildAgentSection()); + } + if (options.workflowActive) { + sections.push(buildWorkflowSection(options.savedWorkflows ?? [])); + } -Root-level parallel delegation is bounded by the extension. If the running limit is reached, extra Agent calls queue and drain as slots free.`; + return sections.join("\n\n"); } diff --git a/test/agent-contract.test.ts b/test/agent-contract.test.ts index b299549..2bf3a87 100644 --- a/test/agent-contract.test.ts +++ b/test/agent-contract.test.ts @@ -1,4 +1,4 @@ -import { chmodSync, existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { mkdirSync, writeFileSync } from "node:fs"; import { join } from "node:path"; import { AuthStorage, @@ -7,54 +7,54 @@ import { ModelRegistry, SessionManager, SettingsManager, - type ExtensionContext, } from "@earendil-works/pi-coding-agent"; +import { fauxAssistantMessage, fauxProvider, type Context, type Model } from "@earendil-works/pi-ai"; +import { describe, expect, it } from "vitest"; +import { getSubagentProfiles } from "../src/profiles.ts"; import { - fauxAssistantMessage, - fauxProvider, - fauxToolCall, - type Context, - type Model, - type SimpleStreamOptions, -} from "@earendil-works/pi-ai"; -import { describe, expect, it, vi } from "vitest"; -import { createSubagentExtension } from "../src/pi-subagent.ts"; -import { getSubagentProfiles, loadBuiltinSubagentProfiles } from "../src/profiles.ts"; -import { buildClaudeArgs, claudeUsageToSubagentUsage, extractClaudeCostUsd, extractClaudeError, extractClaudeFinalText, extractClaudeUsage, spawnClaudeSubagent } from "../src/core/claude.ts"; -import { buildCodexArgs, codexUsageToSubagentUsage, estimateCodexCostUsd, extractCodexFinalText, spawnCodexSubagent } from "../src/core/codex.ts"; + AGENT_USE_POLICY, + DIRECT_WORK_POLICY, + WORKFLOW_USE_POLICY, +} from "../src/prompts.ts"; import { installFauxProvider, packageRoot, setupPiSubagentTestHarness } from "./helpers/pi-subagent-harness.ts"; +function occurrenceCount(text: string, value: string): number { + return text.split(value).length - 1; +} + describe("pi-subagent agent contract", () => { - let tempDir = ""; let cwd = ""; let agentDir = ""; - let originalPathEnv: string | undefined; let registrations: Array<{ unregister: () => void }> = []; - const { - trackSession, - disposeSession, - createSession, - delegateOnce, - makeMockTheme, - stripAnsi, - renderToText, - formatTestTokens, - makeExecutionContext, - getToolNames, - } = setupPiSubagentTestHarness((state) => { - tempDir = state.tempDir; + const { trackSession, disposeSession, createSession } = setupPiSubagentTestHarness((state) => { cwd = state.cwd; agentDir = state.agentDir; - originalPathEnv = state.originalPathEnv; registrations = state.registrations; }); - it("registers the Claude-style Agent tool contract", async () => { + + async function captureRootPrompt(activeTools: string[], systemPrompt?: string): Promise { + const { session, registration } = await createSession({ systemPrompt }); + let rootContext: Context | undefined; + session.setActiveToolsByName(activeTools); + registration.setResponses([ + (context) => { + rootContext = context; + return fauxAssistantMessage("noted"); + }, + ]); + + await session.prompt("Just say noted."); + disposeSession(session); + return rootContext?.systemPrompt ?? ""; + } + + it("registers the Claude-style Agent tool contract without profile-specific guidance", async () => { const { session } = await createSession(); const tool = session.getAllTools().find((candidate) => candidate.name === "Agent"); expect(tool).toBeDefined(); - const properties = (tool?.parameters as { properties: Record } | undefined)?.properties; + const properties = (tool?.parameters as { properties: Record } | undefined)?.properties; expect(properties).toHaveProperty("description"); expect(properties).toHaveProperty("prompt"); expect(properties).toHaveProperty("subagent_type"); @@ -65,15 +65,15 @@ describe("pi-subagent agent contract", () => { expect(properties).not.toHaveProperty("thinking"); expect(properties).not.toHaveProperty("timeout"); expect(properties).not.toHaveProperty("subagentTimeoutMs"); - expect(tool?.description).toContain("Available agents"); - expect(tool?.promptGuidelines).toContain( - "Reach for Agent when the task matches an available agent, when you have independent work to run in parallel, or when answering would mean reading across several files.", - ); + expect(properties?.subagent_type.description).toMatch(/available profiles/i); + expect(properties?.subagent_type.description).not.toContain("Defaults to"); + expect(properties?.session_key.description).toContain("same logical child"); + expect(tool?.promptGuidelines).toContain(AGENT_USE_POLICY); disposeSession(session); }); - it("marks description and prompt required, subagent_type/session_key optional, and adds no tag/label fields", async () => { + it("marks description and prompt required while keeping routing fields optional", async () => { const { session } = await createSession(); const tool = session.getAllTools().find((candidate) => candidate.name === "Agent"); @@ -110,64 +110,81 @@ describe("pi-subagent agent contract", () => { expect(extensions.extensions[0]?.flags.has("subagent-timeout-ms")).toBe(true); }); + it("injects one profile roster and the approved routing boundary", async () => { + const prompt = await captureRootPrompt(["Agent", "workflow"]); + const profiles = getSubagentProfiles(agentDir); + + expect(prompt).toContain("# Subagent Delegation"); + expect(prompt).toContain(`- ${DIRECT_WORK_POLICY}`); + expect(prompt).toContain(`- ${AGENT_USE_POLICY}`); + expect(prompt).toContain(`- ${WORKFLOW_USE_POLICY}`); + expect(prompt).toContain("## Agent"); + expect(prompt).toContain("## Workflow"); + expect(prompt).toContain("same logical child stream"); + expect(prompt).toContain("independent work, including parallel branches, stays fresh"); + expect(prompt).toContain("schema supplied through dynamic options is validated immediately before that child launches"); + expect(prompt).toContain("filename need not match `meta.name`"); + expect(prompt).not.toContain("use a filename that exactly matches"); + expect(prompt).not.toContain("Schemas must be static"); + expect(occurrenceCount(prompt, "Available agents:")).toBe(1); + for (const profile of profiles.values()) { + expect(occurrenceCount(prompt, `- ${profile.name}: ${profile.description}`)).toBe(1); + } + }); - it("injects the coordinator prompt into the root agent's system prompt", async () => { - const { session, registration } = await createSession(); - let rootContext: Context | undefined; - - registration.setResponses([ - (context) => { - rootContext = context; - return fauxAssistantMessage("noted"); - }, - ]); - - await session.prompt("Just say noted."); + it("retains detailed guidance with a custom base system prompt", async () => { + const customPrompt = "CUSTOM_SYSTEM_PROMPT_SENTINEL"; + const prompt = await captureRootPrompt(["Agent", "workflow"], customPrompt); - expect(rootContext?.systemPrompt).toContain("Subagent Delegation"); - expect(rootContext?.systemPrompt).toContain("Pi-backed subagents do not receive Agent"); - expect(rootContext?.systemPrompt).toContain("Root-level parallel delegation is bounded"); - expect(rootContext?.systemPrompt).not.toContain("max concurrency 4"); - expect(rootContext?.systemPrompt).toContain("Available agents"); - expect(rootContext?.systemPrompt).toContain("general-purpose: General-purpose agent for researching complex questions"); - expect(rootContext?.systemPrompt).not.toContain("explorer: Fast read-only search agent"); - expect(rootContext?.systemPrompt).toContain("Reach for Agent when the task matches an available agent"); - expect(rootContext?.systemPrompt).toContain('User asks "explore this repo"'); - expect(rootContext?.systemPrompt).toContain('subagent_type "general-purpose"'); - expect(rootContext?.systemPrompt).toContain("single-fact lookup"); - expect(rootContext?.systemPrompt).toContain("Once you delegate a search"); + expect(prompt.startsWith(customPrompt)).toBe(true); + expect(prompt).toContain("# Subagent Delegation"); + expect(prompt).toContain("## Agent"); + expect(prompt).toContain("## Workflow"); + expect(prompt).toContain("schema supplied through dynamic options"); + }); - disposeSession(session); + it("appends detailed guidance only for active pi-flow tools", async () => { + const both = await captureRootPrompt(["Agent", "workflow"]); + expect(both).toContain("## Agent"); + expect(both).toContain("## Workflow"); + + const agentOnly = await captureRootPrompt(["Agent"]); + expect(agentOnly).toContain("## Agent"); + expect(agentOnly).not.toContain("## Workflow"); + expect(agentOnly).toContain(AGENT_USE_POLICY); + expect(agentOnly).not.toContain(WORKFLOW_USE_POLICY); + + const workflowOnly = await captureRootPrompt(["workflow"]); + expect(workflowOnly).not.toContain("## Agent"); + expect(workflowOnly).toContain("## Workflow"); + expect(workflowOnly).not.toContain(AGENT_USE_POLICY); + expect(workflowOnly).toContain(WORKFLOW_USE_POLICY); + + const neither = await captureRootPrompt([]); + expect(neither).not.toContain("# Subagent Delegation"); + expect(neither).not.toContain(AGENT_USE_POLICY); + expect(neither).not.toContain(WORKFLOW_USE_POLICY); }); - it("advertises saved workflows in the root system prompt", async () => { + it("advertises saved workflows only while workflow is active", async () => { + const workflowName = "saved_contract_probe"; + const description = "Summarize generated fixture findings."; mkdirSync(join(agentDir, "workflows"), { recursive: true }); writeFileSync( - join(agentDir, "workflows", "audit.js"), - `export const meta = { name: 'audit-todos', description: 'Find TODOs and summarize debt. Use before cleanup planning.' };\nreturn await agent('audit');`, + join(agentDir, "workflows", "different-file-name.js"), + `export const meta = { name: '${workflowName}', description: '${description}' };\nreturn await agent('summarize');`, ); - const { session, registration } = await createSession(); - let rootContext: Context | undefined; - - registration.setResponses([ - (context) => { - rootContext = context; - return fauxAssistantMessage("noted"); - }, - ]); - - await session.prompt("Can you clean up technical debt?"); + const workflowPrompt = await captureRootPrompt(["workflow"]); + expect(workflowPrompt).toContain("Saved workflows:"); + expect(workflowPrompt).toContain(`- ${workflowName}: ${description}`); - expect(rootContext?.systemPrompt).toContain("Saved workflows"); - expect(rootContext?.systemPrompt).toContain("audit-todos: Find TODOs and summarize debt. Use before cleanup planning."); - expect(rootContext?.systemPrompt).toContain("Use `{ name: 'saved-workflow-name', args }`"); - - disposeSession(session); + const agentPrompt = await captureRootPrompt(["Agent"]); + expect(agentPrompt).not.toContain("Saved workflows:"); + expect(agentPrompt).not.toContain(workflowName); }); - - it("registers the Agent tool when loaded via additionalExtensionPaths", async () => { + it("registers Agent when loaded through additionalExtensionPaths", async () => { const faux = fauxProvider({ models: [{ id: "faux-thinker", name: "Faux Thinker", reasoning: true }], }); @@ -208,12 +225,8 @@ describe("pi-subagent agent contract", () => { const tool = session.getAllTools().find((candidate) => candidate.name === "Agent"); expect(tool).toBeDefined(); - expect((tool?.parameters as { properties: Record }).properties).toHaveProperty( - "subagent_type", - ); - expect((tool?.parameters as { properties: Record }).properties).toHaveProperty( - "session_key", - ); + expect((tool?.parameters as { properties: Record }).properties).toHaveProperty("subagent_type"); + expect((tool?.parameters as { properties: Record }).properties).toHaveProperty("session_key"); disposeSession(session); }); diff --git a/test/delegation-scenarios.test.ts b/test/delegation-scenarios.test.ts new file mode 100644 index 0000000..7201e10 --- /dev/null +++ b/test/delegation-scenarios.test.ts @@ -0,0 +1,94 @@ +import { fauxAssistantMessage, fauxToolCall, type Context } from "@earendil-works/pi-ai"; +import { describe, expect, it } from "vitest"; +import { getSubagentProfiles } from "../src/profiles.ts"; +import { setupPiSubagentTestHarness } from "./helpers/pi-subagent-harness.ts"; + +function assistantToolCalls(messages: Array<{ role?: string; content?: unknown }>) { + return messages + .filter((message) => message.role === "assistant" && Array.isArray(message.content)) + .flatMap((message) => message.content as Array<{ type?: string; name?: string; arguments?: Record }>) + .filter((item) => item.type === "toolCall"); +} + +describe("delegation scenario execution", () => { + let agentDir = ""; + const { disposeSession, createSession } = setupPiSubagentTestHarness((state) => { + agentDir = state.agentDir; + }); + + function availableProfileName(): string { + const name = getSubagentProfiles(agentDir).keys().next().value; + expect(name).toBeTypeOf("string"); + return name as string; + } + + it("allows narrow work to finish in the root without delegation", async () => { + const { session, registration } = await createSession(); + registration.setResponses([fauxAssistantMessage("direct result")]); + + await session.prompt("Answer this narrow local question directly."); + + expect(assistantToolCalls(session.messages)).toEqual([]); + expect(JSON.stringify(session.messages)).toContain("direct result"); + disposeSession(session); + }); + + it("executes a small flat fan-out as fresh parallel Agent calls", async () => { + const { session, registration } = await createSession(); + const subagentType = availableProfileName(); + registration.setResponses([ + fauxAssistantMessage( + [ + fauxToolCall("Agent", { description: "Inspect source", prompt: "Inspect source correctness.", subagent_type: subagentType }), + fauxToolCall("Agent", { description: "Inspect tests", prompt: "Inspect test coverage.", subagent_type: subagentType }), + ], + { stopReason: "toolUse" }, + ), + fauxAssistantMessage("source result"), + fauxAssistantMessage("test result"), + fauxAssistantMessage("combined result"), + ]); + + await session.prompt("Run two independent investigations and synthesize them."); + + const calls = assistantToolCalls(session.messages).filter((call) => call.name === "Agent"); + expect(calls).toHaveLength(2); + expect(calls.every((call) => call.arguments?.session_key === undefined)).toBe(true); + expect(JSON.stringify(session.messages)).toContain("source result"); + expect(JSON.stringify(session.messages)).toContain("test result"); + expect(JSON.stringify(session.messages)).toContain("combined result"); + disposeSession(session); + }); + + it("continues one logical child stream with the same session key", async () => { + const { session, registration } = await createSession(); + let continuedChildContext: Context | undefined; + const sessionKey = "logical-child-stream"; + const subagentType = availableProfileName(); + registration.setResponses([ + fauxAssistantMessage( + [fauxToolCall("Agent", { description: "Initial investigation", prompt: "Inspect and remember the flow.", subagent_type: subagentType, session_key: sessionKey })], + { stopReason: "toolUse" }, + ), + fauxAssistantMessage("remembered flow"), + fauxAssistantMessage( + [fauxToolCall("Agent", { description: "Continue investigation", prompt: "Recall and refine the flow.", subagent_type: subagentType, session_key: sessionKey })], + { stopReason: "toolUse" }, + ), + (context) => { + continuedChildContext = context; + return fauxAssistantMessage("refined flow"); + }, + fauxAssistantMessage("reported refinement"), + ]); + + await session.prompt("Use the same child for a dependent follow-up."); + + const calls = assistantToolCalls(session.messages).filter((call) => call.name === "Agent"); + expect(calls).toHaveLength(2); + expect(calls.map((call) => call.arguments?.session_key)).toEqual([sessionKey, sessionKey]); + expect(JSON.stringify(continuedChildContext?.messages)).toContain("remembered flow"); + expect(JSON.stringify(session.messages)).toContain("reported refinement"); + disposeSession(session); + }); +}); diff --git a/test/helpers/pi-subagent-harness.ts b/test/helpers/pi-subagent-harness.ts index 50991cc..aa7ee86 100644 --- a/test/helpers/pi-subagent-harness.ts +++ b/test/helpers/pi-subagent-harness.ts @@ -38,6 +38,7 @@ type CreateSessionOptions = { models?: FauxModelDef[]; defaultModelId?: string; thinkingLevel?: ThinkingLevel; + systemPrompt?: string; }; export type HarnessState = { @@ -167,6 +168,7 @@ export function setupPiSubagentTestHarness(onSetup?: (state: HarnessState) => vo models: modelDefs = DEFAULT_MODEL_DEFS, defaultModelId, thinkingLevel = "high", + systemPrompt, } = options; const faux = fauxProvider({ models: modelDefs }); const models = modelDefs.map((def) => faux.getModel(def.id) as Model); @@ -194,6 +196,7 @@ export function setupPiSubagentTestHarness(onSetup?: (state: HarnessState) => vo noPromptTemplates: true, noThemes: true, noContextFiles: true, + systemPrompt, }); await resourceLoader.reload(); if (maxConcurrentSubagentsFlag !== undefined) { diff --git a/test/proactive-routing.test.ts b/test/proactive-routing.test.ts deleted file mode 100644 index 7f1350e..0000000 --- a/test/proactive-routing.test.ts +++ /dev/null @@ -1,212 +0,0 @@ -import { chmodSync, existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; -import { join } from "node:path"; -import { - AuthStorage, - createAgentSession, - DefaultResourceLoader, - ModelRegistry, - SessionManager, - SettingsManager, - type ExtensionContext, -} from "@earendil-works/pi-coding-agent"; -import { - fauxAssistantMessage, - fauxToolCall, - type Context, - type Model, - type SimpleStreamOptions, -} from "@earendil-works/pi-ai"; -import { describe, expect, it, vi } from "vitest"; -import { createSubagentExtension } from "../src/pi-subagent.ts"; -import { getSubagentProfiles, loadBuiltinSubagentProfiles } from "../src/profiles.ts"; -import { buildClaudeArgs, claudeUsageToSubagentUsage, extractClaudeCostUsd, extractClaudeError, extractClaudeFinalText, extractClaudeUsage, spawnClaudeSubagent } from "../src/core/claude.ts"; -import { buildCodexArgs, codexUsageToSubagentUsage, estimateCodexCostUsd, extractCodexFinalText, spawnCodexSubagent } from "../src/core/codex.ts"; -import { packageRoot, setupPiSubagentTestHarness } from "./helpers/pi-subagent-harness.ts"; - -describe("pi-subagent proactive routing", () => { - let tempDir = ""; - let cwd = ""; - let agentDir = ""; - let originalPathEnv: string | undefined; - let registrations: Array<{ unregister: () => void }> = []; - - const { - trackSession, - disposeSession, - createSession, - delegateOnce, - makeMockTheme, - stripAnsi, - renderToText, - formatTestTokens, - makeExecutionContext, - getToolNames, - } = setupPiSubagentTestHarness((state) => { - tempDir = state.tempDir; - cwd = state.cwd; - agentDir = state.agentDir; - originalPathEnv = state.originalPathEnv; - registrations = state.registrations; - }); - describe("proactive routing scenarios", () => { - function makeRouter(decide: (userText: string, systemPrompt: string) => unknown[] | string) { - return (context: Context) => { - const userText = context.messages - .filter((message) => (message as { role?: string }).role === "user") - .map((message) => JSON.stringify((message as { content?: unknown }).content)) - .join("\n"); - const systemPrompt = context.systemPrompt ?? ""; - const decision = decide(userText, systemPrompt); - if (typeof decision === "string") { - return fauxAssistantMessage(decision); - } - return fauxAssistantMessage(decision as never, { stopReason: "toolUse" }); - }; - } - - it("scenario 1: multi-repo research → coordinator-aware router fans out two parallel Agent calls", async () => { - const { session, registration } = await createSession(); - let rootContext: Context | undefined; - - registration.setResponses([ - (context) => { - rootContext = context; - const userText = context.messages - .filter((message) => (message as { role?: string }).role === "user") - .map((message) => JSON.stringify((message as { content?: unknown }).content)) - .join("\n"); - const mentionsTwoRepos = /repo-a/.test(userText) && /repo-b/.test(userText); - const promptSaysUseAgent = (context.systemPrompt ?? "").includes( - "Reach for Agent when the task matches an available agent", - ); - if (mentionsTwoRepos && promptSaysUseAgent) { - return fauxAssistantMessage( - [ - fauxToolCall("Agent", { - description: "Audit repo-a auth", - subagent_type: "general-purpose", - prompt: "Audit auth implementation under repo-a/. Report key files and flow.", - }), - fauxToolCall("Agent", { - description: "Audit repo-b auth", - subagent_type: "general-purpose", - prompt: "Audit auth implementation under repo-b/. Report key files and flow.", - }), - ], - { stopReason: "toolUse" }, - ); - } - return fauxAssistantMessage("would not delegate"); - }, - fauxAssistantMessage("repo-a auth uses session cookies"), - fauxAssistantMessage("repo-b auth uses JWT"), - (context) => fauxAssistantMessage(`Compared: ${JSON.stringify(context.messages).slice(0, 50)}`), - ]); - - await session.prompt("Compare how auth is implemented in repo-a/ and repo-b/."); - - expect(rootContext?.systemPrompt).toContain("Reach for Agent when the task matches an available agent"); - expect(rootContext?.systemPrompt).toContain("multiple Agent calls in the same assistant response"); - const finalSerialized = JSON.stringify(session.messages); - expect(finalSerialized).toContain("repo-a auth uses session cookies"); - expect(finalSerialized).toContain("repo-b auth uses JWT"); - - disposeSession(session); - }); - - it("scenario 2: broad codebase exploration → coordinator-aware router delegates to general-purpose", async () => { - const { session, registration } = await createSession(); - - registration.setResponses([ - makeRouter((userText, systemPrompt) => { - const broad = /across this codebase|where is .* handled/i.test(userText); - const repoSurveyHinted = systemPrompt.includes("explore or survey a repo") && systemPrompt.includes("concise read-only map"); - if (broad && repoSurveyHinted) { - return [ - fauxToolCall("Agent", { - description: "Locate rate limiting", - subagent_type: "general-purpose", - prompt: "Find every place rate limiting is implemented or referenced. Report files and symbols.", - }), - ]; - } - return "no delegation"; - }), - fauxAssistantMessage("found in src/middleware/rate-limit.ts and src/api/throttle.ts"), - fauxAssistantMessage("Rate limiting lives in middleware/rate-limit.ts and api/throttle.ts."), - ]); - - await session.prompt("Where is rate limiting handled across this codebase?"); - - const serialized = JSON.stringify(session.messages); - expect(serialized).toContain("rate-limit.ts"); - expect(registration.getPendingResponseCount()).toBe(0); - - disposeSession(session); - }); - - it("scenario 3: single-file lookup → router does NOT delegate", async () => { - const { session, registration } = await createSession(); - - registration.setResponses([ - makeRouter((userText, systemPrompt) => { - const singleFile = /line \d+ of |what does .* in src\/.*\.(ts|js) do/i.test(userText); - const knowsNotToDelegate = systemPrompt.includes( - "single-fact lookup where you already know the file", - ); - if (singleFile && knowsNotToDelegate) { - return "Line 42 of src/foo.ts does X — answered directly without delegation."; - } - return [ - fauxToolCall("Agent", { - description: "Should not happen", - prompt: "delegated wrongly", - }), - ]; - }), - ]); - - await session.prompt("What does line 42 of src/foo.ts do?"); - - const serialized = JSON.stringify(session.messages); - expect(serialized).toContain("answered directly without delegation"); - expect(serialized).not.toContain("delegated wrongly"); - expect(registration.getPendingResponseCount()).toBe(0); - - disposeSession(session); - }); - - it("scenario 4: explicit parallel audit → router emits multiple parallel Agent calls", async () => { - const { session, registration } = await createSession(); - - registration.setResponses([ - makeRouter((userText, systemPrompt) => { - const fanOut = /parallel.*TODOs.*FIXMEs.*skipped tests/i.test(userText); - const promptSaysParallel = systemPrompt.includes("multiple Agent calls"); - if (fanOut && promptSaysParallel) { - return [ - fauxToolCall("Agent", { description: "Find TODOs", subagent_type: "general-purpose", prompt: "Grep for TODO." }), - fauxToolCall("Agent", { description: "Find FIXMEs", subagent_type: "general-purpose", prompt: "Grep for FIXME." }), - fauxToolCall("Agent", { description: "Find skipped tests", subagent_type: "general-purpose", prompt: "Grep for it.skip / xit / describe.skip." }), - ]; - } - return "no delegation"; - }), - fauxAssistantMessage("3 TODOs"), - fauxAssistantMessage("1 FIXME"), - fauxAssistantMessage("2 skipped tests"), - fauxAssistantMessage("Audit complete: 3 TODOs, 1 FIXME, 2 skipped tests."), - ]); - - await session.prompt("In parallel, audit our code for TODOs, FIXMEs, and skipped tests."); - - const serialized = JSON.stringify(session.messages); - expect(serialized).toContain("3 TODOs"); - expect(serialized).toContain("1 FIXME"); - expect(serialized).toContain("2 skipped tests"); - expect(registration.getPendingResponseCount()).toBe(0); - - disposeSession(session); - }); - }); -});