Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
- Custom profiles may set `backend: pi` (default), `backend: codex`, or `backend: claude`. Codex-backed profiles run external `codex exec --json --dangerously-bypass-approvals-and-sandbox --ephemeral -- -` for one-shot calls, omit `--ephemeral` for keyed first calls, and use `codex exec resume --json ... <session_id> -` for keyed continuation; they send the task prompt on stdin, pass the profile body as `developer_instructions`, pass profile `model`/`thinking` through Codex CLI, parse `thread.started.thread_id`, token usage from Codex JSONL events, and estimate cost for listed models. Claude-backed profiles run external `claude -p --output-format stream-json --verbose --dangerously-skip-permissions --no-session-persistence` for one-shot calls, omit `--no-session-persistence` for keyed first calls, add `--resume <session_id>` for keyed continuation, send the task prompt on stdin, pass the profile body as `--append-system-prompt`, pass profile `model`/`thinking` through Claude Code, parse `system/init.session_id`, parse token usage from stream JSON, and use Claude Code's reported `total_cost_usd` when available. External CLI backends intentionally run in yolo/no-approval mode; only use them in trusted repositories.
- `tools` frontmatter is a pi-backend child-session allowlist only. External CLI profiles use their CLI's own tool and permission surface.
- There is no pi-flow permissions system in v1. Profiles are ordinary agents with optional prompts and tool allow-lists; external backends are explicit user dependencies.
- Pi-backed child sessions cannot launch other pi subagents. Do not give pi child sessions the `Agent` or `workflow` tool, or the coordinator prompt.
- Pi-backed child sessions cannot launch other pi subagents. Do not give pi child sessions the `Agent` or `workflow` tool, or the flow prompt (`buildFlowPrompt`).
- External CLI backends are not given pi `Agent`/`workflow` tools, but their own CLIs may expose nested/delegation features; do not try to block that from this extension.
- Parallel delegation is allowed and bounded by a global `maxConcurrentSubagents` limit (default `12`), which caps how many subagents run concurrently across the whole agent run. A slot is taken on launch and released on completion/failure/abort. In v2 this same cap is shared with the `workflow` tool.
- Do not put exact concurrency values in the model-facing coordinator prompt. The prompt should say parallel delegation is bounded and queued.
Expand Down Expand Up @@ -74,7 +74,7 @@
## CI and release workflow

- CI lives in `.github/workflows/ci.yml` and runs on pull requests plus pushes to `main`. It installs with `npm ci` and runs `npm run check` on Node 22.x and 24.x.
- E2E scripts are intentionally not part of required CI because they use real models and can be slow or inconclusive. Run them manually before risky releases: `npm run e2e -- --timeout-ms 300000`, `npm run e2e:workflow-features`, and `npm run e2e:session-key-resume -- --backend all` when session continuation changes.
- E2E scripts are intentionally not part of required CI because they use real models and can be slow or inconclusive. Run them manually before risky releases: `npm run e2e -- --timeout-ms 300000`, `npm run e2e:workflow-features`, `npm run e2e:prompt-routing` when prompt or routing guidance changes, and `npm run e2e:session-key-resume -- --backend all` when session continuation changes.
- Every real-model E2E driver must install the guard from `scripts/e2e/lib/deepseek-claude-env.mjs` so any Claude Code process routes through DeepSeek's Anthropic-compatible endpoint with isolated settings. Drivers must fail fast without `DEEPSEEK_API_KEY`/`DEEPSEEK_API_TOKEN` (or `--deepseek-api-key-env`) and must not fall back to Anthropic login or another Claude Code provider.
- There is intentionally no automated npm publish workflow right now; do not create tags expecting GitHub Actions to publish, and do not add an `NPM_TOKEN`-based workflow unless the user asks.
- Normal version-prep steps for agents:
Expand All @@ -98,12 +98,16 @@
Interactive tmux TUI runs use `deepseek/deepseek-v4-flash` with high thinking and isolated `--no-*` resource flags.

- `width`: validates eight parallel foreground delegations.
- `proactive-multirepo-v3`: validates proactive parallel delegation for a two-repo auth comparison.
- `proactive-fanout-v3`: validates proactive multi-lane delegation for TODO/FIXME/skipped-test search.
- `proactive-migration-v2`: validates proactive second-opinion delegation for a risky migration review.
- `proactive-multirepo-v3`: validated two-repo parallel Agent fan-out under the previous routing contract.
- `proactive-fanout-v3`: validated three-lane TODO/FIXME/skipped-test Agent fan-out under the previous routing contract.
- `proactive-migration-v2`: validated second-opinion Agent delegation under the previous routing contract.
- `max-concurrent-queue`: validates `--max-concurrent-subagents 1` with two parallel normal `Agent` calls; both completed (`FIRST_OK`, `SECOND_OK`) and no max-concurrency rejection was emitted.

Do not count `proactive-ship-v3` as proactive-pass evidence: the model handled that tiny ship-readiness fixture directly. This is acceptable as a behavioral limitation, but future prompt/tool tuning should continue improving this case.
Root-direct handling of narrow or small fixtures is intentional: `DIRECT_WORK_POLICY` explicitly permits staying in the root when delegation adds no value.

### Prompt routing behavior (current contract)

`scripts/e2e/prompt-routing-evidence.md` records the before/after comparison for the prompt consolidation. It covers the accepted routing boundary (root-direct, flat Agent fan-out, workflow for staged/structured/replay/large), privacy and isolation guarantees, prompt-size measurements, real-model routing outcomes, and variance limitations. The committed deterministic suite in `test/agent-contract.test.ts` and `test/delegation-scenarios.test.ts` covers the active-tool-gated sections, profile-roster deduplication, and session-key continuation contract without a real model.

### Workflow tool (v2)

Expand Down
23 changes: 9 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ Pi stays in control while `pi-flow` gives it two new ways to delegate:

| Primitive | Best for |
| --- | --- |
| **`Agent`** | One focused task: explore a repository, review a diff, investigate a bug, or draft a solution. |
| **`workflow`** | Several independent tasks: parallel research, multi-model review, staged pipelines, and synthesized results. |
| **`Agent`** | One focused delegated task or a small flat fan-out of independent work. |
| **`workflow`** | Saved workflows, dependent stages, control flow, structured results or branching, replay, or larger fan-out. |

Each subagent can run through a different harness and model. Route fast searches to Codex, UI review to Claude Code, local analysis to Pi—or define your own mix.

Expand Down Expand Up @@ -58,10 +58,10 @@ Then ask Pi to delegate naturally:
Use a subagent to map this repository without changing files, then summarize the important entry points.
```

Or ask for a wider workflow:
Or ask for staged orchestration:

```text
Review this PR from three independent angles—correctness, security, and test coverage—and synthesize the findings.
Use a workflow to classify each changed file by risk, run a risk-specific review, and return structured results.
```

Pi decides how to invoke `Agent` or `workflow`, shows live progress, and returns the combined result in the same conversation.
Expand Down Expand Up @@ -102,7 +102,7 @@ The lower-level `@kky42/pi-flow/runtime` export remains available for consumers

## Define your agent team

`pi-flow` includes one profile, `general-purpose`. Add specialists as Markdown files under:
`pi-flow` includes a minimal built-in profile and loads arbitrary custom profiles from Markdown files under:

```text
~/.pi/agent/subagents/<name>.md
Expand Down Expand Up @@ -130,7 +130,6 @@ Map the repository without modifying files. Return concise findings with paths a
---
description: Reviews implementation changes for correctness and missed edge cases.
backend: codex
model: gpt-5.5
thinking: high
---

Expand All @@ -143,7 +142,6 @@ Review the current diff. Lead with concrete findings and identify missing tests.
---
description: Reviews frontend work for UX, accessibility, and visual quality.
backend: claude
model: sonnet
thinking: high
---

Expand All @@ -168,7 +166,6 @@ A fresh call starts a clean child conversation in the same working directory. Pa
```ts
Agent({
description: "Map the authentication flow",
subagent_type: "explorer",
prompt: "Trace login from the HTTP entry point to session creation. Do not edit files.",
});
```
Expand All @@ -178,30 +175,28 @@ For follow-up work, reuse a stable `session_key`:
```ts
Agent({
description: "Draft the migration",
subagent_type: "implementation-expert",
session_key: "auth-migration",
prompt: "Propose a migration plan based on the current implementation.",
});

Agent({
description: "Revise the migration",
subagent_type: "implementation-expert",
session_key: "auth-migration",
prompt: "Revise the plan using the review feedback. Address rollback and compatibility.",
});
```

`pi-flow` maps that key to the backend-native session or thread and keeps the continuation explicit.

## Fan out with workflows
## Choose Agent or workflow by task shape

Use `workflow` when several lanes can run independently or when work benefits from multiple perspectives.
Use parallel `Agent` calls for a small flat set of independent investigations. Use `workflow` when orchestration matches a saved workflow or needs dependent stages, control flow, structured results or decisions, replay, or larger fan-out.

```text
Run a workflow that asks one agent to inspect the API, one to inspect persistence, and one to inspect tests. Synthesize the highest-risk gaps.
Use a workflow to classify each module with a strict risk result, then run the matching follow-up review for each classification.
```

You usually do not need to write workflow code yourself. Pi can generate and run a small trusted JavaScript workflow, then present the result.
You usually do not need to write workflow code yourself. Pi can generate and run a trusted JavaScript workflow, then present the result.

Ask Pi to save repeatable orchestration:

Expand Down
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@
"smoke:runtime": "npm run build:dist && node scripts/smoke-runtime.mjs",
"e2e": "node scripts/e2e/basic-metrics.mjs",
"e2e:workflow-features": "node scripts/e2e/workflow-features.mjs",
"e2e:prompt-routing": "node scripts/e2e/prompt-routing.mjs",
"e2e:session-key-resume": "node scripts/e2e/session-key-resume.mjs",
"test": "vitest run",
"prepack": "npm run build:dist"
Expand Down
123 changes: 123 additions & 0 deletions scripts/e2e/prompt-routing-evidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# Prompt-routing behavior evidence

This document records the bounded real-model comparison for the prompt consolidation based on commit `8954ab8` (`v2.1.5`). It is maintainer evidence, not a claim that stochastic routing is deterministic.

## Accepted behavior boundary

- Narrow local work may stay in the root.
- One focused delegation or a small flat fan-out uses direct `Agent` calls.
- Saved workflows, dependent stages, control flow, structured results or branching, replay, and larger fan-out use `workflow`.
- One logical child stream may reuse a `session_key`; independent work stays fresh.
- Routing expectations do not name or assert any profile, child model, specialist, or default.

## Privacy and isolation

`scripts/e2e/prompt-routing.mjs` creates a five-file report-CLI fixture and an isolated Pi agent directory. It disables discovered extensions, skills, templates, themes, context files, and session persistence, then loads only the extension under test. It checks that every fixture file remains byte-for-byte unchanged.

The driver does not read normal Pi session history or copy local profiles. The Pi process receives a minimal environment containing basic process settings plus one resolved DeepSeek credential; unrelated and alternate credential variables are not forwarded. Only terminal JSON events needed to assess tool calls, results, and usage are retained, high-volume streaming updates are discarded, and every discovered DeepSeek credential value plus local root paths are redacted before artifacts are analyzed or written. The driver always removes its isolated child-session directory and Claude runtime directory, even when other sanitized artifacts are retained.

## Reproduction

Prerequisites for the recorded run:

- Node `v22.23.1`
- npm `10.9.8`
- Pi `0.83.0`
- `DEEPSEEK_API_KEY` or `DEEPSEEK_API_TOKEN`

The driver intentionally requires the root model and thinking level instead of encoding defaults:

```bash
npm run e2e:prompt-routing -- \
--model deepseek/deepseek-v4-flash \
--thinking high \
--repetitions 2
```

Useful scoped form:

```bash
npm run e2e:prompt-routing -- \
--model deepseek/deepseek-v4-flash \
--thinking high \
--repetitions 2 \
--only flat,continuation,staged
```

The run installs the repository's DeepSeek Claude-provider guard even though these scenarios use Pi-backed children. `--extension <entry>` can compare another checkout or archived baseline. `--run-root` must identify a new or empty real directory; the driver marks ownership before writing and refuses to remove an unmarked root. An explicit `--agent-dir` must stay outside that root, while omitting it uses a driver-owned isolated directory. `--keep` retains sanitized fixture artifacts and `report.json`, while driver-owned session/runtime directories are always removed. The default removes the marked artifact root after a passing run.

## Prompt size

The same built-in-only profile map and the same counting method were used before and after. “Native” is the concatenated tool snippets/guidelines; “appended” is the `before_agent_start` pi-flow section.

| Prompt contribution | Before | After |
| --- | ---: | ---: |
| Tool-native metadata | 5,438 chars | 734 chars |
| Appended contract | 5,224 chars | 3,893 chars |
| Combined | 10,662 chars / 1,587 words | 4,627 chars / 675 words |
| Roster entry occurrences | 2 | 1 |

The combined contract shrank 56.6% by characters and 57.5% by words. The detailed appended contract remains because custom Pi system prompts omit normal tool-native snippets and guidelines.

## Real-model comparison

Setup for both sides:

- Root model: `deepseek/deepseek-v4-flash`
- Thinking: `high`
- Two repetitions per scenario
- Active tools: `read`, `bash`, `Agent`, `workflow`
- Ephemeral root session and isolated agent directory
- Same generated fixture, prompts, driver, process timeout, and checks
- Baseline extension: archived `8954ab8`
- Modified extension: this change

### Routing outcomes

| Scenario | Baseline | Modified | Accepted result |
| --- | --- | --- | --- |
| Narrow package lookup | root direct in 2/2 | root direct in 2/2 | yes |
| Focused repository map | root direct in 2/2 | root direct in 2/2 | yes; focused delegation is deliberately soft |
| Small flat three-lane review | three parallel Agent calls in 2/2 | three parallel Agent calls in 2/2 | yes; all calls fresh and completed |
| Same-child follow-up | two sequential Agent calls sharing one non-empty key in 2/2 | same in 2/2 | yes |
| Structured classify-then-follow-up | a pipeline workflow with at least two agent expressions and six completed children in 2/2 | same in 2/2 | yes |

All hard checks passed on both sides. The focused-map delegation observation was inconclusive in all four runs because the root handled the tiny fixture directly without workflow. Under the committed driver, the modified run's duplicate successful staged execution is also reported as inconclusive rather than hidden or treated as deterministic proof.

### Usage and cost

Usage sums root assistant usage plus nested usage from `Agent` and `workflow` tool results. Tokens include cache reads/writes as reported by Pi; costs are provider-reported.

| Side | Runs | Reported tokens | Reported cost | Summed wall time |
| --- | ---: | ---: | ---: | ---: |
| Baseline | 10 | 622,059 | $0.014483 | 267.3s |
| Modified | 10 | 710,465 | $0.014233 | 256.0s |

Modified-run tokens were 14.2% higher, while reported cost was 1.7% lower and summed wall time was 4.2% lower. These small mixed differences are not evidence of an execution-cost improvement: child output and root search behavior dominated this stochastic sample despite the much smaller static prompt.

## Variance and limitations

- Two repetitions expose obvious route variance but cannot prove deterministic behavior or generalize to other root models.
- The focused scenario is intentionally soft. Direct investigation of this tiny fixture is acceptable.
- Both prompt versions sometimes inspected files in the root before delegating. The modified flat scenario did this in one repetition; prompt wording does not reliably eliminate duplicate search.
- An intermediate modified-prompt trial exposed a staged-authoring regression: one run compressed classification and follow-up into three children. Restoring the `pipeline()` stage argument contract and a concise dependent-stage example produced six-child pipeline workflows in the final 2/2 rerun.
- In the final modified sample, one staged run retried an invalid workflow before succeeding, while the other ran two successful six-child workflows despite guidance not to repeat completed branches. The driver reports duplicate successful execution as inconclusive; it accepts the route only when the terminal script contains a schema-bearing pipeline with at least two agent expressions and completes at least six children.
- The driver tests entry-point and continuation behavior only. It neither selects nor grades profile names, child models, or specialist roles.
- Saved-name and replay behavior remain covered by `npm run e2e:workflow-features`; this driver focuses on root prompt routing.
- After artifact-safety hardening, the final driver was rerun once each for flat fan-out, continuation, and staged orchestration: all three passed with 294,296 reported tokens, $0.007482 reported cost, and 107.2s summed wall time.

## Deterministic validation

The committed suite separately covers:

- active Agent/workflow prompt sections for both, either, and neither tool;
- detailed contract retention with a custom Pi base system prompt;
- one dynamic profile roster occurrence;
- same-stream continuation and fresh parallel Agent execution;
- dynamic-schema and saved-workflow filename guidance matching runtime behavior.

Run all required checks with:

```bash
npm run check
```
Loading
Loading