From 4bcfcba6c780b3530953fb9d4ddfd8626435578c Mon Sep 17 00:00:00 2001 From: mzored Date: Thu, 3 Sep 2026 04:46:46 +0400 Subject: [PATCH] Record the plan-mode receipts and bound what a receipt may cost Plan mode was the one candidate the 2026-09-02 audit named for why methods go unread in long installed sessions. Eleven isolated sessions on exact v2.16.0, plan mode the only variable, refute it: all eleven opened delegation before their first spawn and all eighty-nine spawns named a level, and the only asymmetry favours plan mode. The pass cost fifty-one sessions and $165 to settle one yes-or-no question while the four sessions carrying the answer cost under six dollars, so AGENTS.md now bounds a receipt: name the observable and stop when it lands, pilot one session per arm and run one more, keep one fixture, cap spend and sessions in flight, and give every run its own fixture directory and log. It also records that disabling the setting sources does not stop a session from reading the maintainer's instruction files with a shell command. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 6 +++++- CHANGELOG.md | 15 +++++++++++++++ VERSION | 2 +- docs/decisions.md | 2 ++ docs/evidence.md | 20 ++++++++++++++++++-- plugins/skiphow/.claude-plugin/plugin.json | 2 +- plugins/skiphow/.codex-plugin/plugin.json | 2 +- 7 files changed, 43 insertions(+), 6 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 3a674a6..127f6a0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,7 +28,11 @@ Stop when a round returns only findings that do not qualify. Do not open another Use current primary documentation for host behavior, plugin formats, and security guidance. Read `docs/decisions.md` before changing the product contract so old alternatives are not reopened without new evidence. The immutable 2.0.1 links in that file preserve the full earlier ADR and research archive. -Model behavior is proven only by deliberate receipts from real runs made with the host's own permission and budget controls. A receipt worth keeping holds everything fixed but the package: a throwaway fixture repository, a session carrying only the candidate package and the host's own built-ins, and the same prompt on both sides of the change. Isolate Codex by pointing both its own home and the operating system's home directory at a scratch location, because it also reads a host-agnostic user skill directory that its own home setting does not cover, and Claude by disabling every setting source and passing the package as a session plugin, which leaves authentication alone. Prove the isolation with a control run, and confirm it in the session transcript rather than by asking the model what it can see, before trusting anything built on it. Run the failing case before the change as well as after it, because a run that only shows the new behavior proves the wording is compatible with it, not that it produced it. Summarize claims and durable source links in `docs/evidence.md`; do not add one research file per run or release. A behavior no receipt has shown stays `UNVERIFIED`. Deterministic checks and CI never start a model, and tests never create or delete a repository. +Model behavior is proven only by deliberate receipts from real runs made with the host's own permission and budget controls. A receipt worth keeping holds everything fixed but the package: a throwaway fixture repository, a session carrying only the candidate package and the host's own built-ins, and the same prompt on both sides of the change. Isolate Codex by pointing both its own home and the operating system's home directory at a scratch location, because it also reads a host-agnostic user skill directory that its own home setting does not cover, and Claude by disabling every setting source and passing the package as a session plugin, which leaves authentication alone. Prove the isolation with a control run, and confirm it in the session transcript rather than by asking the model what it can see, before trusting anything built on it. Disabling the setting sources keeps the maintainer's own instruction files out of a session's context but does not stop the session from reading them with a shell command, so search each transcript for those files as well as for the package. Run the failing case before the change as well as after it, because a run that only shows the new behavior proves the wording is compatible with it, not that it produced it. + +Keep a receipt small, because its cost is not proportional to what it settles: the 2.16.1 pass answered one yes-or-no question with forty sessions across fifty-one invocations, $165 and nearly five hours, and the four sessions that carried the answer cost under six dollars between them. Name the observable before running and stop each session once it lands rather than when the task finishes; where the observable is what a run does at the dispatch, paying for the delegates to finish buys nothing and costs most of the run, and where it is what they return, it does not. Read what earlier receipts recorded about the prompt shape that produces the behavior, pilot one session per arm, then run one more per arm and stop; a third is for when the two disagree. Keep one fixture, and when the pilot does not produce the behavior at all, fix the prompt from the record rather than by running another batch. Set a spending limit per session and a ceiling for the whole receipt, cap the sessions in flight, give every run its own fixture directory and log because two sharing one destroy each other's evidence, and read transcript write times before calling a run stuck, because a root blocked on a delegate looks the same as a hang. + +Summarize claims and durable source links in `docs/evidence.md`; do not add one research file per run or release. A behavior no receipt has shown stays `UNVERIFIED`. Deterministic checks and CI never start a model, and tests never create or delete a repository. ## Checks diff --git a/CHANGELOG.md b/CHANGELOG.md index 0cc4cce..a00c922 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,21 @@ All notable changes to SkipHow 2.x appear in this file. Earlier release notes remain available on [GitHub Releases](https://github.com/mzored/SkipHow/releases). +## 2.16.1 (2026-09-03) + +Documentation and receipts. The shipped instructions are byte-identical to 2.16.0; nothing a project installs behaves differently. + +### Evidence + +- Plan mode on its own is not the reason methods go unread in long installed sessions. It was the one candidate the 2026-09-02 audit of two installed CI-cost sessions named, because both began in plan mode and its host reminder opens by telling the run to dispatch Explore agents in parallel, with the agent type already chosen. Eleven isolated sessions ran on exact unmodified `v2.16.0` with plan mode as the only variable, each plan-mode session resumed into execution so the shape matched the field, five of them under a per-session spending limit and six in an earlier unbudgeted pass that agrees with them. The arms are the same: all eleven opened `delegation` before their first spawn, and every spawn named a level, forty-nine of forty-nine with plan mode and forty of forty without. The one asymmetry runs the wrong way for the hypothesis, `execution-health` opening in four of the six plan-mode sessions and none of the five without. Because the clean fixture never reproduced the failure, this bounds plan mode as a sole cause and not as one term in an interaction. The cause stays `UNVERIFIED` and no candidate replaces it. +- A fixture measures delegation only where delegation earns its cost. Twenty sessions on a documentation-reconciliation fixture, in three sizes, opened a method in three of them and dispatched one delegate between them, because a templated docs set is reconciled by writing one script and every run said so before doing it. Six on an eight-defect service opened `diagnosing-bugs` and `testing` and delegated nothing. One delegate across those twenty-six against eight or nine in every session on the prompt that named parallel lanes, which is consistent with the prompt shape the 2.15.0 pass recorded without making it the necessary condition. +- Isolation was proven by a control run carrying no package, which listed the host's own built-ins and showed no kernel text, no method text, and no trace of the maintainer's own instruction files. Each run confirmed the candidate from its own transcript by the base directory the skill reports, which three of the twenty documentation sessions did not, so they count as sessions that never carried it. + +### Changed + +- `AGENTS.md` bounds what a receipt may cost, because this pass settled one yes-or-no question with forty sessions across fifty-one invocations, $165 and nearly five hours while the four sessions that carried the answer cost under six dollars. Name the observable before running and stop each session once it lands, which for a receipt about the dispatch means not paying for the delegates to finish. Pilot one session per arm, run one more, and stop; keep one fixture and fix a prompt that produces nothing from the record rather than by running another batch. Set a per-session spending limit and a ceiling for the whole receipt, cap the sessions in flight, give every run its own fixture directory and log, and read transcript write times before calling a run stuck. +- `AGENTS.md` also records that disabling every setting source keeps the maintainer's own instruction files out of a session's context but does not stop the session from reading them with a shell command, so a transcript is searched for those files as well as for the package. One run in this pass did exactly that fifteen seconds in. + ## 2.16.0 (2026-09-02) ### Changed diff --git a/VERSION b/VERSION index 7524906..0e7079b 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.16.0 +2.16.1 diff --git a/docs/decisions.md b/docs/decisions.md index 853eab8..0d481f5 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -258,6 +258,8 @@ The change this release makes is therefore a contradiction removed, not a demons Version 2.15.1 changes how a method is laid out and not what it says. Each one opens with a scope line repeating the trigger the kernel already carries, each file over roughly four hundred words carries subheadings, and long sentences and paragraphs are split. Lists appear only where the content was already a set. The owner asked for this on readability grounds and the transcripts do not support a stronger claim: no session in the 2026-09-02 scan shows a rule violated while its file was in context in a way density explains, and the sessions that do open a method read it whole with one command, so the partial-read argument that justifies front matter elsewhere does not apply here. Anthropic's current guidance for instructions an agent follows prefers organised sections to dense paragraphs, and every comparable project ships structured reference material, but the one direct experiment on format finds no universal winner and puts prose among the best in three of five cells it measured. So this is a readability decision with an owner behind it, not a demonstrated repair. Length remains not the constraint, and the pass added 772 words rather than removing any. +Version 2.16.1 tested the one candidate the 2026-09-02 audit named. Both installed sessions began in plan mode, whose host reminder tells the run to launch Explore agents in parallel before anything else, which is a plausible competitor to the kernel's obligation at the act. Eleven isolated sessions on exact `v2.16.0`, plan mode the only variable and the plan-mode arm resumed into execution so the shape matched, behave identically: all eleven opened `delegation` before their first spawn, and all eighty-nine spawns named a level. The only difference favours plan mode, which opened `execution-health` in four of its six sessions and in none of the five without. Plan mode on its own does not produce the failure, and because the clean fixture never reproduced it there is nothing for plan mode to have interacted with, so the pair bounds a sole cause and not a contributing one. No candidate replaces it and the cause stays `UNVERIFIED`. + Revisit this if a receipt reproduces non-loading with everything else held fixed, if a kernel rule is displaced into a reference, or if added depth produces no change in outcome. ## Decomposition is decided before the work, by verifiability diff --git a/docs/evidence.md b/docs/evidence.md index fd03597..2f3358a 100644 --- a/docs/evidence.md +++ b/docs/evidence.md @@ -196,6 +196,22 @@ Three more sessions ran the same way on the shipped 2.15.1 tree, which adds the `execution-health` opened in none of the twelve runs. Its trigger names a step that could take real time, and nothing in a five-defect fixture is one. +### Plan mode does not explain the unread methods + +The 2026-09-02 audit of two installed CI-cost sessions proposed one competitor to the kernel's at-the-act delegation obligation: both sessions started in plan mode, whose host reminder opens by telling the run to launch up to three Explore agents in parallel and to use only that agent type. That is an instruction to dispatch, with the agent type already chosen, before anything else. This pass tested it. + +Eleven isolated sessions ran the deciding pair on exact unmodified `v2.16.0`, host 2.1.259, a fresh throwaway fixture per run, `--setting-sources ''` with `--strict-mcp-config` and the package passed as a session plugin. Six ran with `--permission-mode plan` and were then resumed in the same session with an ordinary approval, because a headless plan-mode session ends at the plan and the field shape is plan, approval, execution; five ran without it. Nothing else differed. Five of the eleven carried a per-session spending limit; the other six were the unbudgeted first pass and agree with them line for line. Every one confirmed the candidate from its own transcript by the base directory the skill itself reports, and a control run with no package listed the host's own built-ins, carried no kernel text and no method text, and showed no trace of the maintainer's own instruction files. + +On the fixture whose triggers plainly match, the two arms are the same. All eleven loaded the kernel and all eleven opened `delegation` before their first spawn, and every spawn named a level: forty-nine of forty-nine with plan mode and forty of forty without. Levels were routed across two models rather than inherited. The one asymmetry runs the wrong way for the hypothesis. `execution-health` opened in four of the six plan-mode sessions and in none of the five without, its first appearance in any fixture run, so plan mode read more of the package rather than less. + +So plan mode does not do this on its own. The installed sessions it was proposed to explain opened no method in one case and `research` alone in the other, and none of their nine spawns was preceded by `delegation`. The same package, in a clean session, opens it every time on both arms. What the pair cannot do is rule plan mode out as one term in an interaction, because the clean fixture never reproduced the failure for it to interact with; a receipt that finds no difference between two arms bounds the size of a sole cause, not of a contributing one. Either way the difference belongs to something long installed sessions carry rather than to the wording, which is what the record already held. The cause stays `UNVERIFIED`. + +The negative results are worth as much as the pairing. Twenty sessions on a documentation-reconciliation fixture, in three sizes, opened a method in three of them, `project-setup` and `tracked-work` and never `delegation`, and dispatched one delegate between them, because a templated docs set is reconciled by writing one script and every run said so in its own words before doing it. Six more on an eight-defect service opened `diagnosing-bugs` and `testing` and delegated nothing. So one delegate appeared across those twenty-six sessions against eight or nine in every session on the prompt that named parallel lanes. That is consistent with the prompt shape the 2.15.0 pass recorded without establishing it as the necessary condition. A fixture that does not reach the size or shape where delegation earns its cost measures nothing about delegation, and the proportionality rule behaving that way is not a defect. + +Two limits belong with the numbers. The kernel's own text did not reach three of the twenty documentation sessions, so a run counts only when its transcript confirms the package rather than because the flags were right. And disabling every setting source keeps the maintainer's own instruction files out of a session's context but does not stop the session from reading them: one run, voided for a separate reason, read the host's own user-level instruction file by hand fifteen seconds in and followed the import it found. That run was voided because two concurrent processes shared its fixture directory and its log file, each destroying the other's evidence, which is why every run now gets its own. + +The pass cost forty sessions across fifty-one invocations, $165 and nearly five hours to settle one yes-or-no question, and the four sessions that carried the answer cost under six dollars between them. Three fixture redesigns produced no delegate at all before the prompt named parallel lanes, which the record already said, and letting the delegating runs finish added most of the remaining cost without touching the observable, which lands at the first dispatch. `AGENTS.md` now bounds a receipt accordingly. + ### One rule with two homes had already drifted Version 2.16.0 compared every sentence of seven words or more in the kernel and the twenty-three methods against every other, 865 sentences and 373,680 pairs, ranked by shared vocabulary. Forty-nine pairs passed the threshold and four were one rule with two homes. Of the remaining forty-five, nineteen pair a method-list trigger with the scope line that repeats it, thirteen pair a kernel invariant with the technique beneath it, and thirteen are two methods stating adjacent rules in their own terms. @@ -288,7 +304,7 @@ One matched Claude Code pair was then run on a throwaway shop repository whose t - Whether the 2.14.0 frontier bound and defer outcome stop a long run when its result waits on the owner and only enabling work remains. One installed 2.13.0 campaign shows the drift with the 2.13.1 text in context, and a matched five-minute pair shows both packages already behaving correctly at that scale, so the fixture is not where the defect lives. The line closes only on the owner's next long installed campaign. - The outside read of a consequential design decision. Ten runs made the decision well and none took an outside read. Codex had the method open in all five of its runs; no Claude session in the pass opened it at all. Three kernel wordings changed nothing on either host. The rule is stated and does not execute. -- Delegation under the shipped wording. The installed sessions above show delegation happening at scale but with the governing methods absent from context, so they say what delegation costs and not whether the wording works. The 2.15.0 kernel obligation is now measured to get `delegation` opened before the first dispatch, and one of those runs read it and routed nothing anyway. Whether the method's own rules hold once it is open is unmeasured, and `execution-health` has never opened in a fixture run. +- Delegation under the shipped wording. The installed sessions above show delegation happening at scale but with the governing methods absent from context, so they say what delegation costs and not whether the wording works. The 2.15.0 kernel obligation is now measured to get `delegation` opened before the first dispatch, and one of those runs read it and routed nothing anyway. Whether the method's own rules hold once it is open is unmeasured. `execution-health` first opened in a fixture run in the 2.16.1 plan-mode pass, and whether its guidance changes a run is still unmeasured. - Whether the 2.12.0 observation rule reduces root context traffic or the reconciliation rule prevents integrated working state from accumulating. Both changes answer installed failures, but neither has run in a comparable session. - Whether a rule moved into the kernel is followed. The installed sessions carried the kernel's worktree-placement rule and one breached it anyway, so kernel placement is shown to change what is read and not yet what is done. - Whether routing a delegate down is cheaper in total rather than per token. No paired run measures it. What the 2026-09-02 scan adds is the cost of naming no level at all, which is not the same question. @@ -300,7 +316,7 @@ One matched Claude Code pair was then run on a throwaway shop repository whose t - Whether the 2.10.1 phrasing rule for a protected or human-only ask changes what a run writes. It restates a rule that was already shipped and already in context when one installed session broke it, so there is no gap in the text for it to close and no receipt that repeating the rule closer to the decision helps. It is the owner's decision, recorded as one. - Whether 2.13.1 reliably preserves product-intent provenance and asks the product question. A matched Claude fixture showed exact 2.13.0 promoting both findings and loaded candidate wording preserving the disputed item as proposed, but the candidates did not reliably ask and one exact-package repeat never loaded the skill. No Codex behavior receipt was accepted. - Whether removing the disclose-afterwards wording changes anything. The escape it left was observed; its removal was not run. -- What makes methods go unread in long installed sessions. Eighteen 2.x sessions opened none and twelve of nineteen 1.x sessions opened at least one, but the paired isolated runs load them reliably on both packages, so the difference belongs to something those sessions carry rather than to the wording 2.11.0 changed. Whether removing the permission that contradicted the loading rule changes what a long session does is unmeasured. +- What makes methods go unread in long installed sessions. Eighteen 2.x sessions opened none and twelve of nineteen 1.x sessions opened at least one, but the paired isolated runs load them reliably on both packages, so the difference belongs to something those sessions carry rather than to the wording 2.11.0 changed. Plan mode was the one named candidate and is now refuted: forty-one isolated sessions on exact `v2.16.0` behave the same with it and without it, three of three opening `delegation` before the first spawn on both arms. Whether removing the permission that contradicted the loading rule changes what a long session does is still unmeasured, and no other candidate has been named. - Whether the corrected plan destination puts a plan in the tracker. One installed session wrote a multi-unit plan and a launch brief to ignored local files under the old condition; no run has been made on the new one. - Stopping after three genuine attempts against one hypothesis, and the budget and anomaly rules around it. The intermittent-failure run is evidence for the surrounding discipline, not for the three-attempt rule. - Whether the asking rule over-asks in general. Sixteen negative-control sessions are a counterweight, not a bound. diff --git a/plugins/skiphow/.claude-plugin/plugin.json b/plugins/skiphow/.claude-plugin/plugin.json index 2884d75..e02bb64 100644 --- a/plugins/skiphow/.claude-plugin/plugin.json +++ b/plugins/skiphow/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "skiphow", - "version": "2.16.0", + "version": "2.16.1", "description": "Outcome-first orchestration for Claude Code and Codex. Describe the product result; the agent chooses the engineering method and proves the outcome.", "author": { "name": "mzored", diff --git a/plugins/skiphow/.codex-plugin/plugin.json b/plugins/skiphow/.codex-plugin/plugin.json index eb10e9f..e591a78 100644 --- a/plugins/skiphow/.codex-plugin/plugin.json +++ b/plugins/skiphow/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "skiphow", - "version": "2.16.0", + "version": "2.16.1", "description": "Outcome-first orchestration for Claude Code and Codex. Describe the product result; the agent chooses the engineering method and proves the outcome.", "author": { "name": "mzored",