fix: verify every claim the compiler makes about its targets

CHANGELOG.md

@@ -2,6 +2,19 @@
 
 Format: [Keep a Changelog](https://keepachangelog.com). Versioning: semver — for skills *and* for this CLI, breaking prompt changes are breaking changes.
 
+## [0.24.0] — 2026-08-12
+
+A verification pass over every claim the compiler makes about the agents it targets, checked against each client's own source or documentation. **All eleven loading modes are correct** — the 14×–47× standing-cost benchmark stands — but three claims around them were overstated, and one of them was in the published numbers.
+
+### Fixed
+- **Aider does not read `CONVENTIONS.md` on its own.** Its docs are explicit: the file is loaded with `aider --read CONVENTIONS.md` or a `read:` entry in `.aider.conf.yml`. Aider is one of only two eager targets, so reporting a standing cost beside a file that is never in context charged a cost nobody was paying. `compile` now says which of the two situations a repo is in — unconfigured, configured-but-not-reading, or wired up (silent) — and the benchmark and README carry the same caveat. The measured numbers are unchanged; what they mean is now stated correctly.
+- **The README's status line had drifted eight releases** (it still said v0.15.0) and had omitted `import` from the working-command list since 0.16.0, because nothing checked either. Both corrected. The version is now stamped into the README by `site/build.mjs` from the same `package.json` everything else reads, so CI's existing `--check` gate catches it — the drift is eliminated rather than re-fixed. Three tests additionally hold the status paragraph to the real command surface, verified to fail on precisely the two drifts that occurred.
+- The benchmark script crashed after a docs edit whose backticks closed its template literal, and the failure hid behind a `git diff --exit-code` check that reads "script never ran" as "output unchanged". Fixed, and the release gate now checks exit codes explicitly.
+
+### Changed
+- Recorded, in the adapters themselves, three findings that were previously assumed: `.windsurf/rules/` is the **legacy** Devin path (`.devin/rules/` supersedes it and wins where both exist) and is still written deliberately, because it is the form that works on current Devin *and* on unmigrated Windsurf; Claude Code has **merged custom commands into skills**, so the `.claude/commands/` shim is inert on current versions and kept only for older ones; and `trigger: model_decision` is confirmed lazy, which makes a rule with an empty description inert.
+- A test now pins the invariant that an emitted skill's frontmatter `name` equals its directory name. Cline drops a mismatch with a silent `return null` and Zed rejects it outright, so getting this wrong is invisible skill loss rather than an error. The invariant already held; nothing enforced it.
+
 ## [0.23.0] — 2026-08-12
 
 ### Added

README.md

@@ -36,7 +36,7 @@ The identical instructions cost ~40 standing tokens on a target that lazy-loads
 
 Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you. I have not found another tool that surfaces that number.
 
-Kitbash always compiles to the cheapest loading mode a target actually supports — nine of the eleven lazy-load; Aider's `CONVENTIONS.md` and the `AGENTS.md` floor cannot, and carry the whole body every session. `--strict` turns budget overruns and degradation warnings into build failures.
+Kitbash always compiles to the cheapest loading mode a target actually supports — nine of the eleven lazy-load; Aider's `CONVENTIONS.md` and the `AGENTS.md` floor cannot, and carry the whole body every session. (Aider does not read `CONVENTIONS.md` on its own — until you add `read: CONVENTIONS.md` to `.aider.conf.yml`, it costs nothing and does nothing, and `compile` says so.) `--strict` turns budget overruns and degradation warnings into build failures.
 
 ### Why not a sync script?
 
@@ -61,7 +61,7 @@ Already have skills? A plain SKILL.md folder — the skills.sh / Claude Skills c
 
 Already carrying a hand-written `CLAUDE.md`, `.cursor/rules/`, `AGENTS.md`, and the rest of the copy-per-agent set? `kitbash import` reads them back into a single skill, measures what each one costs in standing context, and reports where the copies have drifted apart — so `kitbash compile` can regenerate them all from that one source. It touches nothing on disk until you remove the originals yourself.
 
-**Status.** v0.15.0, on npm and Homebrew, zero runtime dependencies, Node 20+. The KSF core is frozen and additive-only within the major version ([RFC 0002](rfcs/0002-ksf-1.0-stabilization.md)). Everything around it is early and labeled as such: `init`, `install`, `remove`, `list`, `compile`, `doctor`, `update`, `diff`, `lint`, `preview`, `explain`, and `test` work today; `audit`, `gate`, `search`, `publish`, `lore`, and `run` exit `7` and are on the [roadmap](docs/roadmap.md). One first-party skill ships (`prereview`); six more are specified but not built. Adoption is single-digit stars — if the measurement above is what you want, you are early.
+**Status.** v0.24.0, on npm and Homebrew, zero runtime dependencies, Node 20+. The KSF core is frozen and additive-only within the major version ([RFC 0002](rfcs/0002-ksf-1.0-stabilization.md)). Everything around it is early and labeled as such: `init`, `import`, `install`, `remove`, `list`, `compile`, `doctor`, `update`, `diff`, `lint`, `preview`, `explain`, and `test` work today; `audit`, `gate`, `search`, `publish`, `lore`, and `run` exit `7` and are on the [roadmap](docs/roadmap.md). One first-party skill ships (`prereview`); six more are specified but not built. Adoption is single-digit stars — if the measurement above is what you want, you are early.
 
 <p align="center">
   <a href="https://www.npmjs.com/package/kitbash"><img src="https://img.shields.io/npm/v/kitbash?color=ffb454" alt="npm version"></a>

docs/benchmarks/README.md

@@ -9,6 +9,8 @@ Two costs matter:
 - **Loaded** — tokens the agent reads when the skill is in play.
 - **Standing** — tokens sitting in the context window *every session, before the skill is even invoked*. Lazy targets keep only a stub; eager targets carry the whole body.
 
+One caveat on the eager rows: `agentsmd` is loaded by the agent automatically, but **aider does not read `CONVENTIONS.md` on its own** — it is loaded with `aider --read CONVENTIONS.md` or a `read:` entry in `.aider.conf.yml`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and `compile` says which case a repo is in.
+
 Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager *ratio* is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`.
 
 ## `prereview` — manifested (budget 1500, lazy)

packages/cli/package.json

@@ -1,6 +1,6 @@
 {
   "name": "kitbash",
-  "version": "0.23.0",
+  "version": "0.24.0",
   "description": "The package manager and compiler for AI agent skills — write once, run in every coding agent",
   "license": "Apache-2.0",
   "author": "Harsh Singh",

packages/cli/scripts/benchmark.mjs

@@ -145,6 +145,8 @@ Two costs matter:
 - **Loaded** — tokens the agent reads when the skill is in play.
 - **Standing** — tokens sitting in the context window *every session, before the skill is even invoked*. Lazy targets keep only a stub; eager targets carry the whole body.
 
+One caveat on the eager rows: \`agentsmd\` is loaded by the agent automatically, but **aider does not read \`CONVENTIONS.md\` on its own** — it is loaded with \`aider --read CONVENTIONS.md\` or a \`read:\` entry in \`.aider.conf.yml\`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and \`compile\` says which case a repo is in.
+
 Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager *ratio* is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`.
 
 `;

packages/cli/scripts/test.mjs

@@ -1849,6 +1849,78 @@ try {
   rmSync(curOnly, { recursive: true, force: true });
 }
 
+// ── the README's own claims stay true ────────────────────────────────────────
+// The status line drifted eight releases (it still said v0.15.0 at v0.23.0) and
+// the command list omitted `import` for seven, because nothing checked either.
+{
+  const readme = readFileSync(join(repoRoot, "README.md"), "utf8");
+  const pkgVersion = JSON.parse(readFileSync(join(here, "../package.json"), "utf8")).version;
+  const claimed = readme.match(/\*\*Status\.\*\* v(\d+\.\d+\.\d+)/)?.[1];
+  check("readme: the status line names the current version", claimed === pkgVersion, `README says v${claimed}, package.json says v${pkgVersion}`);
+
+  // Every command the CLI actually ships must be listed as working, and nothing
+  // that exits 7 may be — that is the promise the status paragraph makes.
+  const help = run(["--help"], repoRoot).out;
+  const planned = help.includes("Planned (not yet implemented):") ? help.split("Planned (not yet implemented):")[1] : "";
+  const working = (help.split("Planned (not yet implemented):")[0] || "")
+    .split("\n")
+    .map((l) => l.match(/^ {2}(\S+) {2,}/)?.[1])
+    .filter(Boolean);
+  const statusPara = readme.split("**Status.**")[1]?.split("\n\n")[0] ?? "";
+  const missing = working.filter((c) => !statusPara.includes(`\`${c}\``));
+  check("readme: every working command is listed in the status line", missing.length === 0, `missing: ${missing.join(", ")}`);
+  const plannedNames = planned.split("\n").map((l) => l.match(/^ {2}(\S+) {2,}/)?.[1]).filter(Boolean);
+  const misfiled = plannedNames.filter((c) => new RegExp(`\`${c}\`[^.]*work today`).test(statusPara));
+  check("readme: no planned command is claimed as working", misfiled.length === 0, `misfiled: ${misfiled.join(", ")}`);
+}
+
+// ── aider only pays the cost it is charged once it is wired up ───────────────
+// Aider does not auto-read CONVENTIONS.md; it needs `aider --read` or a `read:`
+// entry in .aider.conf.yml. Reporting an eager standing cost without saying so
+// would charge a file that is never in context.
+const aidTmp = mkdtempSync(join(tmpdir(), "kitbash-aider-"));
+try {
+  writeFileSync(join(aidTmp, "CONVENTIONS.md"), "# House rules\n");
+  run(["init"], aidTmp);
+  run(["install", `file:${fixture}`, "--yes"], aidTmp);
+
+  const bare = run(["compile"], aidTmp);
+  check("aider: unconfigured, compile says the file is not read", bare.out.includes("does not read it automatically"), bare.out);
+
+  writeFileSync(join(aidTmp, ".aider.conf.yml"), "model: gpt-4\n");
+  const unwired = run(["compile"], aidTmp);
+  check("aider: a conf without read: is still flagged", unwired.out.includes("does not read it"), unwired.out);
+
+  writeFileSync(join(aidTmp, ".aider.conf.yml"), "model: gpt-4\nread: CONVENTIONS.md\n");
+  const wired = run(["compile"], aidTmp);
+  check("aider: once wired up the note goes away", !wired.out.includes("does not read it"), wired.out);
+  check("aider: the standing cost is still reported when wired", wired.out.includes("aider is eager and cannot lazy-load"), wired.out);
+  check("aider: none of this fails --strict", run(["compile", "--strict"], aidTmp).status === 0);
+} finally {
+  rmSync(aidTmp, { recursive: true, force: true });
+}
+
+// The emitted skill's frontmatter name must equal its directory name: Cline drops
+// a mismatch with a silent `return null` and Zed rejects it outright, so a wrong
+// name here is invisible skill loss rather than an error.
+const nameTmp = mkdtempSync(join(tmpdir(), "kitbash-name-"));
+try {
+  const odd = join(nameTmp, "odd");
+  mkdirSync(odd, { recursive: true });
+  // Frontmatter name that KSF must NOT propagate: wrong charset, differs from dir.
+  writeFileSync(join(odd, "SKILL.md"), "---\nname: Totally-Different_Name\ndescription: A bare skill whose frontmatter name differs from its folder\n---\n\nBody.\n");
+  mkdirSync(join(nameTmp, ".agents"), { recursive: true });
+  run(["init"], nameTmp);
+  run(["install", `file:${odd}`, "--yes"], nameTmp);
+  run(["compile"], nameTmp);
+  const emitted = join(nameTmp, ".agents/skills/odd/SKILL.md");
+  check("name-invariant: the skill dir is named from the manifest", existsSync(emitted));
+  const fm = readFileSync(emitted, "utf8");
+  check("name-invariant: frontmatter name equals the directory name", /^---\nname: odd\n/.test(fm), fm.slice(0, 80));
+} finally {
+  rmSync(nameTmp, { recursive: true, force: true });
+}
+
 if (failures) {
   console.error(`\n${failures} test(s) failed`);
   process.exit(1);

packages/cli/src/adapters.ts

@@ -87,22 +87,48 @@ function eagerStandingNote(skill: LoadedSkill, adapter: Adapter, files: Compiled
 }
 
 /** Shared-file adapters (AGENTS.md, GEMINI.md): eager-loaded, marker-merged. */
-function mergedFileAdapter(id: string, file: string, detect: (root: string) => boolean): Adapter {
+function mergedFileAdapter(id: string, file: string, detect: (root: string) => boolean, extraNote?: (root: string) => string | null): Adapter {
   return {
     id,
     capabilities: [],
     loading: "eager",
     detect,
-    emit(skill, body) {
+    emit(skill, body, root) {
       const { name } = skill.manifest.skill;
       const { begin, end } = markers(name);
       const section = `${begin}\n${header(skill)}\n\n## Skill: ${name}\n\n${body.trim()}\n${end}`;
       const files = [{ path: file, content: section, merge: true }];
-      return { files, warnings: degradationWarnings(skill, this), notes: eagerStandingNote(skill, this, files) };
+      const extra = extraNote?.(root);
+      return {
+        files,
+        warnings: degradationWarnings(skill, this),
+        notes: [...eagerStandingNote(skill, this, files), ...(extra ? [extra] : [])],
+      };
     },
   };
 }
 
+/**
+ * Aider does not read `CONVENTIONS.md` on its own. Its docs are explicit that the
+ * file is loaded with `aider --read CONVENTIONS.md`, or by a `read:` entry in
+ * `.aider.conf.yml` — there is no auto-discovery. That matters more here than it
+ * would elsewhere: this adapter is one of only two eager targets, and the standing
+ * cost reported beside it is real only once the file is actually wired in.
+ * Unconfigured, `CONVENTIONS.md` costs nothing and does nothing, so the compiler
+ * says which of the two situations it is looking at rather than quietly implying
+ * the cost is already being paid.
+ */
+function aiderWiringNote(root: string): string | null {
+  const conf = [".aider.conf.yml", ".aider.conf.yaml"].map((f) => join(root, f)).find((p) => existsSync(p));
+  if (conf) {
+    const text = readFileIfExists(root, conf.slice(root.length + 1));
+    // A `read:` entry naming the file (alone or in a list) is enough to load it.
+    if (/^\s*read:/m.test(text) && text.includes("CONVENTIONS.md")) return null;
+    return `aider: CONVENTIONS.md is written, but ${conf.slice(root.length + 1)} does not read it — add "read: CONVENTIONS.md" there, or the file is never in context and the standing cost above is not actually charged.`;
+  }
+  return `aider: CONVENTIONS.md is written, but aider does not read it automatically — run "aider --read CONVENTIONS.md", or add "read: CONVENTIONS.md" to .aider.conf.yml. Until then the file costs nothing and does nothing.`;
+}
+
 /** Simple per-skill file adapters that differ only in path, loading mode, and frontmatter. */
 function fileAdapter(
   id: string,
@@ -136,6 +162,13 @@ const claudeCode: Adapter = {
   // only alongside the emit code that produces its primitive.
   capabilities: [],
   loading: "lazy",
+  // Verified lazy: Claude Code keeps each skill's description in context and
+  // loads the body only on invoke. Note that custom commands have since been
+  // merged into skills — a skill at .claude/skills/<n>/SKILL.md already creates
+  // /<n>, and when a .claude/commands/<n>.md exists alongside it the skill wins
+  // the name. The command file is therefore inert on current Claude Code and is
+  // still written only because it remains the trigger on versions from before
+  // that merge; it costs nothing standing, since commands load on use.
   detect: (root) => existsSync(join(root, ".claude")),
   emit(skill, body) {
     const { name, description } = skill.manifest.skill;
@@ -389,6 +422,20 @@ function clineConstraints(skill: LoadedSkill): string[] {
  * frontmatter — `model_decision` means the description is what sits in context
  * and the body loads only when the model judges it relevant, i.e. lazy.
  */
+/**
+ * Windsurf, now Devin Desktop. `.windsurf/rules/` is the LEGACY path: Devin's docs
+ * moved to `.devin/rules/`, and where both exist the `.devin/` copy takes
+ * precedence. The legacy path is still read by both Devin Desktop and the Devin
+ * CLI, which is why it is what gets written — a file at `.windsurf/rules/` works
+ * on current Devin *and* on older Windsurf, while `.devin/rules/` would silently
+ * do nothing for anyone who has not migrated. Revisit when the legacy fallback is
+ * announced for removal.
+ *
+ * `trigger: model_decision` is verified lazy: Devin's docs say only the
+ * `description` reaches the system prompt and the body is read when the model
+ * decides it is relevant. A rule with an empty description is therefore inert —
+ * nothing exists for the model to match on.
+ */
 const windsurf = fileAdapter(
   "windsurf",
   [],
@@ -417,6 +464,7 @@ const aider = mergedFileAdapter(
   "aider",
   "CONVENTIONS.md",
   (root) => existsSync(join(root, "CONVENTIONS.md")) || existsSync(join(root, ".aider.conf.yml")),
+  aiderWiringNote,
 );
 
 /** The floor: everything that reads AGENTS.md (Codex and many others). */

site/build.mjs

@@ -110,6 +110,20 @@ for (const e of readdirSync(site, { recursive: true, withFileTypes: false })) {
   else writeFileSync(p, out);
 }
 
+// ---- stamp the README's status line ----
+// It drifted eight releases before anything checked it, which is what a
+// hand-maintained version string in prose does. Stamped from the same source as
+// everything else so it cannot disagree with what is published.
+{
+  const p = join(repoRoot, "README.md");
+  const src = readFileSync(p, "utf8");
+  const out = src.replace(/(\*\*Status\.\*\* )v\d+\.\d+\.\d+/, `$1v${version}`);
+  if (out !== src) {
+    if (checkOnly) stale.push(`README.md (status line is not v${version})`);
+    else writeFileSync(p, out);
+  }
+}
+
 if (checkOnly && stale.length) {
   console.error("site is stale — run `node site/build.mjs` and commit the result:");
   for (const s of stale) console.error(`  ${s}`);