From 38454356d6e58f3ceb7fc2877e2b7c31b50b94d8 Mon Sep 17 00:00:00 2001 From: Harsh Singh Date: Sat, 22 Aug 2026 23:56:50 +0530 Subject: [PATCH] docs: name the prior art on token measurement, and state the estimator's error MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The README claimed no other tool surfaces a skill's token cost. That is no longer true: skills-check (npx skills-check budget) counts tokens per skill with a real cl100k_base tokenizer, breaks them down per section, detects redundancy between skills, and fails a build over a ceiling. Leaving the claim up would be the same credibility problem the 0.11.0 pass existed to fix, on the one number this project is known for. Corrected to what is still specific here: the per-target half. The same skill has a different standing cost on every agent because each loads it differently, and that falls out of a compile step that already knows each target's loading mode — skills-check measures the file, not the pair. Also states the estimator's real error rather than "a few percent". Measured against o200k_base: length/4 runs 2-11% high on this corpus (prereview 567 vs 515, stub 40 vs 35), biased toward failing budgets early, and the 14x ratio is 14.7x under a real tokenizer. --- README.md | 4 +++- docs/benchmarks/README.md | 2 +- packages/cli/scripts/benchmark.mjs | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index a420ca9..506a236 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,9 @@ compiled 1 skill for 3 targets The identical instructions cost ~40 standing tokens on a target that lazy-loads and ~560 on one that can't — a **14× per-session tax**, charged before the skill is ever invoked, on every agent that has no lazy mode. A separate unmanifested fixture measures 19 against 885, a 47× gap. One real third-party skill from the [skills.sh](https://www.skills.sh) convention carries ~5,044 tokens of instructions, which compile to ~5,101 standing tokens on an eager target. -Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you. I have not found another tool that surfaces that number. +Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you. + +Counting a skill's tokens is no longer unusual — [`skills-check`](https://www.skillscheck.ai/commands/budget) does it with a real `cl100k_base` tokenizer, per section, and will fail a build over a ceiling. What is still specific to Kitbash is the *per-target* half: the same skill has a different standing cost on every agent, because each one loads it differently, and that number falls out of the compile step that already knows each target's loading mode. `~40 on a lazy target, ~560 on an eager one` is a fact about the pair, not about the file. (Kitbash's own estimate is `length / 4`, not a tokenizer — measured against `o200k_base` it runs about 2-11% high on this corpus, and the ratio it reports is within half a multiple of the real one.) Kitbash always compiles to the cheapest loading mode a target actually supports — nine of the eleven lazy-load; Aider's `CONVENTIONS.md` and the `AGENTS.md` floor cannot, and carry the whole body every session. (Aider does not read `CONVENTIONS.md` on its own — until you add `read: CONVENTIONS.md` to `.aider.conf.yml`, it costs nothing and does nothing, and `compile` says so.) `--strict` turns budget overruns and degradation warnings into build failures. diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index 089aad5..e22c020 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -11,7 +11,7 @@ Two costs matter: One caveat on the eager rows: `agentsmd` is loaded by the agent automatically, but **aider does not read `CONVENTIONS.md` on its own** — it is loaded with `aider --read CONVENTIONS.md` or a `read:` entry in `.aider.conf.yml`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and `compile` says which case a repo is in. -Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager *ratio* is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`. +Token counts are estimates (`length / 4`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against `o200k_base` on this repo's corpus that estimator runs **about 2-11% high** — `prereview`'s body is 567 here and 515 in real tokens, its stub 40 against 35 — which is the safe direction for budget enforcement, since a skill fails its budget slightly early rather than late. The bias very largely cancels in the *ratio* the argument rests on: the 14x gap above is **14.7x** under a real tokenizer. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`. ## `prereview` — manifested (budget 1500, lazy) diff --git a/packages/cli/scripts/benchmark.mjs b/packages/cli/scripts/benchmark.mjs index 3aed266..6a14d21 100644 --- a/packages/cli/scripts/benchmark.mjs +++ b/packages/cli/scripts/benchmark.mjs @@ -147,7 +147,7 @@ Two costs matter: One caveat on the eager rows: \`agentsmd\` is loaded by the agent automatically, but **aider does not read \`CONVENTIONS.md\` on its own** — it is loaded with \`aider --read CONVENTIONS.md\` or a \`read:\` entry in \`.aider.conf.yml\`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and \`compile\` says which case a repo is in. -Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager *ratio* is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`. +Token counts are estimates (\`length / 4\`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against \`o200k_base\` on this repo's corpus that estimator runs **about 2-11% high** — \`prereview\`'s body is 567 here and 515 in real tokens, its stub 40 against 35 — which is the safe direction for budget enforcement, since a skill fails its budget slightly early rather than late. The bias very largely cancels in the *ratio* the argument rests on: the 14x gap above is **14.7x** under a real tokenizer. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`. `; for (const r of rows) {