Name the answer-engine crawlers, and ship the territory as well as the map

[robbyrob42]

The two gaps from the root-page audit that need no copy decisions and no waiting on external profiles. Worth 9 points on paper; the real argument for each is below.

robots.txt — explicit crawler blocks

The audit's own detail said bots_missing: [], bots_blocked: [], citation_bots_ok: true. Every one of the 27 crawlers it checks was already permitted. The single miss was citation_bots_explicit: false — we served one User-agent: * block and never named anyone.

So the answer-engine crawlers are now named individually — OpenAI, Anthropic, Perplexity, Google, Bing, Apple — each with the same Allow: / the wildcard already gave.

This is not redundant noise, and the file says why. A crawler obeys the most specific group that names it and ignores every other group, the wildcard included. A named block therefore replaces User-agent: * for that crawler rather than adding to it, which means each block has to be complete on its own. Each is. Access is unchanged in both directions — nothing granted, nothing withdrawn.

The maintenance cost is written into the file rather than left to be discovered: every crawler named is one this file has to keep current, and a crawler that renames itself falls back to the wildcard, which still allows it. The failure mode is a stale comment, not a blocked crawler. If that ever stops being true, delete the blocks and keep the wildcard.

llms.txt — the missing half

The convention pairs a map with the territory. We shipped the map.

New /llms-full.txt — every page expanded into one plain-text document, 2,115 words across 13 sections, so a retrieval system can take the substance in one request instead of fetching eight HTML pages and stripping markup from each. Generated from the copy deck like everything else here, which is what makes it safe to ship: it has no content of its own, so it cannot make a claim the site does not make.

/llms.txt gains ## Optional — a defined section name in the convention, not a label of our choosing. It means "skip these if the context window is short." It points at llms-full.txt, the three /ai/ endpoints, and the privacy policy.

Privacy moved into ## Optional from the main page list, where it was competing for attention with pages someone screening an expert actually reads first.

The privacy policy is linked, not inlined. It is long, it is shared with sites/www, and a model summarizing a legal commitment out of a flattened copy is a worse outcome than sending the reader to the authoritative page. Same reasoning excludes the intake form: a form is an action, not a document.

One fix found while building: the testimony-record criterion carries an optional link, and its copy is written to run into it ("Not buried:"). In plain text that left a line hanging on a colon, so the href is resolved and appended — the text equivalent of the anchor the page renders.

Verification

yarn build clean; /llms-full.txt and /llms.txt both emitted. astro check reports the same 2 pre-existing errors in scope.astro and no new ones.

Score movement is unverified until this deploys — the audit reads production.

Not in this PR

The remaining root-page gaps need copy decisions or things that do not exist yet: front-loading and outbound links (−4, copy), an author signal (−1), RSS (−2, needs the blog), Crunchbase and an /about link (Brand & Entity), and areaServed, which is still the deliberate omission #77 was going to re-price.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CpEXBMByKmpV1f9G7UUZhF