From 245c42870e16a1b28de4a614ee943db9c40e6c06 Mon Sep 17 00:00:00 2001 From: Bloxster Date: Mon, 17 Aug 2026 10:14:58 +0200 Subject: [PATCH 1/4] docs(site): make the llms.txt artifacts discoverable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit llms.txt and llms-full.txt are published but nothing points at them: they are static files, so Docusaurus never routes them, the default sitemap omits them, and no page links to them. A crawler or agent can only reach them by guessing the path, which in practice means they are never found — a browsing model asked one flag question read ~20 GitHub issue threads instead, none of which are authoritative. Advertise them three ways: - two head tags, so every page declares where the machine-readable copies live - createSitemapItems, appending both URLs to the generated sitemap. The sibling ignorePatterns/lastmod options are closure-bound inside defaultCreateSitemapItems, so appended items are neither filtered nor double-processed and /search stays excluded - a reader-facing section on the MCP page, with a pointer from "Why using Erigon?" The section goes on the MCP page rather than "Why using Erigon?" because the latter is a card-grid landing page, whose body generate-llms.py replaces with synthesized bullets — prose added there would render on the site but never reach llms-full.txt. robots.txt is left alone: there is no standard directive for advertising llms.txt, and the Sitemap: line already there now leads to both files. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Dx34ND1m4ySTJXqMR8kRDX --- docs/site/docs/fundamentals/mcp.mdx | 20 +++++++++++++ .../docs/get-started/why-using-erigon.mdx | 2 ++ docs/site/docusaurus.config.ts | 30 +++++++++++++++++++ docs/site/static/llms-full.txt | 20 +++++++++++++ llms-full.txt | 20 +++++++++++++ 5 files changed, 92 insertions(+) diff --git a/docs/site/docs/fundamentals/mcp.mdx b/docs/site/docs/fundamentals/mcp.mdx index a1dce4837e7..2b59f2e80f2 100644 --- a/docs/site/docs/fundamentals/mcp.mdx +++ b/docs/site/docs/fundamentals/mcp.mdx @@ -326,3 +326,23 @@ In addition to tools, the MCP server exposes structured **resources** and **prom | `debug_logs` | Triage node issues from recent log output | | `torrent_status` | Snapshot download health check | | `sync_analysis` | Node sync progress and performance | + +--- + +## These Docs as a Single File + +MCP connects a model to your **running node**. To let a model answer questions about **Erigon itself** — flags, pruning modes, RPC namespaces, hardware sizing — point it at this documentation in machine-readable form, following the [llmstxt.org](https://llmstxt.org) convention: + +| File | Contents | Use it when | +|------|----------|-------------| +| [llms.txt](https://docs.erigon.tech/llms.txt) | Page index with titles and descriptions | The model fetches pages on demand, or its context window is small | +| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | Every documentation page, in full, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | + +Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale. + +```bash +# Give a local model the whole documentation, with no network access +curl -sO https://docs.erigon.tech/llms-full.txt +``` + +The full file is around 420 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. diff --git a/docs/site/docs/get-started/why-using-erigon.mdx b/docs/site/docs/get-started/why-using-erigon.mdx index c555338df50..d0d61a4e09a 100644 --- a/docs/site/docs/get-started/why-using-erigon.mdx +++ b/docs/site/docs/get-started/why-using-erigon.mdx @@ -179,3 +179,5 @@ The interface is **read-only** and localhost-bound — no risk of unintended sta # Register with Claude Code in one command claude mcp add --transport sse erigon http://127.0.0.1:8553/sse ``` + +Where MCP lets a model query your *node*, [llms.txt and llms-full.txt](../fundamentals/mcp#these-docs-as-a-single-file) let it read these *docs* — the whole documentation as one plain-text file, for offline use. diff --git a/docs/site/docusaurus.config.ts b/docs/site/docusaurus.config.ts index 2593cad2ce2..d9623d9770b 100644 --- a/docs/site/docusaurus.config.ts +++ b/docs/site/docusaurus.config.ts @@ -86,6 +86,27 @@ export default async function createConfig(): Promise { attributes: {}, innerHTML: 'window.plausible=window.plausible||function(){(plausible.q=plausible.q||[]).push(arguments)},plausible.init=plausible.init||function(i){plausible.o=i||{}};plausible.init()', }, + // Machine-readable copies of this site (llmstxt.org convention). Advertised + // on every page so crawlers and agents can find them without guessing the + // path — they are static files, so nothing else links to them. + { + tagName: 'link', + attributes: { + rel: 'alternate', + type: 'text/plain', + href: 'https://docs.erigon.tech/llms.txt', + title: 'Erigon Documentation — page index for LLMs', + }, + }, + { + tagName: 'link', + attributes: { + rel: 'alternate', + type: 'text/plain', + href: 'https://docs.erigon.tech/llms-full.txt', + title: 'Erigon Documentation — full text for LLMs', + }, + }, ], plugins: [ @@ -150,6 +171,15 @@ export default async function createConfig(): Promise { changefreq: 'weekly', priority: 0.5, ignorePatterns: ['/search'], + // The llms.txt artifacts live in static/, so Docusaurus never routes + // them and the default sitemap omits them. Nothing else on the web + // links to them either, which leaves them unindexable and unreachable + // by search — append them explicitly. + createSitemapItems: async ({defaultCreateSitemapItems, ...rest}) => [ + ...(await defaultCreateSitemapItems(rest)), + {url: 'https://docs.erigon.tech/llms.txt', changefreq: 'weekly', priority: 0.5}, + {url: 'https://docs.erigon.tech/llms-full.txt', changefreq: 'weekly', priority: 0.5}, + ], }, } satisfies Preset.Options], ], diff --git a/docs/site/static/llms-full.txt b/docs/site/static/llms-full.txt index d2eddaf7524..946f9c13c2f 100644 --- a/docs/site/static/llms-full.txt +++ b/docs/site/static/llms-full.txt @@ -3977,6 +3977,26 @@ In addition to tools, the MCP server exposes structured **resources** and **prom --- +## These Docs as a Single File + +MCP connects a model to your **running node**. To let a model answer questions about **Erigon itself** — flags, pruning modes, RPC namespaces, hardware sizing — point it at this documentation in machine-readable form, following the [llmstxt.org](https://llmstxt.org) convention: + +| File | Contents | Use it when | +|------|----------|-------------| +| [llms.txt](https://docs.erigon.tech/llms.txt) | Page index with titles and descriptions | The model fetches pages on demand, or its context window is small | +| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | Every documentation page, in full, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | + +Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale. + +```bash +# Give a local model the whole documentation, with no network access +curl -sO https://docs.erigon.tech/llms-full.txt +``` + +The full file is around 420 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. + +--- + # Otterscan URL: https://docs.erigon.tech/fundamentals/otterscan diff --git a/llms-full.txt b/llms-full.txt index d2eddaf7524..946f9c13c2f 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -3977,6 +3977,26 @@ In addition to tools, the MCP server exposes structured **resources** and **prom --- +## These Docs as a Single File + +MCP connects a model to your **running node**. To let a model answer questions about **Erigon itself** — flags, pruning modes, RPC namespaces, hardware sizing — point it at this documentation in machine-readable form, following the [llmstxt.org](https://llmstxt.org) convention: + +| File | Contents | Use it when | +|------|----------|-------------| +| [llms.txt](https://docs.erigon.tech/llms.txt) | Page index with titles and descriptions | The model fetches pages on demand, or its context window is small | +| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | Every documentation page, in full, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | + +Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale. + +```bash +# Give a local model the whole documentation, with no network access +curl -sO https://docs.erigon.tech/llms-full.txt +``` + +The full file is around 420 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. + +--- + # Otterscan URL: https://docs.erigon.tech/fundamentals/otterscan From eee030a9f1c37e951048dc8420c5af6d61fcce63 Mon Sep 17 00:00:00 2001 From: Bloxster Date: Mon, 17 Aug 2026 10:27:56 +0200 Subject: [PATCH 2/4] fix(docs): stop the landing-page card parser from swallowing cards MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit _LANDING_CARD_RE matched a whole card with one pattern: `[^<]+` for the title and description text, `(?:.*?)` for the gaps, under re.DOTALL. `[^<]+` cannot cross a `<`, so a description containing inline markup (, ) fails to match where it stands — and the engine then scans forward through the permissive gap and matches the *next* card's description and , swallowing the card in between and pairing a title with the wrong description. This is live in the published corpus, on the page whose job is explaining what makes Erigon different. why-using-erigon has 11 cards; llms-full.txt carried 8: Immutable, Decentralised Data <- Staged Sync's description Flexible Pruning <- RPC Providers' description Staged Sync, RPC Providers & Large Stakers, Developers <- absent Parse in two stages instead: match each block first, then find the title and description within that block only. A card boundary is then unrepresentable, so no match can cross one. Add a count guard. This failed silently for as long as it existed because `--check` only compares generated output against committed output, which makes a systematic generator bug invariant under it: CI stays green while the corpus is wrong. The guard compares parsed cards against lp-card-title occurrences and fails loudly on a mismatch. --- docs/site/scripts/generate-llms.py | 51 ++++++++++++++------ docs/site/scripts/test_generate_llms.py | 62 +++++++++++++++++++++++++ docs/site/static/llms-full.txt | 7 ++- llms-full.txt | 7 ++- 4 files changed, 110 insertions(+), 17 deletions(-) diff --git a/docs/site/scripts/generate-llms.py b/docs/site/scripts/generate-llms.py index 7ca09256013..e7a115c78e9 100644 --- a/docs/site/scripts/generate-llms.py +++ b/docs/site/scripts/generate-llms.py @@ -285,16 +285,25 @@ def strip_mdx(text): # index pages. After MDX stripping, they collapse to a structureless pile of # title/desc fragments. We detect the lp-card pattern and synthesize a clean # bullet list instead, keeping link structure intact for LLM consumption. -_LANDING_CARD_RE = re.compile( - r']*\bto=[\'"]([^\'"]+)[\'"][^>]*>' - r'(?:.*?)' - r']*\blp-card-title\b[^>]*>([^<]+)' - r'(?:.*?)' - r']*\blp-card-desc\b[^>]*>([^<]+)' - r'(?:.*?)' - r'', +# Parsed in two stages, deliberately. A single pattern spanning the whole card +# cannot express "and never cross into the next card": with `[^<]+` for the text +# and `.*?` for the gaps, a description containing inline markup (``, +# ``) fails to match locally, and the engine then scans forward and pairs +# the title with the *next* card's description — silently swallowing the card in +# between. Matching each block first makes a card boundary unrepresentable. +_LANDING_LINK_RE = re.compile( + r']*\bto=[\'"]([^\'"]+)[\'"][^>]*>(.*?)', re.DOTALL, ) +_CARD_TITLE_RE = re.compile( + r']*\blp-card-title\b[^>]*>(.*?)', re.DOTALL) +_CARD_DESC_RE = re.compile( + r']*\blp-card-desc\b[^>]*>(.*?)', re.DOTALL) + + +def _card_text(fragment): + """Flatten a card's inner HTML to plain text, collapsing whitespace.""" + return re.sub(r'\s+', ' ', re.sub(r'<[^>]+>', '', fragment)).strip() _LANDING_HERO_RE = re.compile( r']*\blp-hero\b[^>]*>' r'(?:.*?)' @@ -312,10 +321,28 @@ def synthesize_landing(raw_body): Card hrefs that start with `/` are rewritten to absolute BASE_URL form so bullets remain useful when the body is read out of context. """ - cards = list(_LANDING_CARD_RE.finditer(raw_body)) + cards = [] + for link in _LANDING_LINK_RE.finditer(raw_body): + href, block = link.group(1), link.group(2) + title = _CARD_TITLE_RE.search(block) + desc = _CARD_DESC_RE.search(block) + if not title or not desc: + continue + cards.append((href, _card_text(title.group(1)), + _card_text(desc.group(1)))) if not cards: return None + # A card that parses to nothing is invisible in the output, and `--check` + # only compares generated output against committed output — so a systematic + # parser regression stays green in CI while the corpus quietly loses pages. + expected = raw_body.count('lp-card-title') + if len(cards) != expected: + raise SystemExit( + f"synthesize_landing dropped cards: parsed {len(cards)} " + f"of {expected} lp-card-title occurrences" + ) + out = [] hero = _LANDING_HERO_RE.search(raw_body) if hero: @@ -326,12 +353,10 @@ def synthesize_landing(raw_body): out.append("## Sections") out.append("") - for m in cards: - href = m.group(1).strip() + for href, title, desc in cards: + href = href.strip() if href.startswith("/"): href = BASE_URL + href.rstrip("/") - title = m.group(2).strip() - desc = m.group(3).strip() out.append(f"- [{title}]({href}): {desc}") return "\n".join(out) diff --git a/docs/site/scripts/test_generate_llms.py b/docs/site/scripts/test_generate_llms.py index 187ec6c154c..b7cce696977 100644 --- a/docs/site/scripts/test_generate_llms.py +++ b/docs/site/scripts/test_generate_llms.py @@ -159,6 +159,68 @@ def test_no_cards_returns_none(self): out = g.synthesize_landing("# Hello\n\nProse only, no cards.") self.assertIsNone(out) + def test_desc_with_inline_markup_is_extracted(self): + """Inline markup in a description must not defeat the match.""" + body = """ +
+ +
Flexible Pruning
+
--prune.mode=minimal gives a smaller footprint.
+ +
+""" + out = g.synthesize_landing(body) + self.assertIsNotNone(out) + self.assertIn("--prune.mode=minimal gives a smaller footprint.", out) + self.assertNotIn("", out) + self.assertNotIn("", out) + + def test_markup_card_does_not_swallow_the_next_card(self): + """A card whose desc has markup must not steal the following card's desc.""" + body = """ +
+ +
Immutable Data
+
Historical data lives in immutable files.
+ + +
Staged Sync
+
Splits processing into specialised stages.
+ +
+""" + out = g.synthesize_landing(body) + self.assertIsNotNone(out) + bullets = [ln for ln in out.split("\n") if ln.startswith("- [")] + self.assertEqual(len(bullets), 2) + self.assertIn( + "- [Immutable Data](https://docs.erigon.tech/immutable): " + "Historical data lives in immutable files.", + bullets, + ) + self.assertIn( + "- [Staged Sync](https://docs.erigon.tech/staged-sync): " + "Splits processing into specialised stages.", + bullets, + ) + + def test_dropped_card_raises_instead_of_silently_shrinking(self): + """A card the parser cannot read must fail loudly — `--check` cannot see it.""" + body = """ +
+ +
Fine
+
Parses correctly.
+ + +
Missing Its Description
+ +
+""" + with self.assertRaises(SystemExit) as ctx: + g.synthesize_landing(body) + self.assertIn("dropped cards", str(ctx.exception)) + class LeadingH1StripTests(unittest.TestCase): """The build() body-prep step strips a leading H1 to avoid duplicate headings.""" diff --git a/docs/site/static/llms-full.txt b/docs/site/static/llms-full.txt index 946f9c13c2f..0f1ec24c325 100644 --- a/docs/site/static/llms-full.txt +++ b/docs/site/static/llms-full.txt @@ -38,12 +38,15 @@ URL: https://docs.erigon.tech/get-started/why-using-erigon - [Integrated Consensus Layer (Caplin)](../fundamentals/caplin): Built-in consensus client — no need to run and manage separate software like Lighthouse. One binary handles both execution and consensus, simplifying setup for stakers and node runners. - [State Storage (Flat DB)](../fundamentals/optimizing-storage): Replaces the Merkle Patricia Trie with a flat key-value database. Faster reads and writes, and a smaller on-disk footprint for any node type. -- [Immutable, Decentralised Data](../fundamentals/modules/downloader): Splits data processing into specialised stages, minimising random disk I/O and write amplification. Efficiently integrates large datasets into Erigon's flat DB. +- [Immutable, Decentralised Data](../fundamentals/modules/downloader): Historical data lives in immutable files distributed via BitTorrent — up to 10× cheaper to backup, repair, and distribute. No double-disk usage, no complex serialisation. +- [Staged Sync](../fundamentals/pruning-modes): Splits data processing into specialised stages, minimising random disk I/O and write amplification. Efficiently integrates large datasets into Erigon's flat DB. - [Predictable RPC Performance](../fundamentals/modules/rpc-daemon): No background compaction means no unpredictable resource spikes. RPC throughput stays stable and consistent — critical for providers and validators who cannot afford surprises. - [Modularity](../fundamentals/modules/): Core node, RPC Daemon, and TxPool run as independent processes with per-component resource limits. If one crashes the rest stay alive. Run multiple instances without downtime. - [OtterSync](../fundamentals/pruning-modes): Shifts initial sync from CPU-intensive transaction re-execution to network download — similar to BitTorrent for state data. Reduces sync time for both full and Archive nodes. -- [Flexible Pruning](../fundamentals/optimizing-storage): Modular architecture and predictable performance handle high-volume traffic with low latency. Smaller database cuts hardware costs. Caplin reduces missed attestations and slashing risk. +- [Flexible Pruning](../fundamentals/optimizing-storage): --prune.mode=minimal gives a smaller-than-full-node footprint that still satisfies all validator requirements. Right-size your storage without sacrificing functionality. +- [RPC Providers & Large Stakers](../fundamentals/modules/rpc-daemon): Modular architecture and predictable performance handle high-volume traffic with low latency. Smaller database cuts hardware costs. Caplin reduces missed attestations and slashing risk. - [Home Users & Solo Stakers](../staking/): Run a full or archive node on a consumer SSD. Fast sync cuts setup from days to hours. No third-party servers — your node, your keys, your contribution to decentralisation. +- [Developers](../interacting-with-erigon/): Full trace_ namespace, historical eth_getProof, historical blobs, and gRPC API. Modular components let you update, extend, or replace parts without rebuilding the whole stack. --- diff --git a/llms-full.txt b/llms-full.txt index 946f9c13c2f..0f1ec24c325 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -38,12 +38,15 @@ URL: https://docs.erigon.tech/get-started/why-using-erigon - [Integrated Consensus Layer (Caplin)](../fundamentals/caplin): Built-in consensus client — no need to run and manage separate software like Lighthouse. One binary handles both execution and consensus, simplifying setup for stakers and node runners. - [State Storage (Flat DB)](../fundamentals/optimizing-storage): Replaces the Merkle Patricia Trie with a flat key-value database. Faster reads and writes, and a smaller on-disk footprint for any node type. -- [Immutable, Decentralised Data](../fundamentals/modules/downloader): Splits data processing into specialised stages, minimising random disk I/O and write amplification. Efficiently integrates large datasets into Erigon's flat DB. +- [Immutable, Decentralised Data](../fundamentals/modules/downloader): Historical data lives in immutable files distributed via BitTorrent — up to 10× cheaper to backup, repair, and distribute. No double-disk usage, no complex serialisation. +- [Staged Sync](../fundamentals/pruning-modes): Splits data processing into specialised stages, minimising random disk I/O and write amplification. Efficiently integrates large datasets into Erigon's flat DB. - [Predictable RPC Performance](../fundamentals/modules/rpc-daemon): No background compaction means no unpredictable resource spikes. RPC throughput stays stable and consistent — critical for providers and validators who cannot afford surprises. - [Modularity](../fundamentals/modules/): Core node, RPC Daemon, and TxPool run as independent processes with per-component resource limits. If one crashes the rest stay alive. Run multiple instances without downtime. - [OtterSync](../fundamentals/pruning-modes): Shifts initial sync from CPU-intensive transaction re-execution to network download — similar to BitTorrent for state data. Reduces sync time for both full and Archive nodes. -- [Flexible Pruning](../fundamentals/optimizing-storage): Modular architecture and predictable performance handle high-volume traffic with low latency. Smaller database cuts hardware costs. Caplin reduces missed attestations and slashing risk. +- [Flexible Pruning](../fundamentals/optimizing-storage): --prune.mode=minimal gives a smaller-than-full-node footprint that still satisfies all validator requirements. Right-size your storage without sacrificing functionality. +- [RPC Providers & Large Stakers](../fundamentals/modules/rpc-daemon): Modular architecture and predictable performance handle high-volume traffic with low latency. Smaller database cuts hardware costs. Caplin reduces missed attestations and slashing risk. - [Home Users & Solo Stakers](../staking/): Run a full or archive node on a consumer SSD. Fast sync cuts setup from days to hours. No third-party servers — your node, your keys, your contribution to decentralisation. +- [Developers](../interacting-with-erigon/): Full trace_ namespace, historical eth_getProof, historical blobs, and gRPC API. Modular components let you update, extend, or replace parts without rebuilding the whole stack. --- From 1dd741e8c18170cc8c6a7089eff79c119d9a70bb Mon Sep 17 00:00:00 2001 From: Bloxster Date: Mon, 17 Aug 2026 14:53:27 +0200 Subject: [PATCH 3/4] =?UTF-8?q?docs(site):=20address=20review=20=E2=80=94?= =?UTF-8?q?=20describedby=20relation,=20guard=20hole,=20corpus=20claims?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review feedback from @yperbasis and Copilot on #23336. Five fixes. 1. Advertise llms.txt with rel="describedby", not rel="alternate". The llmstxt.org proposal defines describedby for the llms.txt file that covers a page, and reserves alternate + text/markdown for a *per-page* Markdown representation. A site-wide index is not an alternate representation of every page, and v2-aware agents look for describedby. llms-full.txt is no longer advertised in head at all: it describes no single page. It stays discoverable through llms.txt, the sitemap, and the MCP docs page. 2. Close a hole in the card-count guard. `if not cards: return None` ran before the count was taken, so a grid where *every* card failed to parse was indistinguishable from an ordinary prose page: the caller fell back to strip_mdx and the guard never ran — silently degrading the exact case it exists to catch. Count first, parse second, and return None only when the page has no cards at all. 3. Stop claiming llms-full.txt holds "every documentation page, in full". It does not: synthesize_landing replaces the whole body of a card-grid page with its card list, so why-using-erigon loses its introduction and prose. Describe the corpus as cleaned rather than verbatim, and say what is dropped. 4. Stop grouping llms.txt with llms-full.txt as "the whole documentation as one plain-text file" on why-using-erigon. llms.txt is only an index. 5. Drop "nothing else links to them" from the config comment — this PR adds the MCP page links, which makes it false. Also refresh the stated file size, 420 KB -> 430 KB. --- docs/site/docs/fundamentals/mcp.mdx | 6 ++-- .../docs/get-started/why-using-erigon.mdx | 2 +- docs/site/docusaurus.config.ts | 28 ++++++++----------- docs/site/scripts/generate-llms.py | 12 ++++++-- docs/site/scripts/test_generate_llms.py | 20 +++++++++++++ docs/site/static/llms-full.txt | 6 ++-- llms-full.txt | 6 ++-- 7 files changed, 53 insertions(+), 27 deletions(-) diff --git a/docs/site/docs/fundamentals/mcp.mdx b/docs/site/docs/fundamentals/mcp.mdx index 2b59f2e80f2..b9757fb4cc1 100644 --- a/docs/site/docs/fundamentals/mcp.mdx +++ b/docs/site/docs/fundamentals/mcp.mdx @@ -336,13 +336,15 @@ MCP connects a model to your **running node**. To let a model answer questions a | File | Contents | Use it when | |------|----------|-------------| | [llms.txt](https://docs.erigon.tech/llms.txt) | Page index with titles and descriptions | The model fetches pages on demand, or its context window is small | -| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | Every documentation page, in full, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | +| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of every documentation page, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale. +The corpus is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read it as the documentation's prose, not as a byte-for-byte mirror of the rendered site. + ```bash # Give a local model the whole documentation, with no network access curl -sO https://docs.erigon.tech/llms-full.txt ``` -The full file is around 420 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. +The full file is around 430 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. diff --git a/docs/site/docs/get-started/why-using-erigon.mdx b/docs/site/docs/get-started/why-using-erigon.mdx index d0d61a4e09a..66b91ca6c14 100644 --- a/docs/site/docs/get-started/why-using-erigon.mdx +++ b/docs/site/docs/get-started/why-using-erigon.mdx @@ -180,4 +180,4 @@ The interface is **read-only** and localhost-bound — no risk of unintended sta claude mcp add --transport sse erigon http://127.0.0.1:8553/sse ``` -Where MCP lets a model query your *node*, [llms.txt and llms-full.txt](../fundamentals/mcp#these-docs-as-a-single-file) let it read these *docs* — the whole documentation as one plain-text file, for offline use. +Where MCP lets a model query your *node*, [llms.txt and llms-full.txt](../fundamentals/mcp#these-docs-as-a-single-file) let it read these *docs*: an index of every page, and the documentation text as one plain-text file for offline use. diff --git a/docs/site/docusaurus.config.ts b/docs/site/docusaurus.config.ts index d9623d9770b..67d82496de4 100644 --- a/docs/site/docusaurus.config.ts +++ b/docs/site/docusaurus.config.ts @@ -86,27 +86,21 @@ export default async function createConfig(): Promise { attributes: {}, innerHTML: 'window.plausible=window.plausible||function(){(plausible.q=plausible.q||[]).push(arguments)},plausible.init=plausible.init||function(i){plausible.o=i||{}};plausible.init()', }, - // Machine-readable copies of this site (llmstxt.org convention). Advertised - // on every page so crawlers and agents can find them without guessing the - // path — they are static files, so nothing else links to them. + // Point every page at the llms.txt index that covers it. `describedby` is + // the relation the llmstxt.org proposal defines for exactly this; its + // `alternate` + `text/markdown` pairing is reserved for a *per-page* + // Markdown representation, which this site-wide index is not. + // llms-full.txt is deliberately not advertised here: it is not a + // description of any single page. It stays discoverable through llms.txt, + // the sitemap, and the MCP docs page. { tagName: 'link', attributes: { - rel: 'alternate', - type: 'text/plain', + rel: 'describedby', href: 'https://docs.erigon.tech/llms.txt', title: 'Erigon Documentation — page index for LLMs', }, }, - { - tagName: 'link', - attributes: { - rel: 'alternate', - type: 'text/plain', - href: 'https://docs.erigon.tech/llms-full.txt', - title: 'Erigon Documentation — full text for LLMs', - }, - }, ], plugins: [ @@ -172,9 +166,9 @@ export default async function createConfig(): Promise { priority: 0.5, ignorePatterns: ['/search'], // The llms.txt artifacts live in static/, so Docusaurus never routes - // them and the default sitemap omits them. Nothing else on the web - // links to them either, which leaves them unindexable and unreachable - // by search — append them explicitly. + // them and the default sitemap omits them — which leaves them + // unindexable by search. Append them explicitly. This is also how + // llms-full.txt stays reachable, since it is not advertised in head. createSitemapItems: async ({defaultCreateSitemapItems, ...rest}) => [ ...(await defaultCreateSitemapItems(rest)), {url: 'https://docs.erigon.tech/llms.txt', changefreq: 'weekly', priority: 0.5}, diff --git a/docs/site/scripts/generate-llms.py b/docs/site/scripts/generate-llms.py index e7a115c78e9..cc028272ea8 100644 --- a/docs/site/scripts/generate-llms.py +++ b/docs/site/scripts/generate-llms.py @@ -321,6 +321,15 @@ def synthesize_landing(raw_body): Card hrefs that start with `/` are rewritten to absolute BASE_URL form so bullets remain useful when the body is read out of context. """ + # Count first, parse second. Deciding "this is not a card grid" on an empty + # parse result would make total parser failure indistinguishable from an + # ordinary prose page: the caller would fall back to strip_mdx and the guard + # below would never run — silently degrading exactly the case it exists to + # catch. + expected = raw_body.count('lp-card-title') + if not expected: + return None + cards = [] for link in _LANDING_LINK_RE.finditer(raw_body): href, block = link.group(1), link.group(2) @@ -330,13 +339,10 @@ def synthesize_landing(raw_body): continue cards.append((href, _card_text(title.group(1)), _card_text(desc.group(1)))) - if not cards: - return None # A card that parses to nothing is invisible in the output, and `--check` # only compares generated output against committed output — so a systematic # parser regression stays green in CI while the corpus quietly loses pages. - expected = raw_body.count('lp-card-title') if len(cards) != expected: raise SystemExit( f"synthesize_landing dropped cards: parsed {len(cards)} " diff --git a/docs/site/scripts/test_generate_llms.py b/docs/site/scripts/test_generate_llms.py index b7cce696977..db31798a339 100644 --- a/docs/site/scripts/test_generate_llms.py +++ b/docs/site/scripts/test_generate_llms.py @@ -221,6 +221,26 @@ def test_dropped_card_raises_instead_of_silently_shrinking(self): g.synthesize_landing(body) self.assertIn("dropped cards", str(ctx.exception)) + def test_all_cards_malformed_raises_rather_than_falling_back(self): + """Total parser failure must not look like an ordinary prose page. + + Returning None here would send the caller to strip_mdx, degrading the + page silently — the exact outcome the guard exists to prevent. + """ + body = """ +
+ +
A
+ + +
B
+ +
+""" + with self.assertRaises(SystemExit) as ctx: + g.synthesize_landing(body) + self.assertIn("parsed 0 of 2", str(ctx.exception)) + class LeadingH1StripTests(unittest.TestCase): """The build() body-prep step strips a leading H1 to avoid duplicate headings.""" diff --git a/docs/site/static/llms-full.txt b/docs/site/static/llms-full.txt index 0f1ec24c325..da5548b747c 100644 --- a/docs/site/static/llms-full.txt +++ b/docs/site/static/llms-full.txt @@ -3987,16 +3987,18 @@ MCP connects a model to your **running node**. To let a model answer questions a | File | Contents | Use it when | |------|----------|-------------| | [llms.txt](https://docs.erigon.tech/llms.txt) | Page index with titles and descriptions | The model fetches pages on demand, or its context window is small | -| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | Every documentation page, in full, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | +| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of every documentation page, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale. +The corpus is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read it as the documentation's prose, not as a byte-for-byte mirror of the rendered site. + ```bash # Give a local model the whole documentation, with no network access curl -sO https://docs.erigon.tech/llms-full.txt ``` -The full file is around 420 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. +The full file is around 430 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. --- diff --git a/llms-full.txt b/llms-full.txt index 0f1ec24c325..da5548b747c 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -3987,16 +3987,18 @@ MCP connects a model to your **running node**. To let a model answer questions a | File | Contents | Use it when | |------|----------|-------------| | [llms.txt](https://docs.erigon.tech/llms.txt) | Page index with titles and descriptions | The model fetches pages on demand, or its context window is small | -| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | Every documentation page, in full, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | +| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of every documentation page, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale. +The corpus is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read it as the documentation's prose, not as a byte-for-byte mirror of the rendered site. + ```bash # Give a local model the whole documentation, with no network access curl -sO https://docs.erigon.tech/llms-full.txt ``` -The full file is around 420 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. +The full file is around 430 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. --- From a97ed68440b0e34e5ca564d7ea44f922bc2cfd49 Mon Sep 17 00:00:00 2001 From: Bloxster Date: Mon, 17 Aug 2026 16:07:20 +0200 Subject: [PATCH 4/4] =?UTF-8?q?docs(site):=20second=20review=20round=20?= =?UTF-8?q?=E2=80=94=20independent=20card=20count,=20index=20route,=20scop?= =?UTF-8?q?e?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up review from @yperbasis on #23336. Three findings, all confirmed by reproduction before fixing. 1. The guard counted the marker it was validating. `expected` came from `lp-card-title`, so if a card lost or renamed that marker the count shrank in step with the loss it was meant to detect: verified that a two-card grid with one renamed marker emitted one bullet and raised nothing. If every marker changed, `expected` hit zero and the page fell back to strip_mdx. Count `lp-card` containers instead — the wrapper is not consumed by the parse, so the two signals stay independent. Attributes are now matched order-independently and `to=` is read separately, which the container match no longer pins down. 2. llms.txt had no route to llms-full.txt. Neither committed index contained the string at all, so once the llms-full head tag was removed, an agent following rel="describedby" reached an index with no way to find the full corpus. The generator now emits that link, and both copies are regenerated. 3. "Every documentation page" was still wrong. SECTIONS scans only docs/ and help-center/, while the site also publishes v3.3 and v3.4 from versioned_docs/ — neither artifact contains those URLs. Say current documentation, and state the exclusion outright. Two new tests: a renamed title marker must raise rather than be absorbed (verified to fail against the previous count), and card attributes must parse in either order. --- docs/site/docs/fundamentals/mcp.mdx | 6 +-- .../docs/get-started/why-using-erigon.mdx | 2 +- docs/site/scripts/generate-llms.py | 34 ++++++++++++----- docs/site/scripts/test_generate_llms.py | 37 +++++++++++++++++++ docs/site/static/llms-full.txt | 6 +-- docs/site/static/llms.txt | 2 + llms-full.txt | 6 +-- llms.txt | 2 + 8 files changed, 75 insertions(+), 20 deletions(-) diff --git a/docs/site/docs/fundamentals/mcp.mdx b/docs/site/docs/fundamentals/mcp.mdx index b9757fb4cc1..6e010349613 100644 --- a/docs/site/docs/fundamentals/mcp.mdx +++ b/docs/site/docs/fundamentals/mcp.mdx @@ -335,12 +335,12 @@ MCP connects a model to your **running node**. To let a model answer questions a | File | Contents | Use it when | |------|----------|-------------| -| [llms.txt](https://docs.erigon.tech/llms.txt) | Page index with titles and descriptions | The model fetches pages on demand, or its context window is small | -| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of every documentation page, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | +| [llms.txt](https://docs.erigon.tech/llms.txt) | Index of the current documentation, with titles and descriptions | The model fetches pages on demand, or its context window is small | +| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of those same pages, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale. -The corpus is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read it as the documentation's prose, not as a byte-for-byte mirror of the rendered site. +Two things to know about their scope. They cover the **current** documentation and the help center only — the archived versions published under `/v3.3/` and `/v3.4/` are not included. And the text is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read the corpus as the documentation's prose, not as a byte-for-byte mirror of the rendered site. ```bash # Give a local model the whole documentation, with no network access diff --git a/docs/site/docs/get-started/why-using-erigon.mdx b/docs/site/docs/get-started/why-using-erigon.mdx index 66b91ca6c14..e2551825c83 100644 --- a/docs/site/docs/get-started/why-using-erigon.mdx +++ b/docs/site/docs/get-started/why-using-erigon.mdx @@ -180,4 +180,4 @@ The interface is **read-only** and localhost-bound — no risk of unintended sta claude mcp add --transport sse erigon http://127.0.0.1:8553/sse ``` -Where MCP lets a model query your *node*, [llms.txt and llms-full.txt](../fundamentals/mcp#these-docs-as-a-single-file) let it read these *docs*: an index of every page, and the documentation text as one plain-text file for offline use. +Where MCP lets a model query your *node*, [llms.txt and llms-full.txt](../fundamentals/mcp#these-docs-as-a-single-file) let it read these *docs*: an index of the current documentation, and the same pages' text as one plain-text file for offline use. diff --git a/docs/site/scripts/generate-llms.py b/docs/site/scripts/generate-llms.py index cc028272ea8..d2e6b970059 100644 --- a/docs/site/scripts/generate-llms.py +++ b/docs/site/scripts/generate-llms.py @@ -291,10 +291,15 @@ def strip_mdx(text): # ``) fails to match locally, and the engine then scans forward and pairs # the title with the *next* card's description — silently swallowing the card in # between. Matching each block first makes a card boundary unrepresentable. -_LANDING_LINK_RE = re.compile( - r']*\bto=[\'"]([^\'"]+)[\'"][^>]*>(.*?)', - re.DOTALL, -) +# +# The container match is also what the guard below counts. It deliberately keys +# on `lp-card` — the wrapper — and not on any marker the parse itself consumes: +# a count taken from `lp-card-title` would shrink in step with the very loss it +# is supposed to detect, so a renamed title marker would drop its card in +# silence. Attributes are matched order-independently; `to=` is read separately. +_LANDING_CARD_LINK_RE = re.compile( + r']*\blp-card\b[^>]*)>(.*?)', re.DOTALL) +_LINK_TO_RE = re.compile(r'\bto=[\'"]([^\'"]+)[\'"]') _CARD_TITLE_RE = re.compile( r']*\blp-card-title\b[^>]*>(.*?)', re.DOTALL) _CARD_DESC_RE = re.compile( @@ -326,18 +331,19 @@ def synthesize_landing(raw_body): # ordinary prose page: the caller would fall back to strip_mdx and the guard # below would never run — silently degrading exactly the case it exists to # catch. - expected = raw_body.count('lp-card-title') + containers = _LANDING_CARD_LINK_RE.findall(raw_body) + expected = len(containers) if not expected: return None cards = [] - for link in _LANDING_LINK_RE.finditer(raw_body): - href, block = link.group(1), link.group(2) + for attrs, block in containers: + href = _LINK_TO_RE.search(attrs) title = _CARD_TITLE_RE.search(block) desc = _CARD_DESC_RE.search(block) - if not title or not desc: + if not href or not title or not desc: continue - cards.append((href, _card_text(title.group(1)), + cards.append((href.group(1), _card_text(title.group(1)), _card_text(desc.group(1)))) # A card that parses to nothing is invisible in the output, and `--check` @@ -346,7 +352,7 @@ def synthesize_landing(raw_body): if len(cards) != expected: raise SystemExit( f"synthesize_landing dropped cards: parsed {len(cards)} " - f"of {expected} lp-card-title occurrences" + f"of {expected} lp-card containers" ) out = [] @@ -555,6 +561,14 @@ def display_section(p): lines.append("> modularity, and minimal disk footprint. It features an integrated consensus layer") lines.append("> (Caplin), BitTorrent-based historical data distribution, and fast node synchronization.") lines.append("") + # The only machine-readable route from this index to the full corpus. Pages + # advertise llms.txt via , not llms-full.txt, so an + # agent that arrives here would otherwise have no way to reach it. + lines.append( + f"The full text of every page listed below is also available as a single " + f"file: [llms-full.txt]({BASE_URL}/llms-full.txt)" + ) + lines.append("") current_section = None for p in all_pages: diff --git a/docs/site/scripts/test_generate_llms.py b/docs/site/scripts/test_generate_llms.py index db31798a339..f49ca8b8a35 100644 --- a/docs/site/scripts/test_generate_llms.py +++ b/docs/site/scripts/test_generate_llms.py @@ -221,6 +221,43 @@ def test_dropped_card_raises_instead_of_silently_shrinking(self): g.synthesize_landing(body) self.assertIn("dropped cards", str(ctx.exception)) + def test_renamed_title_marker_is_caught_not_absorbed(self): + """The guard must not count the marker it validates. + + Counting `lp-card-title` would shrink in step with the very loss it is + meant to detect, so a renamed marker would drop its card in silence. + Counting `lp-card` containers keeps the two signals independent. + """ + body = """ +
+ +
A
+
Desc A.
+ + +
B
+
Desc B.
+ +
+""" + with self.assertRaises(SystemExit) as ctx: + g.synthesize_landing(body) + self.assertIn("parsed 1 of 2", str(ctx.exception)) + + def test_card_attribute_order_does_not_matter(self): + """`to=` may precede or follow the class; both must parse.""" + body = """ +
+ +
A
+
Desc A.
+ +
+""" + out = g.synthesize_landing(body) + self.assertIsNotNone(out) + self.assertIn("[A](https://docs.erigon.tech/a): Desc A.", out) + def test_all_cards_malformed_raises_rather_than_falling_back(self): """Total parser failure must not look like an ordinary prose page. diff --git a/docs/site/static/llms-full.txt b/docs/site/static/llms-full.txt index da5548b747c..6aadeb4593d 100644 --- a/docs/site/static/llms-full.txt +++ b/docs/site/static/llms-full.txt @@ -3986,12 +3986,12 @@ MCP connects a model to your **running node**. To let a model answer questions a | File | Contents | Use it when | |------|----------|-------------| -| [llms.txt](https://docs.erigon.tech/llms.txt) | Page index with titles and descriptions | The model fetches pages on demand, or its context window is small | -| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of every documentation page, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | +| [llms.txt](https://docs.erigon.tech/llms.txt) | Index of the current documentation, with titles and descriptions | The model fetches pages on demand, or its context window is small | +| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of those same pages, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale. -The corpus is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read it as the documentation's prose, not as a byte-for-byte mirror of the rendered site. +Two things to know about their scope. They cover the **current** documentation and the help center only — the archived versions published under `/v3.3/` and `/v3.4/` are not included. And the text is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read the corpus as the documentation's prose, not as a byte-for-byte mirror of the rendered site. ```bash # Give a local model the whole documentation, with no network access diff --git a/docs/site/static/llms.txt b/docs/site/static/llms.txt index d7b26ab9446..6e5927f3913 100644 --- a/docs/site/static/llms.txt +++ b/docs/site/static/llms.txt @@ -4,6 +4,8 @@ > modularity, and minimal disk footprint. It features an integrated consensus layer > (Caplin), BitTorrent-based historical data distribution, and fast node synchronization. +The full text of every page listed below is also available as a single file: [llms-full.txt](https://docs.erigon.tech/llms-full.txt) + - [Introduction](https://docs.erigon.tech/): Official documentation for Erigon — the efficient, modular Ethereum execution client. Get started, explore the JSON-RPC API, and learn how to run a full or archive node. ## Get Started diff --git a/llms-full.txt b/llms-full.txt index da5548b747c..6aadeb4593d 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -3986,12 +3986,12 @@ MCP connects a model to your **running node**. To let a model answer questions a | File | Contents | Use it when | |------|----------|-------------| -| [llms.txt](https://docs.erigon.tech/llms.txt) | Page index with titles and descriptions | The model fetches pages on demand, or its context window is small | -| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of every documentation page, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | +| [llms.txt](https://docs.erigon.tech/llms.txt) | Index of the current documentation, with titles and descriptions | The model fetches pages on demand, or its context window is small | +| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of those same pages, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale. -The corpus is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read it as the documentation's prose, not as a byte-for-byte mirror of the rendered site. +Two things to know about their scope. They cover the **current** documentation and the help center only — the archived versions published under `/v3.3/` and `/v3.4/` are not included. And the text is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read the corpus as the documentation's prose, not as a byte-for-byte mirror of the rendered site. ```bash # Give a local model the whole documentation, with no network access diff --git a/llms.txt b/llms.txt index d7b26ab9446..6e5927f3913 100644 --- a/llms.txt +++ b/llms.txt @@ -4,6 +4,8 @@ > modularity, and minimal disk footprint. It features an integrated consensus layer > (Caplin), BitTorrent-based historical data distribution, and fast node synchronization. +The full text of every page listed below is also available as a single file: [llms-full.txt](https://docs.erigon.tech/llms-full.txt) + - [Introduction](https://docs.erigon.tech/): Official documentation for Erigon — the efficient, modular Ethereum execution client. Get started, explore the JSON-RPC API, and learn how to run a full or archive node. ## Get Started