docs(site): make the llms.txt artifacts discoverable, and fix the landing-card parser

docs/site/docs/fundamentals/mcp.mdx

@@ -326,3 +326,25 @@ In addition to tools, the MCP server exposes structured **resources** and **prom
 | `debug_logs` | Triage node issues from recent log output |
 | `torrent_status` | Snapshot download health check |
 | `sync_analysis` | Node sync progress and performance |
+
+---
+
+## These Docs as a Single File
+
+MCP connects a model to your **running node**. To let a model answer questions about **Erigon itself** — flags, pruning modes, RPC namespaces, hardware sizing — point it at this documentation in machine-readable form, following the [llmstxt.org](https://llmstxt.org) convention:
+
+| File | Contents | Use it when |
+|------|----------|-------------|
+| [llms.txt](https://docs.erigon.tech/llms.txt) | Index of the current documentation, with titles and descriptions | The model fetches pages on demand, or its context window is small |
+| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of those same pages, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped |
+
+Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale.
+
+Two things to know about their scope. They cover the **current** documentation and the help center only — the archived versions published under `/v3.3/` and `/v3.4/` are not included. And the text is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read the corpus as the documentation's prose, not as a byte-for-byte mirror of the rendered site.
+
+```bash
+# Give a local model the whole documentation, with no network access
+curl -sO https://docs.erigon.tech/llms-full.txt
+```
+
+The full file is around 430 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading.

docs/site/docs/get-started/why-using-erigon.mdx

@@ -179,3 +179,5 @@ The interface is **read-only** and localhost-bound — no risk of unintended sta
 # Register with Claude Code in one command
 claude mcp add --transport sse erigon http://127.0.0.1:8553/sse
 ```
+
+Where MCP lets a model query your *node*, [llms.txt and llms-full.txt](../fundamentals/mcp#these-docs-as-a-single-file) let it read these *docs*: an index of the current documentation, and the same pages' text as one plain-text file for offline use.

docs/site/docusaurus.config.ts

@@ -86,6 +86,21 @@ export default async function createConfig(): Promise<Config> {
         attributes: {},
         innerHTML: 'window.plausible=window.plausible||function(){(plausible.q=plausible.q||[]).push(arguments)},plausible.init=plausible.init||function(i){plausible.o=i||{}};plausible.init()',
       },
+      // Point every page at the llms.txt index that covers it. `describedby` is
+      // the relation the llmstxt.org proposal defines for exactly this; its
+      // `alternate` + `text/markdown` pairing is reserved for a *per-page*
+      // Markdown representation, which this site-wide index is not.
+      // llms-full.txt is deliberately not advertised here: it is not a
+      // description of any single page. It stays discoverable through llms.txt,
+      // the sitemap, and the MCP docs page.
+      {
+        tagName: 'link',
+        attributes: {
+          rel: 'describedby',
+          href: 'https://docs.erigon.tech/llms.txt',
+          title: 'Erigon Documentation — page index for LLMs',
+        },
+      },
     ],
 
     plugins: [
@@ -150,6 +165,15 @@ export default async function createConfig(): Promise<Config> {
           changefreq: 'weekly',
           priority: 0.5,
           ignorePatterns: ['/search'],
+          // The llms.txt artifacts live in static/, so Docusaurus never routes
+          // them and the default sitemap omits them — which leaves them
+          // unindexable by search. Append them explicitly. This is also how
+          // llms-full.txt stays reachable, since it is not advertised in head.
+          createSitemapItems: async ({defaultCreateSitemapItems, ...rest}) => [
+            ...(await defaultCreateSitemapItems(rest)),
+            {url: 'https://docs.erigon.tech/llms.txt', changefreq: 'weekly', priority: 0.5},
+            {url: 'https://docs.erigon.tech/llms-full.txt', changefreq: 'weekly', priority: 0.5},
+          ],
         },
       } satisfies Preset.Options],
     ],

docs/site/scripts/generate-llms.py

@@ -285,16 +285,30 @@ def strip_mdx(text):
 # index pages. After MDX stripping, they collapse to a structureless pile of
 # title/desc fragments. We detect the lp-card pattern and synthesize a clean
 # bullet list instead, keeping link structure intact for LLM consumption.
-_LANDING_CARD_RE = re.compile(
-    r'<Link\s[^>]*\bto=[\'"]([^\'"]+)[\'"][^>]*>'
-    r'(?:.*?)'
-    r'<div[^>]*\blp-card-title\b[^>]*>([^<]+)</div>'
-    r'(?:.*?)'
-    r'<div[^>]*\blp-card-desc\b[^>]*>([^<]+)</div>'
-    r'(?:.*?)'
-    r'</Link>',
-    re.DOTALL,
-)
+# Parsed in two stages, deliberately. A single pattern spanning the whole card
+# cannot express "and never cross into the next card": with `[^<]+` for the text
+# and `.*?` for the gaps, a description containing inline markup (`<strong>`,
+# `<code>`) fails to match locally, and the engine then scans forward and pairs
+# the title with the *next* card's description — silently swallowing the card in
+# between. Matching each <Link> block first makes a card boundary unrepresentable.
+#
+# The container match is also what the guard below counts. It deliberately keys
+# on `lp-card` — the wrapper — and not on any marker the parse itself consumes:
+# a count taken from `lp-card-title` would shrink in step with the very loss it
+# is supposed to detect, so a renamed title marker would drop its card in
+# silence. Attributes are matched order-independently; `to=` is read separately.
+_LANDING_CARD_LINK_RE = re.compile(
+    r'<Link\s([^>]*\blp-card\b[^>]*)>(.*?)</Link>', re.DOTALL)
+_LINK_TO_RE = re.compile(r'\bto=[\'"]([^\'"]+)[\'"]')
+_CARD_TITLE_RE = re.compile(
+    r'<div[^>]*\blp-card-title\b[^>]*>(.*?)</div>', re.DOTALL)
+_CARD_DESC_RE = re.compile(
+    r'<div[^>]*\blp-card-desc\b[^>]*>(.*?)</div>', re.DOTALL)
+
+
+def _card_text(fragment):
+    """Flatten a card's inner HTML to plain text, collapsing whitespace."""
+    return re.sub(r'\s+', ' ', re.sub(r'<[^>]+>', '', fragment)).strip()
 _LANDING_HERO_RE = re.compile(
     r'<div[^>]*\blp-hero\b[^>]*>'
     r'(?:.*?)'
@@ -312,10 +326,35 @@ def synthesize_landing(raw_body):
     Card hrefs that start with `/` are rewritten to absolute BASE_URL form so
     bullets remain useful when the body is read out of context.
     """
-    cards = list(_LANDING_CARD_RE.finditer(raw_body))
-    if not cards:
+    # Count first, parse second. Deciding "this is not a card grid" on an empty
+    # parse result would make total parser failure indistinguishable from an
+    # ordinary prose page: the caller would fall back to strip_mdx and the guard
+    # below would never run — silently degrading exactly the case it exists to
+    # catch.
+    containers = _LANDING_CARD_LINK_RE.findall(raw_body)
+    expected = len(containers)
+    if not expected:
         return None
 
+    cards = []
+    for attrs, block in containers:
+        href = _LINK_TO_RE.search(attrs)
+        title = _CARD_TITLE_RE.search(block)
+        desc = _CARD_DESC_RE.search(block)
+        if not href or not title or not desc:
+            continue
+        cards.append((href.group(1), _card_text(title.group(1)),
+                      _card_text(desc.group(1))))
+
+    # A card that parses to nothing is invisible in the output, and `--check`
+    # only compares generated output against committed output — so a systematic
+    # parser regression stays green in CI while the corpus quietly loses pages.
+    if len(cards) != expected:
+        raise SystemExit(
+            f"synthesize_landing dropped cards: parsed {len(cards)} "
+            f"of {expected} lp-card containers"
+        )
+
     out = []
     hero = _LANDING_HERO_RE.search(raw_body)
     if hero:
@@ -326,12 +365,10 @@ def synthesize_landing(raw_body):
 
     out.append("## Sections")
     out.append("")
-    for m in cards:
-        href = m.group(1).strip()
+    for href, title, desc in cards:
+        href = href.strip()
         if href.startswith("/"):
             href = BASE_URL + href.rstrip("/")
-        title = m.group(2).strip()
-        desc = m.group(3).strip()
         out.append(f"- [{title}]({href}): {desc}")
     return "\n".join(out)
 
@@ -524,6 +561,14 @@ def display_section(p):
     lines.append("> modularity, and minimal disk footprint. It features an integrated consensus layer")
     lines.append("> (Caplin), BitTorrent-based historical data distribution, and fast node synchronization.")
     lines.append("")
+    # The only machine-readable route from this index to the full corpus. Pages
+    # advertise llms.txt via <link rel="describedby">, not llms-full.txt, so an
+    # agent that arrives here would otherwise have no way to reach it.
+    lines.append(
+        f"The full text of every page listed below is also available as a single "
+        f"file: [llms-full.txt]({BASE_URL}/llms-full.txt)"
+    )
+    lines.append("")
 
     current_section = None
     for p in all_pages:

docs/site/scripts/test_generate_llms.py

@@ -159,6 +159,125 @@ def test_no_cards_returns_none(self):
         out = g.synthesize_landing("# Hello\n\nProse only, no cards.")
         self.assertIsNone(out)
 
+    def test_desc_with_inline_markup_is_extracted(self):
+        """Inline markup in a description must not defeat the match."""
+        body = """
+<div className="lp-grid">
+<Link className="lp-card" to="/pruning/">
+  <div className="lp-card-title">Flexible Pruning</div>
+  <div className="lp-card-desc"><code>--prune.mode=minimal</code> gives a <strong>smaller</strong> footprint.</div>
+</Link>
+</div>
+"""
+        out = g.synthesize_landing(body)
+        self.assertIsNotNone(out)
+        self.assertIn("--prune.mode=minimal gives a smaller footprint.", out)
+        self.assertNotIn("<code>", out)
+        self.assertNotIn("<strong>", out)
+
+    def test_markup_card_does_not_swallow_the_next_card(self):
+        """A card whose desc has markup must not steal the following card's desc."""
+        body = """
+<div className="lp-grid">
+<Link className="lp-card" to="/immutable/">
+  <div className="lp-card-title">Immutable Data</div>
+  <div className="lp-card-desc">Historical data lives in <strong>immutable</strong> files.</div>
+</Link>
+<Link className="lp-card" to="/staged-sync/">
+  <div className="lp-card-title">Staged Sync</div>
+  <div className="lp-card-desc">Splits processing into specialised stages.</div>
+</Link>
+</div>
+"""
+        out = g.synthesize_landing(body)
+        self.assertIsNotNone(out)
+        bullets = [ln for ln in out.split("\n") if ln.startswith("- [")]
+        self.assertEqual(len(bullets), 2)
+        self.assertIn(
+            "- [Immutable Data](https://docs.erigon.tech/immutable): "
+            "Historical data lives in immutable files.",
+            bullets,
+        )
+        self.assertIn(
+            "- [Staged Sync](https://docs.erigon.tech/staged-sync): "
+            "Splits processing into specialised stages.",
+            bullets,
+        )
+
+    def test_dropped_card_raises_instead_of_silently_shrinking(self):
+        """A card the parser cannot read must fail loudly — `--check` cannot see it."""
+        body = """
+<div className="lp-grid">
+<Link className="lp-card" to="/ok/">
+  <div className="lp-card-title">Fine</div>
+  <div className="lp-card-desc">Parses correctly.</div>
+</Link>
+<Link className="lp-card" to="/broken/">
+  <div className="lp-card-title">Missing Its Description</div>
+</Link>
+</div>
+"""
+        with self.assertRaises(SystemExit) as ctx:
+            g.synthesize_landing(body)
+        self.assertIn("dropped cards", str(ctx.exception))
+
+    def test_renamed_title_marker_is_caught_not_absorbed(self):
+        """The guard must not count the marker it validates.
+
+        Counting `lp-card-title` would shrink in step with the very loss it is
+        meant to detect, so a renamed marker would drop its card in silence.
+        Counting `lp-card` containers keeps the two signals independent.
+        """
+        body = """
+<div className="lp-grid">
+<Link className="lp-card" to="/a/">
+  <div className="lp-card-title">A</div>
+  <div className="lp-card-desc">Desc A.</div>
+</Link>
+<Link className="lp-card" to="/b/">
+  <div className="lp-card-heading">B</div>
+  <div className="lp-card-desc">Desc B.</div>
+</Link>
+</div>
+"""
+        with self.assertRaises(SystemExit) as ctx:
+            g.synthesize_landing(body)
+        self.assertIn("parsed 1 of 2", str(ctx.exception))
+
+    def test_card_attribute_order_does_not_matter(self):
+        """`to=` may precede or follow the class; both must parse."""
+        body = """
+<div className="lp-grid">
+<Link to="/a/" className="lp-card">
+  <div className="lp-card-title">A</div>
+  <div className="lp-card-desc">Desc A.</div>
+</Link>
+</div>
+"""
+        out = g.synthesize_landing(body)
+        self.assertIsNotNone(out)
+        self.assertIn("[A](https://docs.erigon.tech/a): Desc A.", out)
+
+    def test_all_cards_malformed_raises_rather_than_falling_back(self):
+        """Total parser failure must not look like an ordinary prose page.
+
+        Returning None here would send the caller to strip_mdx, degrading the
+        page silently — the exact outcome the guard exists to prevent.
+        """
+        body = """
+<div className="lp-grid">
+<Link className="lp-card" to="/a/">
+  <div className="lp-card-title">A</div>
+</Link>
+<Link className="lp-card" to="/b/">
+  <div className="lp-card-title">B</div>
+</Link>
+</div>
+"""
+        with self.assertRaises(SystemExit) as ctx:
+            g.synthesize_landing(body)
+        self.assertIn("parsed 0 of 2", str(ctx.exception))
+
 
 class LeadingH1StripTests(unittest.TestCase):
     """The build() body-prep step strips a leading H1 to avoid duplicate headings."""

docs/site/static/llms-full.txt

@@ -38,12 +38,15 @@ URL: https://docs.erigon.tech/get-started/why-using-erigon
 
 - [Integrated Consensus Layer (Caplin)](../fundamentals/caplin): Built-in consensus client — no need to run and manage separate software like Lighthouse. One binary handles both execution and consensus, simplifying setup for stakers and node runners.
 - [State Storage (Flat DB)](../fundamentals/optimizing-storage): Replaces the Merkle Patricia Trie with a flat key-value database. Faster reads and writes, and a smaller on-disk footprint for any node type.
-- [Immutable, Decentralised Data](../fundamentals/modules/downloader): Splits data processing into specialised stages, minimising random disk I/O and write amplification. Efficiently integrates large datasets into Erigon's flat DB.
+- [Immutable, Decentralised Data](../fundamentals/modules/downloader): Historical data lives in immutable files distributed via BitTorrent — up to 10× cheaper to backup, repair, and distribute. No double-disk usage, no complex serialisation.
+- [Staged Sync](../fundamentals/pruning-modes): Splits data processing into specialised stages, minimising random disk I/O and write amplification. Efficiently integrates large datasets into Erigon's flat DB.
 - [Predictable RPC Performance](../fundamentals/modules/rpc-daemon): No background compaction means no unpredictable resource spikes. RPC throughput stays stable and consistent — critical for providers and validators who cannot afford surprises.
 - [Modularity](../fundamentals/modules/): Core node, RPC Daemon, and TxPool run as independent processes with per-component resource limits. If one crashes the rest stay alive. Run multiple instances without downtime.
 - [OtterSync](../fundamentals/pruning-modes): Shifts initial sync from CPU-intensive transaction re-execution to network download — similar to BitTorrent for state data. Reduces sync time for both full and Archive nodes.
-- [Flexible Pruning](../fundamentals/optimizing-storage): Modular architecture and predictable performance handle high-volume traffic with low latency. Smaller database cuts hardware costs. Caplin reduces missed attestations and slashing risk.
+- [Flexible Pruning](../fundamentals/optimizing-storage): --prune.mode=minimal gives a smaller-than-full-node footprint that still satisfies all validator requirements. Right-size your storage without sacrificing functionality.
+- [RPC Providers & Large Stakers](../fundamentals/modules/rpc-daemon): Modular architecture and predictable performance handle high-volume traffic with low latency. Smaller database cuts hardware costs. Caplin reduces missed attestations and slashing risk.
 - [Home Users & Solo Stakers](../staking/): Run a full or archive node on a consumer SSD. Fast sync cuts setup from days to hours. No third-party servers — your node, your keys, your contribution to decentralisation.
+- [Developers](../interacting-with-erigon/): Full trace_ namespace, historical eth_getProof, historical blobs, and gRPC API. Modular components let you update, extend, or replace parts without rebuilding the whole stack.
 
 ---
 
@@ -3977,6 +3980,28 @@ In addition to tools, the MCP server exposes structured **resources** and **prom
 
 ---
 
+## These Docs as a Single File
+
+MCP connects a model to your **running node**. To let a model answer questions about **Erigon itself** — flags, pruning modes, RPC namespaces, hardware sizing — point it at this documentation in machine-readable form, following the [llmstxt.org](https://llmstxt.org) convention:
+
+| File | Contents | Use it when |
+|------|----------|-------------|
+| [llms.txt](https://docs.erigon.tech/llms.txt) | Index of the current documentation, with titles and descriptions | The model fetches pages on demand, or its context window is small |
+| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of those same pages, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped |
+
+Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale.
+
+Two things to know about their scope. They cover the **current** documentation and the help center only — the archived versions published under `/v3.3/` and `/v3.4/` are not included. And the text is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read the corpus as the documentation's prose, not as a byte-for-byte mirror of the rendered site.
+
+```bash
+# Give a local model the whole documentation, with no network access
+curl -sO https://docs.erigon.tech/llms-full.txt
+```
+
+The full file is around 430 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading.
+
+---
+
 # Otterscan
 URL: https://docs.erigon.tech/fundamentals/otterscan
 

docs/site/static/llms.txt

@@ -4,6 +4,8 @@
 > modularity, and minimal disk footprint. It features an integrated consensus layer
 > (Caplin), BitTorrent-based historical data distribution, and fast node synchronization.
 
+The full text of every page listed below is also available as a single file: [llms-full.txt](https://docs.erigon.tech/llms-full.txt)
+
 - [Introduction](https://docs.erigon.tech/): Official documentation for Erigon — the efficient, modular Ethereum execution client. Get started, explore the JSON-RPC API, and learn how to run a full or archive node.
 
 ## Get Started