docs(site): address review on #23335 — describedby relation, guard hole, corpus claims

docs/site/docs/fundamentals/mcp.mdx

@@ -316,14 +316,16 @@ MCP connects a model to your **running node**. To let a model answer questions a
 
 | File | Contents | Use it when |
 |------|----------|-------------|
-| [llms.txt](https://docs.erigon.tech/llms.txt) | Page index with titles and descriptions | The model fetches pages on demand, or its context window is small |
-| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | Every documentation page, in full, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped |
+| [llms.txt](https://docs.erigon.tech/llms.txt) | Index of the current documentation, with titles and descriptions | The model fetches pages on demand, or its context window is small |
+| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of those same pages, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped |
 
 Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale.
 
+Two things to know about their scope. They cover the **current** documentation and the help center only — the archived versions published under `/v3.3/` and `/v3.4/` are not included. And the text is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read the corpus as the documentation's prose, not as a byte-for-byte mirror of the rendered site.
+
 ```bash
 # Give a local model the whole documentation, with no network access
 curl -sO https://docs.erigon.tech/llms-full.txt
 ```
 
-The full file is around 420 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading.
+The full file is around 430 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading.

docs/site/docs/get-started/why-using-erigon.mdx

@@ -180,4 +180,4 @@ The interface is **read-only** and localhost-bound — no risk of unintended sta
 claude mcp add --transport sse erigon http://127.0.0.1:8553/sse
 ```
 
-Where MCP lets a model query your *node*, [llms.txt and llms-full.txt](../fundamentals/mcp#these-docs-as-a-single-file) let it read these *docs* — the whole documentation as one plain-text file, for offline use.
+Where MCP lets a model query your *node*, [llms.txt and llms-full.txt](../fundamentals/mcp#these-docs-as-a-single-file) let it read these *docs*: an index of the current documentation, and the same pages' text as one plain-text file for offline use.

docs/site/docusaurus.config.ts

@@ -86,27 +86,21 @@ export default async function createConfig(): Promise<Config> {
         attributes: {},
         innerHTML: 'window.plausible=window.plausible||function(){(plausible.q=plausible.q||[]).push(arguments)},plausible.init=plausible.init||function(i){plausible.o=i||{}};plausible.init()',
       },
-      // Machine-readable copies of this site (llmstxt.org convention). Advertised
-      // on every page so crawlers and agents can find them without guessing the
-      // path — they are static files, so nothing else links to them.
+      // Point every page at the llms.txt index that covers it. `describedby` is
+      // the relation the llmstxt.org proposal defines for exactly this; its
+      // `alternate` + `text/markdown` pairing is reserved for a *per-page*
+      // Markdown representation, which this site-wide index is not.
+      // llms-full.txt is deliberately not advertised here: it is not a
+      // description of any single page. It stays discoverable through llms.txt,
+      // the sitemap, and the MCP docs page.
       {
         tagName: 'link',
         attributes: {
-          rel: 'alternate',
-          type: 'text/plain',
+          rel: 'describedby',
           href: 'https://docs.erigon.tech/llms.txt',
           title: 'Erigon Documentation — page index for LLMs',
         },
       },
-      {
-        tagName: 'link',
-        attributes: {
-          rel: 'alternate',
-          type: 'text/plain',
-          href: 'https://docs.erigon.tech/llms-full.txt',
-          title: 'Erigon Documentation — full text for LLMs',
-        },
-      },
     ],
 
     plugins: [
@@ -172,9 +166,9 @@ export default async function createConfig(): Promise<Config> {
           priority: 0.5,
           ignorePatterns: ['/search'],
           // The llms.txt artifacts live in static/, so Docusaurus never routes
-          // them and the default sitemap omits them. Nothing else on the web
-          // links to them either, which leaves them unindexable and unreachable
-          // by search — append them explicitly.
+          // them and the default sitemap omits them — which leaves them
+          // unindexable by search. Append them explicitly. This is also how
+          // llms-full.txt stays reachable, since it is not advertised in head.
           createSitemapItems: async ({defaultCreateSitemapItems, ...rest}) => [
             ...(await defaultCreateSitemapItems(rest)),
             {url: 'https://docs.erigon.tech/llms.txt', changefreq: 'weekly', priority: 0.5},

docs/site/scripts/generate-llms.py

@@ -291,10 +291,15 @@ def strip_mdx(text):
 # `<code>`) fails to match locally, and the engine then scans forward and pairs
 # the title with the *next* card's description — silently swallowing the card in
 # between. Matching each <Link> block first makes a card boundary unrepresentable.
-_LANDING_LINK_RE = re.compile(
-    r'<Link\s[^>]*\bto=[\'"]([^\'"]+)[\'"][^>]*>(.*?)</Link>',
-    re.DOTALL,
-)
+#
+# The container match is also what the guard below counts. It deliberately keys
+# on `lp-card` — the wrapper — and not on any marker the parse itself consumes:
+# a count taken from `lp-card-title` would shrink in step with the very loss it
+# is supposed to detect, so a renamed title marker would drop its card in
+# silence. Attributes are matched order-independently; `to=` is read separately.
+_LANDING_CARD_LINK_RE = re.compile(
+    r'<Link\s([^>]*\blp-card\b[^>]*)>(.*?)</Link>', re.DOTALL)
+_LINK_TO_RE = re.compile(r'\bto=[\'"]([^\'"]+)[\'"]')
 _CARD_TITLE_RE = re.compile(
     r'<div[^>]*\blp-card-title\b[^>]*>(.*?)</div>', re.DOTALL)
 _CARD_DESC_RE = re.compile(
@@ -321,26 +326,33 @@ def synthesize_landing(raw_body):
     Card hrefs that start with `/` are rewritten to absolute BASE_URL form so
     bullets remain useful when the body is read out of context.
     """
+    # Count first, parse second. Deciding "this is not a card grid" on an empty
+    # parse result would make total parser failure indistinguishable from an
+    # ordinary prose page: the caller would fall back to strip_mdx and the guard
+    # below would never run — silently degrading exactly the case it exists to
+    # catch.
+    containers = _LANDING_CARD_LINK_RE.findall(raw_body)
+    expected = len(containers)
+    if not expected:
+        return None
+
     cards = []
-    for link in _LANDING_LINK_RE.finditer(raw_body):
-        href, block = link.group(1), link.group(2)
+    for attrs, block in containers:
+        href = _LINK_TO_RE.search(attrs)
         title = _CARD_TITLE_RE.search(block)
         desc = _CARD_DESC_RE.search(block)
-        if not title or not desc:
+        if not href or not title or not desc:
             continue
-        cards.append((href, _card_text(title.group(1)),
+        cards.append((href.group(1), _card_text(title.group(1)),
                       _card_text(desc.group(1))))
-    if not cards:
-        return None
 
     # A card that parses to nothing is invisible in the output, and `--check`
     # only compares generated output against committed output — so a systematic
     # parser regression stays green in CI while the corpus quietly loses pages.
-    expected = raw_body.count('lp-card-title')
     if len(cards) != expected:
         raise SystemExit(
             f"synthesize_landing dropped cards: parsed {len(cards)} "
-            f"of {expected} lp-card-title occurrences"
+            f"of {expected} lp-card containers"
         )
 
     out = []
@@ -549,6 +561,14 @@ def display_section(p):
     lines.append("> modularity, and minimal disk footprint. It features an integrated consensus layer")
     lines.append("> (Caplin), BitTorrent-based historical data distribution, and fast node synchronization.")
     lines.append("")
+    # The only machine-readable route from this index to the full corpus. Pages
+    # advertise llms.txt via <link rel="describedby">, not llms-full.txt, so an
+    # agent that arrives here would otherwise have no way to reach it.
+    lines.append(
+        f"The full text of every page listed below is also available as a single "
+        f"file: [llms-full.txt]({BASE_URL}/llms-full.txt)"
+    )
+    lines.append("")
 
     current_section = None
     for p in all_pages:

docs/site/scripts/test_generate_llms.py

@@ -221,6 +221,63 @@ def test_dropped_card_raises_instead_of_silently_shrinking(self):
             g.synthesize_landing(body)
         self.assertIn("dropped cards", str(ctx.exception))
 
+    def test_renamed_title_marker_is_caught_not_absorbed(self):
+        """The guard must not count the marker it validates.
+
+        Counting `lp-card-title` would shrink in step with the very loss it is
+        meant to detect, so a renamed marker would drop its card in silence.
+        Counting `lp-card` containers keeps the two signals independent.
+        """
+        body = """
+<div className="lp-grid">
+<Link className="lp-card" to="/a/">
+  <div className="lp-card-title">A</div>
+  <div className="lp-card-desc">Desc A.</div>
+</Link>
+<Link className="lp-card" to="/b/">
+  <div className="lp-card-heading">B</div>
+  <div className="lp-card-desc">Desc B.</div>
+</Link>
+</div>
+"""
+        with self.assertRaises(SystemExit) as ctx:
+            g.synthesize_landing(body)
+        self.assertIn("parsed 1 of 2", str(ctx.exception))
+
+    def test_card_attribute_order_does_not_matter(self):
+        """`to=` may precede or follow the class; both must parse."""
+        body = """
+<div className="lp-grid">
+<Link to="/a/" className="lp-card">
+  <div className="lp-card-title">A</div>
+  <div className="lp-card-desc">Desc A.</div>
+</Link>
+</div>
+"""
+        out = g.synthesize_landing(body)
+        self.assertIsNotNone(out)
+        self.assertIn("[A](https://docs.erigon.tech/a): Desc A.", out)
+
+    def test_all_cards_malformed_raises_rather_than_falling_back(self):
+        """Total parser failure must not look like an ordinary prose page.
+
+        Returning None here would send the caller to strip_mdx, degrading the
+        page silently — the exact outcome the guard exists to prevent.
+        """
+        body = """
+<div className="lp-grid">
+<Link className="lp-card" to="/a/">
+  <div className="lp-card-title">A</div>
+</Link>
+<Link className="lp-card" to="/b/">
+  <div className="lp-card-title">B</div>
+</Link>
+</div>
+"""
+        with self.assertRaises(SystemExit) as ctx:
+            g.synthesize_landing(body)
+        self.assertIn("parsed 0 of 2", str(ctx.exception))
+
 
 class LeadingH1StripTests(unittest.TestCase):
     """The build() body-prep step strips a leading H1 to avoid duplicate headings."""

docs/site/static/llms-full.txt

@@ -3929,17 +3929,19 @@ MCP connects a model to your **running node**. To let a model answer questions a
 
 | File | Contents | Use it when |
 |------|----------|-------------|
-| [llms.txt](https://docs.erigon.tech/llms.txt) | Page index with titles and descriptions | The model fetches pages on demand, or its context window is small |
-| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | Every documentation page, in full, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped |
+| [llms.txt](https://docs.erigon.tech/llms.txt) | Index of the current documentation, with titles and descriptions | The model fetches pages on demand, or its context window is small |
+| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of those same pages, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped |
 
 Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale.
 
+Two things to know about their scope. They cover the **current** documentation and the help center only — the archived versions published under `/v3.3/` and `/v3.4/` are not included. And the text is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read the corpus as the documentation's prose, not as a byte-for-byte mirror of the rendered site.
+
 ```bash
 # Give a local model the whole documentation, with no network access
 curl -sO https://docs.erigon.tech/llms-full.txt
 ```
 
-The full file is around 420 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading.
+The full file is around 430 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading.
 
 ---
 

docs/site/static/llms.txt

@@ -4,6 +4,8 @@
 > modularity, and minimal disk footprint. It features an integrated consensus layer
 > (Caplin), BitTorrent-based historical data distribution, and fast node synchronization.
 
+The full text of every page listed below is also available as a single file: [llms-full.txt](https://docs.erigon.tech/llms-full.txt)
+
 - [Introduction](https://docs.erigon.tech/): Official documentation for Erigon — the efficient, modular Ethereum execution client. Get started, explore the JSON-RPC API, and learn how to run a full or archive node.
 
 ## Get Started

llms-full.txt

@@ -3929,17 +3929,19 @@ MCP connects a model to your **running node**. To let a model answer questions a
 
 | File | Contents | Use it when |
 |------|----------|-------------|
-| [llms.txt](https://docs.erigon.tech/llms.txt) | Page index with titles and descriptions | The model fetches pages on demand, or its context window is small |
-| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | Every documentation page, in full, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped |
+| [llms.txt](https://docs.erigon.tech/llms.txt) | Index of the current documentation, with titles and descriptions | The model fetches pages on demand, or its context window is small |
+| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of those same pages, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped |
 
 Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale.
 
+Two things to know about their scope. They cover the **current** documentation and the help center only — the archived versions published under `/v3.3/` and `/v3.4/` are not included. And the text is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read the corpus as the documentation's prose, not as a byte-for-byte mirror of the rendered site.
+
 ```bash
 # Give a local model the whole documentation, with no network access
 curl -sO https://docs.erigon.tech/llms-full.txt
 ```
 
-The full file is around 420 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading.
+The full file is around 430 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading.
 
 ---
 

llms.txt

@@ -4,6 +4,8 @@
 > modularity, and minimal disk footprint. It features an integrated consensus layer
 > (Caplin), BitTorrent-based historical data distribution, and fast node synchronization.
 
+The full text of every page listed below is also available as a single file: [llms-full.txt](https://docs.erigon.tech/llms-full.txt)
+
 - [Introduction](https://docs.erigon.tech/): Official documentation for Erigon — the efficient, modular Ethereum execution client. Get started, explore the JSON-RPC API, and learn how to run a full or archive node.
 
 ## Get Started