diff --git a/docs/site/docs/fundamentals/mcp.mdx b/docs/site/docs/fundamentals/mcp.mdx index a31888e6122..198cbe5e396 100644 --- a/docs/site/docs/fundamentals/mcp.mdx +++ b/docs/site/docs/fundamentals/mcp.mdx @@ -316,14 +316,16 @@ MCP connects a model to your **running node**. To let a model answer questions a | File | Contents | Use it when | |------|----------|-------------| -| [llms.txt](https://docs.erigon.tech/llms.txt) | Page index with titles and descriptions | The model fetches pages on demand, or its context window is small | -| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | Every documentation page, in full, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | +| [llms.txt](https://docs.erigon.tech/llms.txt) | Index of the current documentation, with titles and descriptions | The model fetches pages on demand, or its context window is small | +| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of those same pages, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale. +Two things to know about their scope. They cover the **current** documentation and the help center only — the archived versions published under `/v3.3/` and `/v3.4/` are not included. And the text is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read the corpus as the documentation's prose, not as a byte-for-byte mirror of the rendered site. + ```bash # Give a local model the whole documentation, with no network access curl -sO https://docs.erigon.tech/llms-full.txt ``` -The full file is around 420 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. +The full file is around 430 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. diff --git a/docs/site/docs/get-started/why-using-erigon.mdx b/docs/site/docs/get-started/why-using-erigon.mdx index d0d61a4e09a..e2551825c83 100644 --- a/docs/site/docs/get-started/why-using-erigon.mdx +++ b/docs/site/docs/get-started/why-using-erigon.mdx @@ -180,4 +180,4 @@ The interface is **read-only** and localhost-bound — no risk of unintended sta claude mcp add --transport sse erigon http://127.0.0.1:8553/sse ``` -Where MCP lets a model query your *node*, [llms.txt and llms-full.txt](../fundamentals/mcp#these-docs-as-a-single-file) let it read these *docs* — the whole documentation as one plain-text file, for offline use. +Where MCP lets a model query your *node*, [llms.txt and llms-full.txt](../fundamentals/mcp#these-docs-as-a-single-file) let it read these *docs*: an index of the current documentation, and the same pages' text as one plain-text file for offline use. diff --git a/docs/site/docusaurus.config.ts b/docs/site/docusaurus.config.ts index d9623d9770b..67d82496de4 100644 --- a/docs/site/docusaurus.config.ts +++ b/docs/site/docusaurus.config.ts @@ -86,27 +86,21 @@ export default async function createConfig(): Promise { attributes: {}, innerHTML: 'window.plausible=window.plausible||function(){(plausible.q=plausible.q||[]).push(arguments)},plausible.init=plausible.init||function(i){plausible.o=i||{}};plausible.init()', }, - // Machine-readable copies of this site (llmstxt.org convention). Advertised - // on every page so crawlers and agents can find them without guessing the - // path — they are static files, so nothing else links to them. + // Point every page at the llms.txt index that covers it. `describedby` is + // the relation the llmstxt.org proposal defines for exactly this; its + // `alternate` + `text/markdown` pairing is reserved for a *per-page* + // Markdown representation, which this site-wide index is not. + // llms-full.txt is deliberately not advertised here: it is not a + // description of any single page. It stays discoverable through llms.txt, + // the sitemap, and the MCP docs page. { tagName: 'link', attributes: { - rel: 'alternate', - type: 'text/plain', + rel: 'describedby', href: 'https://docs.erigon.tech/llms.txt', title: 'Erigon Documentation — page index for LLMs', }, }, - { - tagName: 'link', - attributes: { - rel: 'alternate', - type: 'text/plain', - href: 'https://docs.erigon.tech/llms-full.txt', - title: 'Erigon Documentation — full text for LLMs', - }, - }, ], plugins: [ @@ -172,9 +166,9 @@ export default async function createConfig(): Promise { priority: 0.5, ignorePatterns: ['/search'], // The llms.txt artifacts live in static/, so Docusaurus never routes - // them and the default sitemap omits them. Nothing else on the web - // links to them either, which leaves them unindexable and unreachable - // by search — append them explicitly. + // them and the default sitemap omits them — which leaves them + // unindexable by search. Append them explicitly. This is also how + // llms-full.txt stays reachable, since it is not advertised in head. createSitemapItems: async ({defaultCreateSitemapItems, ...rest}) => [ ...(await defaultCreateSitemapItems(rest)), {url: 'https://docs.erigon.tech/llms.txt', changefreq: 'weekly', priority: 0.5}, diff --git a/docs/site/scripts/generate-llms.py b/docs/site/scripts/generate-llms.py index e7a115c78e9..d2e6b970059 100644 --- a/docs/site/scripts/generate-llms.py +++ b/docs/site/scripts/generate-llms.py @@ -291,10 +291,15 @@ def strip_mdx(text): # ``) fails to match locally, and the engine then scans forward and pairs # the title with the *next* card's description — silently swallowing the card in # between. Matching each block first makes a card boundary unrepresentable. -_LANDING_LINK_RE = re.compile( - r']*\bto=[\'"]([^\'"]+)[\'"][^>]*>(.*?)', - re.DOTALL, -) +# +# The container match is also what the guard below counts. It deliberately keys +# on `lp-card` — the wrapper — and not on any marker the parse itself consumes: +# a count taken from `lp-card-title` would shrink in step with the very loss it +# is supposed to detect, so a renamed title marker would drop its card in +# silence. Attributes are matched order-independently; `to=` is read separately. +_LANDING_CARD_LINK_RE = re.compile( + r']*\blp-card\b[^>]*)>(.*?)', re.DOTALL) +_LINK_TO_RE = re.compile(r'\bto=[\'"]([^\'"]+)[\'"]') _CARD_TITLE_RE = re.compile( r']*\blp-card-title\b[^>]*>(.*?)', re.DOTALL) _CARD_DESC_RE = re.compile( @@ -321,26 +326,33 @@ def synthesize_landing(raw_body): Card hrefs that start with `/` are rewritten to absolute BASE_URL form so bullets remain useful when the body is read out of context. """ + # Count first, parse second. Deciding "this is not a card grid" on an empty + # parse result would make total parser failure indistinguishable from an + # ordinary prose page: the caller would fall back to strip_mdx and the guard + # below would never run — silently degrading exactly the case it exists to + # catch. + containers = _LANDING_CARD_LINK_RE.findall(raw_body) + expected = len(containers) + if not expected: + return None + cards = [] - for link in _LANDING_LINK_RE.finditer(raw_body): - href, block = link.group(1), link.group(2) + for attrs, block in containers: + href = _LINK_TO_RE.search(attrs) title = _CARD_TITLE_RE.search(block) desc = _CARD_DESC_RE.search(block) - if not title or not desc: + if not href or not title or not desc: continue - cards.append((href, _card_text(title.group(1)), + cards.append((href.group(1), _card_text(title.group(1)), _card_text(desc.group(1)))) - if not cards: - return None # A card that parses to nothing is invisible in the output, and `--check` # only compares generated output against committed output — so a systematic # parser regression stays green in CI while the corpus quietly loses pages. - expected = raw_body.count('lp-card-title') if len(cards) != expected: raise SystemExit( f"synthesize_landing dropped cards: parsed {len(cards)} " - f"of {expected} lp-card-title occurrences" + f"of {expected} lp-card containers" ) out = [] @@ -549,6 +561,14 @@ def display_section(p): lines.append("> modularity, and minimal disk footprint. It features an integrated consensus layer") lines.append("> (Caplin), BitTorrent-based historical data distribution, and fast node synchronization.") lines.append("") + # The only machine-readable route from this index to the full corpus. Pages + # advertise llms.txt via , not llms-full.txt, so an + # agent that arrives here would otherwise have no way to reach it. + lines.append( + f"The full text of every page listed below is also available as a single " + f"file: [llms-full.txt]({BASE_URL}/llms-full.txt)" + ) + lines.append("") current_section = None for p in all_pages: diff --git a/docs/site/scripts/test_generate_llms.py b/docs/site/scripts/test_generate_llms.py index b7cce696977..f49ca8b8a35 100644 --- a/docs/site/scripts/test_generate_llms.py +++ b/docs/site/scripts/test_generate_llms.py @@ -221,6 +221,63 @@ def test_dropped_card_raises_instead_of_silently_shrinking(self): g.synthesize_landing(body) self.assertIn("dropped cards", str(ctx.exception)) + def test_renamed_title_marker_is_caught_not_absorbed(self): + """The guard must not count the marker it validates. + + Counting `lp-card-title` would shrink in step with the very loss it is + meant to detect, so a renamed marker would drop its card in silence. + Counting `lp-card` containers keeps the two signals independent. + """ + body = """ +
+ +
A
+
Desc A.
+ + +
B
+
Desc B.
+ +
+""" + with self.assertRaises(SystemExit) as ctx: + g.synthesize_landing(body) + self.assertIn("parsed 1 of 2", str(ctx.exception)) + + def test_card_attribute_order_does_not_matter(self): + """`to=` may precede or follow the class; both must parse.""" + body = """ +
+ +
A
+
Desc A.
+ +
+""" + out = g.synthesize_landing(body) + self.assertIsNotNone(out) + self.assertIn("[A](https://docs.erigon.tech/a): Desc A.", out) + + def test_all_cards_malformed_raises_rather_than_falling_back(self): + """Total parser failure must not look like an ordinary prose page. + + Returning None here would send the caller to strip_mdx, degrading the + page silently — the exact outcome the guard exists to prevent. + """ + body = """ +
+ +
A
+ + +
B
+ +
+""" + with self.assertRaises(SystemExit) as ctx: + g.synthesize_landing(body) + self.assertIn("parsed 0 of 2", str(ctx.exception)) + class LeadingH1StripTests(unittest.TestCase): """The build() body-prep step strips a leading H1 to avoid duplicate headings.""" diff --git a/docs/site/static/llms-full.txt b/docs/site/static/llms-full.txt index b875b3d5a63..442bfefb595 100644 --- a/docs/site/static/llms-full.txt +++ b/docs/site/static/llms-full.txt @@ -3929,17 +3929,19 @@ MCP connects a model to your **running node**. To let a model answer questions a | File | Contents | Use it when | |------|----------|-------------| -| [llms.txt](https://docs.erigon.tech/llms.txt) | Page index with titles and descriptions | The model fetches pages on demand, or its context window is small | -| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | Every documentation page, in full, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | +| [llms.txt](https://docs.erigon.tech/llms.txt) | Index of the current documentation, with titles and descriptions | The model fetches pages on demand, or its context window is small | +| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of those same pages, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale. +Two things to know about their scope. They cover the **current** documentation and the help center only — the archived versions published under `/v3.3/` and `/v3.4/` are not included. And the text is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read the corpus as the documentation's prose, not as a byte-for-byte mirror of the rendered site. + ```bash # Give a local model the whole documentation, with no network access curl -sO https://docs.erigon.tech/llms-full.txt ``` -The full file is around 420 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. +The full file is around 430 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. --- diff --git a/docs/site/static/llms.txt b/docs/site/static/llms.txt index d7b26ab9446..6e5927f3913 100644 --- a/docs/site/static/llms.txt +++ b/docs/site/static/llms.txt @@ -4,6 +4,8 @@ > modularity, and minimal disk footprint. It features an integrated consensus layer > (Caplin), BitTorrent-based historical data distribution, and fast node synchronization. +The full text of every page listed below is also available as a single file: [llms-full.txt](https://docs.erigon.tech/llms-full.txt) + - [Introduction](https://docs.erigon.tech/): Official documentation for Erigon — the efficient, modular Ethereum execution client. Get started, explore the JSON-RPC API, and learn how to run a full or archive node. ## Get Started diff --git a/llms-full.txt b/llms-full.txt index b875b3d5a63..442bfefb595 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -3929,17 +3929,19 @@ MCP connects a model to your **running node**. To let a model answer questions a | File | Contents | Use it when | |------|----------|-------------| -| [llms.txt](https://docs.erigon.tech/llms.txt) | Page index with titles and descriptions | The model fetches pages on demand, or its context window is small | -| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | Every documentation page, in full, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | +| [llms.txt](https://docs.erigon.tech/llms.txt) | Index of the current documentation, with titles and descriptions | The model fetches pages on demand, or its context window is small | +| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of those same pages, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale. +Two things to know about their scope. They cover the **current** documentation and the help center only — the archived versions published under `/v3.3/` and `/v3.4/` are not included. And the text is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read the corpus as the documentation's prose, not as a byte-for-byte mirror of the rendered site. + ```bash # Give a local model the whole documentation, with no network access curl -sO https://docs.erigon.tech/llms-full.txt ``` -The full file is around 420 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. +The full file is around 430 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. --- diff --git a/llms.txt b/llms.txt index d7b26ab9446..6e5927f3913 100644 --- a/llms.txt +++ b/llms.txt @@ -4,6 +4,8 @@ > modularity, and minimal disk footprint. It features an integrated consensus layer > (Caplin), BitTorrent-based historical data distribution, and fast node synchronization. +The full text of every page listed below is also available as a single file: [llms-full.txt](https://docs.erigon.tech/llms-full.txt) + - [Introduction](https://docs.erigon.tech/): Official documentation for Erigon — the efficient, modular Ethereum execution client. Get started, explore the JSON-RPC API, and learn how to run a full or archive node. ## Get Started