diff --git a/docs/site/docs/fundamentals/mcp.mdx b/docs/site/docs/fundamentals/mcp.mdx index a1dce4837e7..6e010349613 100644 --- a/docs/site/docs/fundamentals/mcp.mdx +++ b/docs/site/docs/fundamentals/mcp.mdx @@ -326,3 +326,25 @@ In addition to tools, the MCP server exposes structured **resources** and **prom | `debug_logs` | Triage node issues from recent log output | | `torrent_status` | Snapshot download health check | | `sync_analysis` | Node sync progress and performance | + +--- + +## These Docs as a Single File + +MCP connects a model to your **running node**. To let a model answer questions about **Erigon itself** — flags, pruning modes, RPC namespaces, hardware sizing — point it at this documentation in machine-readable form, following the [llmstxt.org](https://llmstxt.org) convention: + +| File | Contents | Use it when | +|------|----------|-------------| +| [llms.txt](https://docs.erigon.tech/llms.txt) | Index of the current documentation, with titles and descriptions | The model fetches pages on demand, or its context window is small | +| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of those same pages, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | + +Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale. + +Two things to know about their scope. They cover the **current** documentation and the help center only — the archived versions published under `/v3.3/` and `/v3.4/` are not included. And the text is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read the corpus as the documentation's prose, not as a byte-for-byte mirror of the rendered site. + +```bash +# Give a local model the whole documentation, with no network access +curl -sO https://docs.erigon.tech/llms-full.txt +``` + +The full file is around 430 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. diff --git a/docs/site/docs/get-started/why-using-erigon.mdx b/docs/site/docs/get-started/why-using-erigon.mdx index c555338df50..e2551825c83 100644 --- a/docs/site/docs/get-started/why-using-erigon.mdx +++ b/docs/site/docs/get-started/why-using-erigon.mdx @@ -179,3 +179,5 @@ The interface is **read-only** and localhost-bound — no risk of unintended sta # Register with Claude Code in one command claude mcp add --transport sse erigon http://127.0.0.1:8553/sse ``` + +Where MCP lets a model query your *node*, [llms.txt and llms-full.txt](../fundamentals/mcp#these-docs-as-a-single-file) let it read these *docs*: an index of the current documentation, and the same pages' text as one plain-text file for offline use. diff --git a/docs/site/docusaurus.config.ts b/docs/site/docusaurus.config.ts index 2593cad2ce2..67d82496de4 100644 --- a/docs/site/docusaurus.config.ts +++ b/docs/site/docusaurus.config.ts @@ -86,6 +86,21 @@ export default async function createConfig(): Promise { attributes: {}, innerHTML: 'window.plausible=window.plausible||function(){(plausible.q=plausible.q||[]).push(arguments)},plausible.init=plausible.init||function(i){plausible.o=i||{}};plausible.init()', }, + // Point every page at the llms.txt index that covers it. `describedby` is + // the relation the llmstxt.org proposal defines for exactly this; its + // `alternate` + `text/markdown` pairing is reserved for a *per-page* + // Markdown representation, which this site-wide index is not. + // llms-full.txt is deliberately not advertised here: it is not a + // description of any single page. It stays discoverable through llms.txt, + // the sitemap, and the MCP docs page. + { + tagName: 'link', + attributes: { + rel: 'describedby', + href: 'https://docs.erigon.tech/llms.txt', + title: 'Erigon Documentation — page index for LLMs', + }, + }, ], plugins: [ @@ -150,6 +165,15 @@ export default async function createConfig(): Promise { changefreq: 'weekly', priority: 0.5, ignorePatterns: ['/search'], + // The llms.txt artifacts live in static/, so Docusaurus never routes + // them and the default sitemap omits them — which leaves them + // unindexable by search. Append them explicitly. This is also how + // llms-full.txt stays reachable, since it is not advertised in head. + createSitemapItems: async ({defaultCreateSitemapItems, ...rest}) => [ + ...(await defaultCreateSitemapItems(rest)), + {url: 'https://docs.erigon.tech/llms.txt', changefreq: 'weekly', priority: 0.5}, + {url: 'https://docs.erigon.tech/llms-full.txt', changefreq: 'weekly', priority: 0.5}, + ], }, } satisfies Preset.Options], ], diff --git a/docs/site/scripts/generate-llms.py b/docs/site/scripts/generate-llms.py index 7ca09256013..d2e6b970059 100644 --- a/docs/site/scripts/generate-llms.py +++ b/docs/site/scripts/generate-llms.py @@ -285,16 +285,30 @@ def strip_mdx(text): # index pages. After MDX stripping, they collapse to a structureless pile of # title/desc fragments. We detect the lp-card pattern and synthesize a clean # bullet list instead, keeping link structure intact for LLM consumption. -_LANDING_CARD_RE = re.compile( - r']*\bto=[\'"]([^\'"]+)[\'"][^>]*>' - r'(?:.*?)' - r']*\blp-card-title\b[^>]*>([^<]+)' - r'(?:.*?)' - r']*\blp-card-desc\b[^>]*>([^<]+)' - r'(?:.*?)' - r'', - re.DOTALL, -) +# Parsed in two stages, deliberately. A single pattern spanning the whole card +# cannot express "and never cross into the next card": with `[^<]+` for the text +# and `.*?` for the gaps, a description containing inline markup (``, +# ``) fails to match locally, and the engine then scans forward and pairs +# the title with the *next* card's description — silently swallowing the card in +# between. Matching each block first makes a card boundary unrepresentable. +# +# The container match is also what the guard below counts. It deliberately keys +# on `lp-card` — the wrapper — and not on any marker the parse itself consumes: +# a count taken from `lp-card-title` would shrink in step with the very loss it +# is supposed to detect, so a renamed title marker would drop its card in +# silence. Attributes are matched order-independently; `to=` is read separately. +_LANDING_CARD_LINK_RE = re.compile( + r']*\blp-card\b[^>]*)>(.*?)', re.DOTALL) +_LINK_TO_RE = re.compile(r'\bto=[\'"]([^\'"]+)[\'"]') +_CARD_TITLE_RE = re.compile( + r']*\blp-card-title\b[^>]*>(.*?)', re.DOTALL) +_CARD_DESC_RE = re.compile( + r']*\blp-card-desc\b[^>]*>(.*?)', re.DOTALL) + + +def _card_text(fragment): + """Flatten a card's inner HTML to plain text, collapsing whitespace.""" + return re.sub(r'\s+', ' ', re.sub(r'<[^>]+>', '', fragment)).strip() _LANDING_HERO_RE = re.compile( r']*\blp-hero\b[^>]*>' r'(?:.*?)' @@ -312,10 +326,35 @@ def synthesize_landing(raw_body): Card hrefs that start with `/` are rewritten to absolute BASE_URL form so bullets remain useful when the body is read out of context. """ - cards = list(_LANDING_CARD_RE.finditer(raw_body)) - if not cards: + # Count first, parse second. Deciding "this is not a card grid" on an empty + # parse result would make total parser failure indistinguishable from an + # ordinary prose page: the caller would fall back to strip_mdx and the guard + # below would never run — silently degrading exactly the case it exists to + # catch. + containers = _LANDING_CARD_LINK_RE.findall(raw_body) + expected = len(containers) + if not expected: return None + cards = [] + for attrs, block in containers: + href = _LINK_TO_RE.search(attrs) + title = _CARD_TITLE_RE.search(block) + desc = _CARD_DESC_RE.search(block) + if not href or not title or not desc: + continue + cards.append((href.group(1), _card_text(title.group(1)), + _card_text(desc.group(1)))) + + # A card that parses to nothing is invisible in the output, and `--check` + # only compares generated output against committed output — so a systematic + # parser regression stays green in CI while the corpus quietly loses pages. + if len(cards) != expected: + raise SystemExit( + f"synthesize_landing dropped cards: parsed {len(cards)} " + f"of {expected} lp-card containers" + ) + out = [] hero = _LANDING_HERO_RE.search(raw_body) if hero: @@ -326,12 +365,10 @@ def synthesize_landing(raw_body): out.append("## Sections") out.append("") - for m in cards: - href = m.group(1).strip() + for href, title, desc in cards: + href = href.strip() if href.startswith("/"): href = BASE_URL + href.rstrip("/") - title = m.group(2).strip() - desc = m.group(3).strip() out.append(f"- [{title}]({href}): {desc}") return "\n".join(out) @@ -524,6 +561,14 @@ def display_section(p): lines.append("> modularity, and minimal disk footprint. It features an integrated consensus layer") lines.append("> (Caplin), BitTorrent-based historical data distribution, and fast node synchronization.") lines.append("") + # The only machine-readable route from this index to the full corpus. Pages + # advertise llms.txt via , not llms-full.txt, so an + # agent that arrives here would otherwise have no way to reach it. + lines.append( + f"The full text of every page listed below is also available as a single " + f"file: [llms-full.txt]({BASE_URL}/llms-full.txt)" + ) + lines.append("") current_section = None for p in all_pages: diff --git a/docs/site/scripts/test_generate_llms.py b/docs/site/scripts/test_generate_llms.py index 187ec6c154c..f49ca8b8a35 100644 --- a/docs/site/scripts/test_generate_llms.py +++ b/docs/site/scripts/test_generate_llms.py @@ -159,6 +159,125 @@ def test_no_cards_returns_none(self): out = g.synthesize_landing("# Hello\n\nProse only, no cards.") self.assertIsNone(out) + def test_desc_with_inline_markup_is_extracted(self): + """Inline markup in a description must not defeat the match.""" + body = """ +
+ +
Flexible Pruning
+
--prune.mode=minimal gives a smaller footprint.
+ +
+""" + out = g.synthesize_landing(body) + self.assertIsNotNone(out) + self.assertIn("--prune.mode=minimal gives a smaller footprint.", out) + self.assertNotIn("", out) + self.assertNotIn("", out) + + def test_markup_card_does_not_swallow_the_next_card(self): + """A card whose desc has markup must not steal the following card's desc.""" + body = """ +
+ +
Immutable Data
+
Historical data lives in immutable files.
+ + +
Staged Sync
+
Splits processing into specialised stages.
+ +
+""" + out = g.synthesize_landing(body) + self.assertIsNotNone(out) + bullets = [ln for ln in out.split("\n") if ln.startswith("- [")] + self.assertEqual(len(bullets), 2) + self.assertIn( + "- [Immutable Data](https://docs.erigon.tech/immutable): " + "Historical data lives in immutable files.", + bullets, + ) + self.assertIn( + "- [Staged Sync](https://docs.erigon.tech/staged-sync): " + "Splits processing into specialised stages.", + bullets, + ) + + def test_dropped_card_raises_instead_of_silently_shrinking(self): + """A card the parser cannot read must fail loudly — `--check` cannot see it.""" + body = """ +
+ +
Fine
+
Parses correctly.
+ + +
Missing Its Description
+ +
+""" + with self.assertRaises(SystemExit) as ctx: + g.synthesize_landing(body) + self.assertIn("dropped cards", str(ctx.exception)) + + def test_renamed_title_marker_is_caught_not_absorbed(self): + """The guard must not count the marker it validates. + + Counting `lp-card-title` would shrink in step with the very loss it is + meant to detect, so a renamed marker would drop its card in silence. + Counting `lp-card` containers keeps the two signals independent. + """ + body = """ +
+ +
A
+
Desc A.
+ + +
B
+
Desc B.
+ +
+""" + with self.assertRaises(SystemExit) as ctx: + g.synthesize_landing(body) + self.assertIn("parsed 1 of 2", str(ctx.exception)) + + def test_card_attribute_order_does_not_matter(self): + """`to=` may precede or follow the class; both must parse.""" + body = """ +
+ +
A
+
Desc A.
+ +
+""" + out = g.synthesize_landing(body) + self.assertIsNotNone(out) + self.assertIn("[A](https://docs.erigon.tech/a): Desc A.", out) + + def test_all_cards_malformed_raises_rather_than_falling_back(self): + """Total parser failure must not look like an ordinary prose page. + + Returning None here would send the caller to strip_mdx, degrading the + page silently — the exact outcome the guard exists to prevent. + """ + body = """ +
+ +
A
+ + +
B
+ +
+""" + with self.assertRaises(SystemExit) as ctx: + g.synthesize_landing(body) + self.assertIn("parsed 0 of 2", str(ctx.exception)) + class LeadingH1StripTests(unittest.TestCase): """The build() body-prep step strips a leading H1 to avoid duplicate headings.""" diff --git a/docs/site/static/llms-full.txt b/docs/site/static/llms-full.txt index d2eddaf7524..6aadeb4593d 100644 --- a/docs/site/static/llms-full.txt +++ b/docs/site/static/llms-full.txt @@ -38,12 +38,15 @@ URL: https://docs.erigon.tech/get-started/why-using-erigon - [Integrated Consensus Layer (Caplin)](../fundamentals/caplin): Built-in consensus client — no need to run and manage separate software like Lighthouse. One binary handles both execution and consensus, simplifying setup for stakers and node runners. - [State Storage (Flat DB)](../fundamentals/optimizing-storage): Replaces the Merkle Patricia Trie with a flat key-value database. Faster reads and writes, and a smaller on-disk footprint for any node type. -- [Immutable, Decentralised Data](../fundamentals/modules/downloader): Splits data processing into specialised stages, minimising random disk I/O and write amplification. Efficiently integrates large datasets into Erigon's flat DB. +- [Immutable, Decentralised Data](../fundamentals/modules/downloader): Historical data lives in immutable files distributed via BitTorrent — up to 10× cheaper to backup, repair, and distribute. No double-disk usage, no complex serialisation. +- [Staged Sync](../fundamentals/pruning-modes): Splits data processing into specialised stages, minimising random disk I/O and write amplification. Efficiently integrates large datasets into Erigon's flat DB. - [Predictable RPC Performance](../fundamentals/modules/rpc-daemon): No background compaction means no unpredictable resource spikes. RPC throughput stays stable and consistent — critical for providers and validators who cannot afford surprises. - [Modularity](../fundamentals/modules/): Core node, RPC Daemon, and TxPool run as independent processes with per-component resource limits. If one crashes the rest stay alive. Run multiple instances without downtime. - [OtterSync](../fundamentals/pruning-modes): Shifts initial sync from CPU-intensive transaction re-execution to network download — similar to BitTorrent for state data. Reduces sync time for both full and Archive nodes. -- [Flexible Pruning](../fundamentals/optimizing-storage): Modular architecture and predictable performance handle high-volume traffic with low latency. Smaller database cuts hardware costs. Caplin reduces missed attestations and slashing risk. +- [Flexible Pruning](../fundamentals/optimizing-storage): --prune.mode=minimal gives a smaller-than-full-node footprint that still satisfies all validator requirements. Right-size your storage without sacrificing functionality. +- [RPC Providers & Large Stakers](../fundamentals/modules/rpc-daemon): Modular architecture and predictable performance handle high-volume traffic with low latency. Smaller database cuts hardware costs. Caplin reduces missed attestations and slashing risk. - [Home Users & Solo Stakers](../staking/): Run a full or archive node on a consumer SSD. Fast sync cuts setup from days to hours. No third-party servers — your node, your keys, your contribution to decentralisation. +- [Developers](../interacting-with-erigon/): Full trace_ namespace, historical eth_getProof, historical blobs, and gRPC API. Modular components let you update, extend, or replace parts without rebuilding the whole stack. --- @@ -3977,6 +3980,28 @@ In addition to tools, the MCP server exposes structured **resources** and **prom --- +## These Docs as a Single File + +MCP connects a model to your **running node**. To let a model answer questions about **Erigon itself** — flags, pruning modes, RPC namespaces, hardware sizing — point it at this documentation in machine-readable form, following the [llmstxt.org](https://llmstxt.org) convention: + +| File | Contents | Use it when | +|------|----------|-------------| +| [llms.txt](https://docs.erigon.tech/llms.txt) | Index of the current documentation, with titles and descriptions | The model fetches pages on demand, or its context window is small | +| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of those same pages, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | + +Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale. + +Two things to know about their scope. They cover the **current** documentation and the help center only — the archived versions published under `/v3.3/` and `/v3.4/` are not included. And the text is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read the corpus as the documentation's prose, not as a byte-for-byte mirror of the rendered site. + +```bash +# Give a local model the whole documentation, with no network access +curl -sO https://docs.erigon.tech/llms-full.txt +``` + +The full file is around 430 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. + +--- + # Otterscan URL: https://docs.erigon.tech/fundamentals/otterscan diff --git a/docs/site/static/llms.txt b/docs/site/static/llms.txt index d7b26ab9446..6e5927f3913 100644 --- a/docs/site/static/llms.txt +++ b/docs/site/static/llms.txt @@ -4,6 +4,8 @@ > modularity, and minimal disk footprint. It features an integrated consensus layer > (Caplin), BitTorrent-based historical data distribution, and fast node synchronization. +The full text of every page listed below is also available as a single file: [llms-full.txt](https://docs.erigon.tech/llms-full.txt) + - [Introduction](https://docs.erigon.tech/): Official documentation for Erigon — the efficient, modular Ethereum execution client. Get started, explore the JSON-RPC API, and learn how to run a full or archive node. ## Get Started diff --git a/llms-full.txt b/llms-full.txt index d2eddaf7524..6aadeb4593d 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -38,12 +38,15 @@ URL: https://docs.erigon.tech/get-started/why-using-erigon - [Integrated Consensus Layer (Caplin)](../fundamentals/caplin): Built-in consensus client — no need to run and manage separate software like Lighthouse. One binary handles both execution and consensus, simplifying setup for stakers and node runners. - [State Storage (Flat DB)](../fundamentals/optimizing-storage): Replaces the Merkle Patricia Trie with a flat key-value database. Faster reads and writes, and a smaller on-disk footprint for any node type. -- [Immutable, Decentralised Data](../fundamentals/modules/downloader): Splits data processing into specialised stages, minimising random disk I/O and write amplification. Efficiently integrates large datasets into Erigon's flat DB. +- [Immutable, Decentralised Data](../fundamentals/modules/downloader): Historical data lives in immutable files distributed via BitTorrent — up to 10× cheaper to backup, repair, and distribute. No double-disk usage, no complex serialisation. +- [Staged Sync](../fundamentals/pruning-modes): Splits data processing into specialised stages, minimising random disk I/O and write amplification. Efficiently integrates large datasets into Erigon's flat DB. - [Predictable RPC Performance](../fundamentals/modules/rpc-daemon): No background compaction means no unpredictable resource spikes. RPC throughput stays stable and consistent — critical for providers and validators who cannot afford surprises. - [Modularity](../fundamentals/modules/): Core node, RPC Daemon, and TxPool run as independent processes with per-component resource limits. If one crashes the rest stay alive. Run multiple instances without downtime. - [OtterSync](../fundamentals/pruning-modes): Shifts initial sync from CPU-intensive transaction re-execution to network download — similar to BitTorrent for state data. Reduces sync time for both full and Archive nodes. -- [Flexible Pruning](../fundamentals/optimizing-storage): Modular architecture and predictable performance handle high-volume traffic with low latency. Smaller database cuts hardware costs. Caplin reduces missed attestations and slashing risk. +- [Flexible Pruning](../fundamentals/optimizing-storage): --prune.mode=minimal gives a smaller-than-full-node footprint that still satisfies all validator requirements. Right-size your storage without sacrificing functionality. +- [RPC Providers & Large Stakers](../fundamentals/modules/rpc-daemon): Modular architecture and predictable performance handle high-volume traffic with low latency. Smaller database cuts hardware costs. Caplin reduces missed attestations and slashing risk. - [Home Users & Solo Stakers](../staking/): Run a full or archive node on a consumer SSD. Fast sync cuts setup from days to hours. No third-party servers — your node, your keys, your contribution to decentralisation. +- [Developers](../interacting-with-erigon/): Full trace_ namespace, historical eth_getProof, historical blobs, and gRPC API. Modular components let you update, extend, or replace parts without rebuilding the whole stack. --- @@ -3977,6 +3980,28 @@ In addition to tools, the MCP server exposes structured **resources** and **prom --- +## These Docs as a Single File + +MCP connects a model to your **running node**. To let a model answer questions about **Erigon itself** — flags, pruning modes, RPC namespaces, hardware sizing — point it at this documentation in machine-readable form, following the [llmstxt.org](https://llmstxt.org) convention: + +| File | Contents | Use it when | +|------|----------|-------------| +| [llms.txt](https://docs.erigon.tech/llms.txt) | Index of the current documentation, with titles and descriptions | The model fetches pages on demand, or its context window is small | +| [llms-full.txt](https://docs.erigon.tech/llms-full.txt) | The text of those same pages, as one plain-text file | You want the whole corpus in context, offline, with nothing scraped | + +Both are generated from the same Markdown source as this site by a script in the repository, and CI fails when a documentation change lands without regenerating them, so they cannot quietly go stale. + +Two things to know about their scope. They cover the **current** documentation and the help center only — the archived versions published under `/v3.3/` and `/v3.4/` are not included. And the text is cleaned rather than copied verbatim: MDX components are stripped, and section index pages built as card grids are reduced to the list of links they present. Read the corpus as the documentation's prose, not as a byte-for-byte mirror of the rendered site. + +```bash +# Give a local model the whole documentation, with no network access +curl -sO https://docs.erigon.tech/llms-full.txt +``` + +The full file is around 430 KB. If your model's context cannot hold it, use `llms.txt` to pick the pages you need, or slice the sections you want out of the full file — each page begins with a `# Title` heading. + +--- + # Otterscan URL: https://docs.erigon.tech/fundamentals/otterscan diff --git a/llms.txt b/llms.txt index d7b26ab9446..6e5927f3913 100644 --- a/llms.txt +++ b/llms.txt @@ -4,6 +4,8 @@ > modularity, and minimal disk footprint. It features an integrated consensus layer > (Caplin), BitTorrent-based historical data distribution, and fast node synchronization. +The full text of every page listed below is also available as a single file: [llms-full.txt](https://docs.erigon.tech/llms-full.txt) + - [Introduction](https://docs.erigon.tech/): Official documentation for Erigon — the efficient, modular Ethereum execution client. Get started, explore the JSON-RPC API, and learn how to run a full or archive node. ## Get Started