docs(site): address review on #23335 — describedby relation, guard hole, corpus claims - #23341
Open
bloxster wants to merge 2 commits into
Open
docs(site): address review on #23335 — describedby relation, guard hole, corpus claims#23341bloxster wants to merge 2 commits into
bloxster wants to merge 2 commits into
Conversation
… claims Review feedback from @yperbasis and Copilot on #23336. Five fixes. 1. Advertise llms.txt with rel="describedby", not rel="alternate". The llmstxt.org proposal defines describedby for the llms.txt file that covers a page, and reserves alternate + text/markdown for a *per-page* Markdown representation. A site-wide index is not an alternate representation of every page, and v2-aware agents look for describedby. llms-full.txt is no longer advertised in head at all: it describes no single page. It stays discoverable through llms.txt, the sitemap, and the MCP docs page. 2. Close a hole in the card-count guard. `if not cards: return None` ran before the count was taken, so a grid where *every* card failed to parse was indistinguishable from an ordinary prose page: the caller fell back to strip_mdx and the guard never ran — silently degrading the exact case it exists to catch. Count first, parse second, and return None only when the page has no cards at all. 3. Stop claiming llms-full.txt holds "every documentation page, in full". It does not: synthesize_landing replaces the whole body of a card-grid page with its card list, so why-using-erigon loses its introduction and prose. Describe the corpus as cleaned rather than verbatim, and say what is dropped. 4. Stop grouping llms.txt with llms-full.txt as "the whole documentation as one plain-text file" on why-using-erigon. llms.txt is only an index. 5. Drop "nothing else links to them" from the config comment — this PR adds the MCP page links, which makes it false. Also refresh the stated file size, 420 KB -> 430 KB.
Copilot stopped reviewing on behalf of
bloxster due to an error
August 17, 2026 14:04
…, scope Follow-up review from @yperbasis on #23336. Three findings, all confirmed by reproduction before fixing. 1. The guard counted the marker it was validating. `expected` came from `lp-card-title`, so if a card lost or renamed that marker the count shrank in step with the loss it was meant to detect: verified that a two-card grid with one renamed marker emitted one bullet and raised nothing. If every marker changed, `expected` hit zero and the page fell back to strip_mdx. Count `lp-card` containers instead — the wrapper is not consumed by the parse, so the two signals stay independent. Attributes are now matched order-independently and `to=` is read separately, which the container match no longer pins down. 2. llms.txt had no route to llms-full.txt. Neither committed index contained the string at all, so once the llms-full head tag was removed, an agent following rel="describedby" reached an index with no way to find the full corpus. The generator now emits that link, and both copies are regenerated. 3. "Every documentation page" was still wrong. SECTIONS scans only docs/ and help-center/, while the site also publishes v3.3 and v3.4 from versioned_docs/ — neither artifact contains those URLs. Say current documentation, and state the exclusion outright. Two new tests: a renamed title marker must raise rather than be absorbed (verified to fail against the previous count), and card attributes must parse in either order.
Collaborator
Author
|
Updated with the second review round from #23336 (005311b), so this stays the exact 3.6 counterpart:
|
Copilot stopped reviewing on behalf of
bloxster due to an error
August 17, 2026 14:21
Collaborator
Author
|
Note on automated review: Copilot has been re-requested four times here and returned "encountered an error and was unable to review this pull request" every time (14:02, 14:04, 14:13, 14:21 UTC), so there is no bot review on this PR. For reviewer confidence, the substantive content is identical to #23336, which Copilot reviewed twice and @yperbasis reviewed twice:
The only differences are pre-existing branch divergence unrelated to this change: Happy to re-request Copilot again if it recovers, but it looks like a persistent failure on this PR rather than a transient one. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #23335, which merged before review feedback on its
maincounterpart (#23336) landed. Three defects shipped with it; this carries the same fixes.1.
rel="describedby", notrel="alternate". The llmstxt.org proposal definesdescribedbyfor the llms.txt file that covers a page, and reservesalternate+text/markdownfor a per-page Markdown representation. A site-wide index is not an alternate representation of every page, and v2-aware agents look fordescribedby.llms-full.txtis no longer advertised inheadat all — it describes no single page, so it does not belong in a per-page relation. It stays discoverable through llms.txt, the sitemap, and the MCP docs page.2. A hole in the card-count guard.
if not cards: return Noneran before the count was taken, so a grid where every card failed to parse was indistinguishable from an ordinary prose page:collect_pagesfell back tostrip_mdxand the guard never ran — silently degrading precisely the case it exists to catch. Now the count comes first andNoneis returned only when the page has no cards at all. New regression test, verified to fail against the previous ordering.3.
llms-full.txtdid not contain what we claimed. The table advertised "every documentation page, in full". It does not:synthesize_landingreplaces the entire body of a card-grid page with its card list, sowhy-using-erigon.mdx— 183 source lines — reaches the corpus as its 11 bullets alone, without its introduction, prose, or MCP section. The table now says "the text of every documentation page", with an explicit sentence that the corpus is cleaned rather than verbatim and that card-grid index pages are reduced to their links.Plus two smaller corrections: the
why-using-erigonpointer no longer groups llms.txt with llms-full.txt as "the whole documentation as one plain-text file" (llms.txt is only an index), and the config comment drops "nothing else links to them", which #23335 itself made false.Verification
npm ci && npm run buildclean;generate-llms.py --checkgreen; 82 tests pass;describedbypresent on 198/199 pages; zeroalternate/text/plainand zerollms-fullhead tags remaining; sitemap still carries both files.Credit to @yperbasis and Copilot for catching all three on #23336.