feat: extraction honours role=main and strips non-content elements - #212
Merged
Conversation
Narrowing to <main>/<article> was not enough on real pages, which nest chrome inside them: one Substack post carried four subscribe widgets and an author-bio footer inside its single <article>, and a document search for the article's results section returned the subscription plea. Two changes, both semantic rather than per-site. contentRoot now falls back to an ARIA role="main" landmark when neither <main> nor <article> exists. And the extracted subtree has nav, header, footer, aside, form, button, script, style, noscript and template removed, along with anything marked aria-hidden or hidden -- all non-content by HTML's own definition, and content a reader cannot see is not content. The precedence was corrected by measurement, not reasoning. role="main" was tried above <article> on the theory that a declared landmark beats an inferred one; against the real Substack post that made the extraction 2KB LARGER, because the main landmark wraps the article plus its surrounding furniture. A page has one main region but that region can hold more than the piece, where <article> is the piece. Class-name matching is deliberately not done. Matching "subscribe", "promo", "related" catches more and is how readability heuristics work, but we keep no raw copy of the fetched bytes, so an over-eager match silently deletes article text with nothing to recover from. Keeping too much is fixable later by ranking; keeping nothing is not. For the same reason stripping refuses to empty a document: a page that is only a footer still yields its text. Measured over the four pages currently in the corpus, before -> after: gist 80,238 -> 76,141; Chroma 100,770 -> 98,231; vinish 10,146 -> 8,088; Substack 44,928 -> 44,869. Three improved substantially. The fourth barely moved, which is the honest limit of this approach -- Substack's widgets are divs with class names, reachable only by the matching this commit rules out. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Narrowing to
<main>/<article>was not enough on real pages, which nest chrome inside them. One Substack post carried four subscribe widgets and an author-bio footer inside its single<article>, and a document search for the article's results section returned the subscription plea.Two changes, both semantic rather than per-site:
contentRootfalls back to an ARIArole="main"landmark when neither<main>nor<article>exists.nav,header,footer,aside,form,button,script,style,noscriptandtemplateremoved, along with anything markedaria-hidden="true"orhidden-- all non-content by HTML's own definition, and content a reader cannot see is not content.The precedence was corrected by measurement, not reasoning
role="main"was tried above<article>on the theory that a declared landmark beats an inferred one. Against the real Substack post that made the extraction 2KB larger, because the main landmark wraps the article plus its surrounding furniture. A page has one main region but that region can hold more than the piece;<article>is the piece. The unit tests passed either way -- only measuring against real pages caught it.Measured, before -> after
Three improved substantially. The fourth barely moved, and that is the honest limit of this approach: Substack's widgets are
<div class="subscribe-widget">, reachable only by the class-name matching this PR deliberately rules out.Why no class-name matching
Matching "subscribe", "promo", "related" catches more and is how readability heuristics work, but we keep no raw copy of the fetched bytes (
docs/document-ingest.md), so an over-eager match silently deletes article text with nothing to recover from. Keeping too much is fixable later by ranking; keeping nothing is not. For the same reason the stripper refuses to empty a document -- a page that is only a footer still yields its text, and there is a test for it.Task 8418 stays open for the readability-style text-density extraction that would reach the rest, with its own class of silent mistakes.
🤖 Generated with Claude Code