Skip to content

Fix PsycNet full-text section extraction #362

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
absurd wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix PsycNet full-text section extraction #362

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/constants.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -618,7 +618,7 @@ export const PARTIAL_SELECTORS = [
'jswarning',
'js-warning',
'jumplink',
'jumpto',
'(?<![a-z0-9])jumpto(?![a-z0-9])', // avoid APA PsycNet's ftJumpToAnchor article sections
'jump-to-',
'js-skip-to-content',

Expand Down
36 changes: 36 additions & 0 deletions tests/expected/issues--psycnet-ft-jump-to-anchor.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
```json
{
"title": "Article Sections With Jump-To Anchors",
"author": "Example Author",
"site": "Example Author",
"published": ""
}
```

## Abstract

This invented article demonstrates a publishing layout in which the full text is divided into anchored sections. The abstract remains outside those sections, just as it does on the source site. It is deliberately substantial so the incomplete extraction produced by the regression remains above the short-content retry threshold and cannot silently recover by disabling partial-selector removal.

The example contains no names, findings, quotations, or other text from a real publication. Its only site-specific detail is the structural class used by the article sections. That is enough to reproduce the extraction failure while keeping the fixture small, deterministic, and suitable for the repository.

The unaffected material also needs enough length to mirror the real failure mode. On a full paper, the abstract, front matter, and reference list can easily exceed the parser's retry threshold even after every body section has vanished. This additional invented paragraph keeps the broken first pass above that threshold, ensuring the regression test observes the original omission instead of a recovery path intended for genuinely short pages.

## First anchored section

This section is ordinary article prose and must survive extraction. APA PsycNet applies the mixed-case class name to full-text section wrappers so that its jump navigation can target them. A case-insensitive partial match for the shorter clutter token appears inside that longer class name and previously caused the entire wrapper, including its heading and paragraphs, to be discarded.

Removing the wrapper creates a severe but quiet failure. Readers receive the abstract and bibliography, which makes the result appear plausible, while the substantive discussion between them is absent. Preserving this section therefore verifies the behavior that matters to people clipping a full paper for later reading.

## Second anchored section

A second wrapper confirms that the repair is structural rather than dependent on one identifier or heading. The longer class token should not be treated as navigation merely because its internal letters happen to spell the shorter removal pattern. Only a standalone selector token, such as the actual jump control above, should match.

The expected output retains both paragraphs and their section heading while omitting the navigation control. Together those assertions protect both sides of the change: article content stays available, and genuine page chrome continues to be filtered from the cleaned document.

## References

Example Author. (2026). A fictional reference included only to model the position of a bibliography after the article body.

Sample Researcher. (2025). Another invented citation supplies additional unaffected content so the buggy result is long enough to avoid Defuddle's automatic retry path. Journal of Reproducible Fixtures, 1, 1–10.

Test Writer. (2024). Minimal examples can isolate selector behavior without reproducing copyrighted source material. Testing Review, 2, 20–30.
51 changes: 51 additions & 0 deletions tests/fixtures/issues--psycnet-ft-jump-to-anchor.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
<!-- {"url": "https://psycnet.apa.org/fulltext/example-article.html"} -->
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Article Sections With Jump-To Anchors</title>
<meta name="author" content="Example Author">
</head>
<body>
<main id="psycnet_fulltext_article_content">
<article class="contentItem ftDocument">
<h1>Article Sections With Jump-To Anchors</h1>

<div class="articleAbstract">
<h2>Abstract</h2>
<p>This invented article demonstrates a publishing layout in which the full text is divided into anchored sections. The abstract remains outside those sections, just as it does on the source site. It is deliberately substantial so the incomplete extraction produced by the regression remains above the short-content retry threshold and cannot silently recover by disabling partial-selector removal.</p>
<p>The example contains no names, findings, quotations, or other text from a real publication. Its only site-specific detail is the structural class used by the article sections. That is enough to reproduce the extraction failure while keeping the fixture small, deterministic, and suitable for the repository.</p>
<p>The unaffected material also needs enough length to mirror the real failure mode. On a full paper, the abstract, front matter, and reference list can easily exceed the parser's retry threshold even after every body section has vanished. This additional invented paragraph keeps the broken first pass above that threshold, ensuring the regression test observes the original omission instead of a recovery path intended for genuinely short pages.</p>
</div>

<div class="jumpto">
<h2>Jump to</h2>
<p>This navigation control should be removed from the extracted article.</p>
<a href="#s2">First anchored section</a>
<a href="#s3">Second anchored section</a>
</div>

<div class="articleBody">
<div class="section ftJumpToAnchor" id="s2">
<h2>First anchored section</h2>
<p>This section is ordinary article prose and must survive extraction. APA PsycNet applies the mixed-case class name to full-text section wrappers so that its jump navigation can target them. A case-insensitive partial match for the shorter clutter token appears inside that longer class name and previously caused the entire wrapper, including its heading and paragraphs, to be discarded.</p>
<p>Removing the wrapper creates a severe but quiet failure. Readers receive the abstract and bibliography, which makes the result appear plausible, while the substantive discussion between them is absent. Preserving this section therefore verifies the behavior that matters to people clipping a full paper for later reading.</p>
</div>

<div class="section ftJumpToAnchor" id="s3">
<h2>Second anchored section</h2>
<p>A second wrapper confirms that the repair is structural rather than dependent on one identifier or heading. The longer class token should not be treated as navigation merely because its internal letters happen to spell the shorter removal pattern. Only a standalone selector token, such as the actual jump control above, should match.</p>
<p>The expected output retains both paragraphs and their section heading while omitting the navigation control. Together those assertions protect both sides of the change: article content stays available, and genuine page chrome continues to be filtered from the cleaned document.</p>
</div>
</div>

<div class="refList">
<h2>References</h2>
<p>Example Author. (2026). A fictional reference included only to model the position of a bibliography after the article body.</p>
<p>Sample Researcher. (2025). Another invented citation supplies additional unaffected content so the buggy result is long enough to avoid Defuddle's automatic retry path. Journal of Reproducible Fixtures, 1, 1–10.</p>
<p>Test Writer. (2024). Minimal examples can isolate selector behavior without reproducing copyrighted source material. Testing Review, 2, 20–30.</p>
</div>
</article>
</main>
</body>
</html>
Loading