Collect PDF links from sitemap-driven spiders too#6
Open
MirjamOdile wants to merge 1 commit into
Open
Conversation
PDF_MODE=links_only (the default) records every .pdf link on a crawled page as a URL-only item — but the link-scan was wired only into the rule-based spider (database_spider.parse_article). Sitemap-driven spiders emitted no PDF items at all, so any project relying on the framework's PDF harvest silently collected zero PDFs from them. parse_article() in sitemap_spider gains the exact block database_spider has, with source_label="sitemap_spider". See docs/requests/12-sitemap-pdf-collection.md (ships with the quality tool PR) for the full write-up.
iRanadheer
force-pushed
the
pr/01-sitemap-pdf-collection
branch
from
b7ea136 to
e9cf4c9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
PDF_MODE=links_only(the default) records every.pdflink on a crawled pageas a URL-only item — but the link-scan is wired only into the rule-based spider
(
database_spider.py::parse_article). Sitemap-driven spiders — the majority ofa typical fleet — emit no PDF items at all, so any project relying on the
framework's PDF harvest silently collects zero PDFs from them.
Change
sitemap_spider.parse_article()gains the exact blockdatabase_spiderhas,with
source_label="sitemap_spider". The item shape is identical:{url, title=<filename>, content: "", metadata_json: {content_type: "pdf", found_on: <linking page>}}— nothing is downloaded.Verification
test_sitemap_links_only_yields_url_only_itemsexercises thereal
parse_articlewith anHtmlResponse: same-org + external link, itemshape including
found_onprovenance and the source label.USE_SITEMAPspider's first test crawl produced 21 PDF rowsalongside 36 HTML articles with zero per-spider config.
Notes
Known shared limitation (inherited from the rule-based spider, unchanged here):
pages routed to named callbacks don't get the PDF scan, and
PDF_MODE=extractstill has no link-following path for sitemap spiders. A single post-processing
hook on yielded responses would remove the per-spider-type duplication —
follow-up material, out of scope for this fix.