feat(spiders): XML and CSV feed spiders + transparent gzip bodies (upstream v0.4.13) - #108
Merged
Merged
Conversation
…stream v0.4.13) Ports the one applicable feature pair from upstream v0.4.13 (v0.4.14 is a Python-packaging fix, n/a): - XmlFeedSpider: iterate feed nodes (RSS <item> by default, any CSS selector via iter_tag, e.g. "entry" for Atom). Without a callback, each node becomes an object of its child elements' text — RSS-to-items with zero parsing code. HTML-void feed tags (notably <link>, which carries every RSS item's URL and whose text html5ever would swallow) are rewritten to xmlfeed-* for parsing and translated back in item keys; custom callbacks address them by the rewritten name (documented). - CsvFeedSpider: rows as header-keyed objects via a hand-rolled RFC 4180 parser (quoted fields with "" escapes, embedded delimiters and newlines, LF/CRLF, no phantom trailing row), configurable delimiter, header override (first row becomes data), short rows padded, extra cells dropped. - Transparent gzip decompression in the body decoder (magic-byte detection, flate2): raw .xml.gz / .csv.gz FILES served without Content-Encoding now arrive as text — this also closes the documented SitemapSpider .xml.gz gap. Decompression is capped at the fetcher's max_body_bytes so a decompression bomb cannot bypass the body-size cap; oversized or corrupt gzip falls back to the raw bytes (previous behavior). Tests: 4 gzip decoder unit tests (roundtrip, bomb fallback, corrupt fallback, non-gzip untouched), 3 CSV parser unit tests, 4 feed spider parse tests (RSS default incl. the <link> case, Atom iter_tag + custom callback, CSV header/override/delimiter), and a live-server end-to-end test proving a .gz file body arrives decompressed. README gains a feed spider section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDFsMaKk764vogjUW3nqpk
- rewrite_void_tags: expand self-closing <link/> to an explicit end tag (html5ever ignores the self-closing flag on unknown elements, which left the rewritten tag open and swallowed all following Atom entry siblings); accept CR/LF as tag-name boundaries per XML S; track quoted attribute values when locating the tag-closing '>' - gunzip_capped: use MultiGzDecoder so concatenated gzip members (pigz/bgzip output) are fully decompressed instead of truncated - parse_csv: correct the blank-line comment to match behavior (skipped, like Python's csv module) and pin it with a test - docs: XmlFeedSpider limitations (namespaces, CDATA); decode_body's uncapped transparent gzip Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDFsMaKk764vogjUW3nqpk
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports the two portable features of upstream Scrapling v0.4.13 (
XMLFeedSpider,CSVFeedSpider) plus the transparent gzip handling they rely on. Upstream v0.4.14 was evaluated and is a Python-packaging-only fix (uv), not applicable here.What's new
XmlFeedSpider(src/spiders/templates/xml_feed.rs)iter_tag(default"item"for RSS; use"entry"for Atom — any CSS selector works).<title>X</title>→"title": "X", last duplicate wins); an optionalparse_nodecallback receives the response + nodeSelectorfor custom extraction.<link>and<meta>as void elements and swallows their text — and<link>carries every RSS item's URL. Before parsing, whole-tag case-insensitive occurrences are rewritten toxmlfeed-link/xmlfeed-meta(multibyte-safe byte scanner), item keys are translated back, and a useriter_tagnaming a rewritten tag is translated too. Custom callbacks address the rewritten names in CSS queries (documented on the builder).parsenever emits follow-up requests.CsvFeedSpider(src/spiders/templates/csv_feed.rs)"-quoted fields with""escapes and embedded delimiters/newlines, LF/CRLF endings, no phantom row from a trailing newline.headers(...)overrides it (then the first row is data). Short rows are padded with empty strings, extra cells dropped. Optionalparse_rowcallback over the header→cellserde_json::Map; without it each row object is the item.Transparent gzip bodies (
src/fetchers/encoding.rs, wired inclient.rs)decode_body_cappeddetects the1f 8bgzip magic (raw.xml.gz/.csv.gzfiles served withoutContent-Encoding: gzip) and decompresses before charset decoding — matching upstream's feed handling and benefitingSitemapSpider(sitemap.xml.gz) for free.max_bodyviaRead::take(limit + 1); oversized or corrupt gzip falls back to decoding the raw bytes (the exact pre-gzip behavior). New dependency:flate2(already in the tree transitively).Tests (12 new)
<link>URL, Atom viaiter_tag("entry")+ customparse_node, CSV with header row, CSV with header override +;delimiter + short/extra rows..gzbody with noContent-Encoding.All 20 test suites green locally; clippy
-D warnings, rustfmt, andcargo doc --no-depsclean.Generated by Claude Code