crawl-all: enqueue full crawls via Pueue instead of running inline#8
Open
MirjamOdile wants to merge 1 commit into
Open
crawl-all: enqueue full crawls via Pueue instead of running inline#8MirjamOdile wants to merge 1 commit into
MirjamOdile wants to merge 1 commit into
Conversation
Since production crawls auto-detach into Pueue, a single `scrapai crawl` is parallel-safe and disconnect-safe — but crawl-all bypassed that (detached=True forced every spider inline), so "run the whole project" meant strictly sequential, one spider at a time, bound to the terminal, and the whole batch died on Ctrl+C or an SSH drop. crawl_all now calls each spider through the same path as a single crawl: a full run submits itself to Pueue and returns, so the whole project is queued in seconds, runs parallel (Pueue group default), and survives disconnects. --limit keeps the old inline sequential behaviour (test crawls are foreground by design). A closing hint points at ./scrapai crawl-status. See docs/requests/14-crawl-all-pueue.md (ships with the quality tool PR) for the full write-up.
5f4aae6 to
3148a76
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Since production crawls auto-detach into Pueue, a single
scrapai crawlisparallel-safe and disconnect-safe — but
crawl-alldeliberately bypassed that(
detached=Trueforced every spider inline). So "run the whole project" meant:strictly sequential, one spider at a time, bound to the terminal, and the whole
batch dies on Ctrl+C or an SSH drop. A 15-spider project would block a terminal
for hours with no resumability.
Change
crawl_allnow calls each spider through the SAME path as a singlecrawl(
detached=False): a full run submits itself to Pueue and returns, so thewhole project is queued in seconds, runs at whatever concurrency the Pueue
group is set to, and survives disconnects.
--limit Nkeeps the old inlinesequential behaviour (test crawls are foreground by design). A closing hint
points at
./scrapai crawl-status/pueue status.Verification
test_crawl_all_enqueues_each_spider— every spider goes through the detachpath (
detached=False), the queue hint prints.test_crawl_all_with_limit_stays_inline—--limitruns stay inline, noqueue hint.
Trade-offs
crawl-allfull runs (it already was for anysingle full crawl; the error message is the same).
pueue parallel 1away.pueue addfails mid-loop, spiders after the failure aren't queued(the CLI exits with the same clear error a single crawl gives).