Skip to content

crawl-all: enqueue full crawls via Pueue instead of running inline#8

Open
MirjamOdile wants to merge 1 commit into
mainfrom
pr/06-crawl-all-pueue
Open

crawl-all: enqueue full crawls via Pueue instead of running inline#8
MirjamOdile wants to merge 1 commit into
mainfrom
pr/06-crawl-all-pueue

Conversation

@MirjamOdile

Copy link
Copy Markdown

Problem

Since production crawls auto-detach into Pueue, a single scrapai crawl is
parallel-safe and disconnect-safe — but crawl-all deliberately bypassed that
(detached=True forced every spider inline). So "run the whole project" meant:
strictly sequential, one spider at a time, bound to the terminal, and the whole
batch dies on Ctrl+C or an SSH drop. A 15-spider project would block a terminal
for hours with no resumability.

Change

crawl_all now calls each spider through the SAME path as a single crawl
(detached=False): a full run submits itself to Pueue and returns, so the
whole project is queued in seconds, runs at whatever concurrency the Pueue
group is set to, and survives disconnects. --limit N keeps the old inline
sequential behaviour (test crawls are foreground by design). A closing hint
points at ./scrapai crawl-status / pueue status.

Verification

  • test_crawl_all_enqueues_each_spider — every spider goes through the detach
    path (detached=False), the queue hint prints.
  • test_crawl_all_with_limit_stays_inline--limit runs stay inline, no
    queue hint.
  • Full unit suite green; used live to queue a 15-spider project.

Trade-offs

  • Pueue becomes required for crawl-all full runs (it already was for any
    single full crawl; the error message is the same).
  • Sequential-inline behaviour, if ever wanted, is one pueue parallel 1 away.
  • If a pueue add fails mid-loop, spiders after the failure aren't queued
    (the CLI exits with the same clear error a single crawl gives).

Since production crawls auto-detach into Pueue, a single `scrapai
crawl` is parallel-safe and disconnect-safe — but crawl-all bypassed
that (detached=True forced every spider inline), so "run the whole
project" meant strictly sequential, one spider at a time, bound to the
terminal, and the whole batch died on Ctrl+C or an SSH drop.

crawl_all now calls each spider through the same path as a single
crawl: a full run submits itself to Pueue and returns, so the whole
project is queued in seconds, runs parallel (Pueue group default), and
survives disconnects. --limit keeps the old inline sequential
behaviour (test crawls are foreground by design). A closing hint
points at ./scrapai crawl-status.

See docs/requests/14-crawl-all-pueue.md (ships with the quality tool
PR) for the full write-up.
@iRanadheer iRanadheer force-pushed the pr/06-crawl-all-pueue branch from 5f4aae6 to 3148a76 Compare July 10, 2026 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant