Capture robots.txt + llms.txt as crawl-time compliance witnesses

[MirjamOdile]

Problem

For compliance review you want the robots.txt / llms.txt as the crawl saw
them — fetched through the same Cloudflare/proxy/TLS path as the real crawl,
date-synced to the output they govern. A separate after-the-fact fetch can see
a different file (or be blocked entirely on CF-protected sites the spider
itself gets through).

Change

New ComplianceFileCapture extension (enabled via EXTENSIONS in
settings.py): when a spider opens, it schedules one request per file through
the spider's OWN downloader and writes the raw body beside the crawl output as
crawls/{robots,llms}_DDMMYYYY.txt.

Details that earn their code:

• Redirects are followed manually (capped at 4 hops), so witnesses are
  captured even when the spider runs with REDIRECT_ENABLED=False.
• HTML soft-404s are never stored: many sites serve their 200 HTML page for
  a missing file; a real robots/llms is plain text and never starts with <.
• Output paths are built from the configurable DATA_DIR (cwd-independent),
  same as the crawl JSONL export.
• Capture failures only log — they can never break a crawl.

Verification

5 unit tests: plain-text witness saved date-stamped under crawls/, HTML
soft-404 rejected, non-200 rejected, redirect re-scheduled with depth
tracking, redirect loop capped. Full unit suite green. Run live across a
15-spider project migration.

Notes

• Fires on every spider open (including --limit test crawls) — cost is two
  requests; a same-day re-run refreshes the dated file.
• Under a JOBDIR (checkpoint) run, Scrapy logs a harmless "Unable to serialize
  request" warning for the extension's callback and keeps the request in the
  memory queue — captured fine, just not resumable.