feat: --discover mode for full-site seed enumeration

[Hydaspex]

Follow-up to the "scrape the entire website" question: full-site runs previously required a hand-maintained --seed URL=REGION per trust (~137 acute providers). --discover enumerates them from the live site itself. Follows on from #8.

How it works

The site is structured /<region-slug>/<trust>/. Discovery scrapes 8 pages total — the homepage plus 7 region pages — and emits the (url, region) seed pairs run_pipeline already consumes, then the normal pipeline (preflight → crawl → extract) runs unchanged.

nhs-scraper --discover --output output/full.csv
# discovered 137 trust seeds across the site
# run <id>: wrote N records to output/full.csv

What's included

pipeline/discover.py — two pure parsers + one thin async orchestrator:

• discover_region_urls(html) — homepage anchors → region URLs. An allowlist of the 7 known region slugs (not "any single path segment") keeps utility pages like /find-my-hospital/ out; deduped, host-filtered, document order preserved
• discover_trust_seeds(region_url, html) — region page anchors → (trust_url, region) seeds, where a trust link is exactly one path level below the region: excludes self links, specialty pages (two levels down), and other regions' trusts. Unknown slugs fall back to the slug as the region name — a newly added region still works, just unpretty until the map is updated
• discover_seeds(backend) — orchestrates the scrapes; raises on empty discovery (no regions / no trusts) because a silent empty seed list would produce a silently empty CSV — the exact failure mode the preflight probe exists to prevent

cli.py — --discover flag, mutually exclusive with --seed. A failed discovery exits 1 with a clear message and never falls back to DEFAULT_SEEDS — silent degradation from a full-site run to a one-trust run would be worse than failing.

Fixtures + tests (13) — synthetic homepage (7 regions + duplicate/external/utility-link traps) and South East region page (2 trusts + duplicate/self/too-deep/wrong-region traps). Tests cover both parsers, orchestration via a fake backend, empty-discovery errors, the CLI conflict path, the no-fallback failure path, and a full --discover run producing 8 golden records from 2 discovered seeds.

Verification

Parser verified in the sandbox against every trap pattern in the fixtures before pushing (dedupe, host filter, allowlist, self/deep/wrong-region exclusion, slug fallback, relative-URL resolution).

Design decisions to review

1. 8 targeted scrapes, not a BFS crawl — discovery fetches exactly homepage + 7 region pages; a crawl() from the homepage would fetch every trust and specialty page just to throw the content away.
2. Region-slug allowlist with fallback — trades "discovers new regions automatically with a pretty name" for "never mistakes a utility page for a region". The preflight probe backstops anything the allowlist gets wrong.
3. Discovery reuses scrape, not crawl — keeps the retry policy (from feat: transient-error retry policy with failure telemetry #8) applied to discovery requests for free.