feat: pipeline orchestration, normalisation and CSV output - #5
Merged
Conversation
- normalise_records: dedupe on (provider, specialty, metric), first occurrence wins; deterministic sort (region, provider, specialty, metric) — verified locally including duplicate-first-wins - run_pipeline: async orchestration crawl -> extract -> normalise with CrawlRun provenance; PipelineResult frozen dataclass - write_records_csv: fixed column order, None -> empty cell, header-only on empty input, creates parent dirs (verified locally) - nhs-scraper CLI entry point: --backend/--seed URL=REGION/--output/ --limit/--max-depth - Fake-backend pipeline integration tests reproduce the golden dataset end-to-end, offline
test_multiple_seeds_aggregate exposed a real flaw: identity was (provider, specialty, metric), so identical trust records crawled under two different region seeds were collapsed 8 -> 4. Identity is now (region, provider, specialty, metric), matching the sort key and the documented intent. Adds a multi-region regression test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the runnable pipeline: crawl → extract → normalise → persist, with provenance — all driven through the
CrawlBackendport, so the whole chain is testable offline with a fake backend. Follows on from #4.What's included
pipeline/normalise.py—normalise_records: dedupe on identity(region, provider, specialty, metric)(first occurrence wins — callers order sources by trustworthiness) and deterministic sort(region, provider, specialty, metric)with outpatient before treatment. Stable ordering makes run-over-run diffs meaningful. Region is part of identity: the same trust surfaced under two region seeds remains two records.pipeline/run.py— asyncrun_pipeline(backend, seeds, options):Seed = (url, region); the region is threaded into every record discovered under that URLCrawlRunprovenance captured automatically (backendfrom the concrete class name, uuid4run_id, tz-awarestarted_at)PipelineResult(run, records)io/csv_handler.py—write_records_csvwith the fixedCOLUMNScontract (versioned with the golden schema):None→ empty cell, empty input → header-only file, parent dirs created.cli.py—nhs-scraperentry point:--backend, repeatable--seed URL=REGION,--output,--limit,--max-depth; backend constructed lazily so importing the CLI never pulls crawl dependencies.Tests (24 new)
test_normalise.py— first-wins dedupe, distinct metrics kept, multi-region identity regression, sort order, empty inputtest_pipeline_run.py— keystone: the full crawl → extract → normalise chain over the characterisation fixture reproduces the golden dataset exactly, plus provenance, region threading, multi-seed aggregation, empty-seed rejection, empty crawltest_csv_handler.py— column contract,None→ empty, header-only, nested dirstest_cli.py— seed parsing (incl.=in values), arg defaults, repeated seeds, unknown/missing-backend errorspyproject.tomlgains[project.scripts] nhs-scraper.Verification
Pipeline (golden-order output, dedupe/sort, region threading, edge cases) and CSV handler (column order,
Nonehandling, empty case) both exercised locally in the sandbox — all green.Design decisions to review
(provider, specialty, metric);test_multiple_seeds_aggregatecaught identical trust records under two region seeds collapsing 8 → 4. Identity is now(region, provider, specialty, metric), with a regression test. Duplicates only collapse within identical identity.type(backend).__name__— zero-config provenance; alternatives (explicit name field) add plumbing for little gain.nhs-scraperis a smoke test, not a full crawl.