Skip to content

feat: pipeline orchestration, normalisation and CSV output - #5

Merged
Hydaspex merged 2 commits into
mainfrom
feat/pipeline-orchestration
Aug 8, 2026
Merged

feat: pipeline orchestration, normalisation and CSV output#5
Hydaspex merged 2 commits into
mainfrom
feat/pipeline-orchestration

Conversation

@Hydaspex

@Hydaspex Hydaspex commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Completes the runnable pipeline: crawl → extract → normalise → persist, with provenance — all driven through the CrawlBackend port, so the whole chain is testable offline with a fake backend. Follows on from #4.

What's included

pipeline/normalise.pynormalise_records: dedupe on identity (region, provider, specialty, metric) (first occurrence wins — callers order sources by trustworthiness) and deterministic sort (region, provider, specialty, metric) with outpatient before treatment. Stable ordering makes run-over-run diffs meaningful. Region is part of identity: the same trust surfaced under two region seeds remains two records.

pipeline/run.py — async run_pipeline(backend, seeds, options):

  • Seed = (url, region); the region is threaded into every record discovered under that URL
  • CrawlRun provenance captured automatically (backend from the concrete class name, uuid4 run_id, tz-aware started_at)
  • frozen PipelineResult(run, records)
  • extractor failures surface as absent records (its contract); backend failures propagate loudly

io/csv_handler.pywrite_records_csv with the fixed COLUMNS contract (versioned with the golden schema): None → empty cell, empty input → header-only file, parent dirs created.

cli.pynhs-scraper entry point: --backend, repeatable --seed URL=REGION, --output, --limit, --max-depth; backend constructed lazily so importing the CLI never pulls crawl dependencies.

Tests (24 new)

  • test_normalise.py — first-wins dedupe, distinct metrics kept, multi-region identity regression, sort order, empty input
  • test_pipeline_run.pykeystone: the full crawl → extract → normalise chain over the characterisation fixture reproduces the golden dataset exactly, plus provenance, region threading, multi-seed aggregation, empty-seed rejection, empty crawl
  • test_csv_handler.py — column contract, None → empty, header-only, nested dirs
  • test_cli.py — seed parsing (incl. = in values), arg defaults, repeated seeds, unknown/missing-backend errors

pyproject.toml gains [project.scripts] nhs-scraper.

Verification

Pipeline (golden-order output, dedupe/sort, region threading, edge cases) and CSV handler (column order, None handling, empty case) both exercised locally in the sandbox — all green.

Design decisions to review

  1. Identity includes region (corrected) — the initial commit keyed dedupe on (provider, specialty, metric); test_multiple_seeds_aggregate caught identical trust records under two region seeds collapsing 8 → 4. Identity is now (region, provider, specialty, metric), with a regression test. Duplicates only collapse within identical identity.
  2. Backend name from type(backend).__name__ — zero-config provenance; alternatives (explicit name field) add plumbing for little gain.
  3. CLI defaults to a single example seed — deliberate: running bare nhs-scraper is a smoke test, not a full crawl.

- normalise_records: dedupe on (provider, specialty, metric), first
  occurrence wins; deterministic sort (region, provider, specialty,
  metric) — verified locally including duplicate-first-wins
- run_pipeline: async orchestration crawl -> extract -> normalise with
  CrawlRun provenance; PipelineResult frozen dataclass
- write_records_csv: fixed column order, None -> empty cell, header-only
  on empty input, creates parent dirs (verified locally)
- nhs-scraper CLI entry point: --backend/--seed URL=REGION/--output/
  --limit/--max-depth
- Fake-backend pipeline integration tests reproduce the golden dataset
  end-to-end, offline
test_multiple_seeds_aggregate exposed a real flaw: identity was
(provider, specialty, metric), so identical trust records crawled under
two different region seeds were collapsed 8 -> 4. Identity is now
(region, provider, specialty, metric), matching the sort key and the
documented intent. Adds a multi-region regression test.
@Hydaspex
Hydaspex merged commit b9ad541 into main Aug 8, 2026
3 checks passed
@Hydaspex
Hydaspex deleted the feat/pipeline-orchestration branch August 8, 2026 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant