Skip to content

feat: row-level parity diff tooling - #6

Merged
Hydaspex merged 1 commit into
mainfrom
feat/parity-diff
Aug 8, 2026
Merged

feat: row-level parity diff tooling#6
Hydaspex merged 1 commit into
mainfrom
feat/parity-diff

Conversation

@Hydaspex

@Hydaspex Hydaspex commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Final roadmap increment: row-level added/removed/changed reporting between record sets — usable for run-over-run weekly comparisons and, later, backend parity checks. Follows on from #5.

What's included

pipeline/diff.py

  • diff_records(old, new) -> ParityReport — keys records on the same identity as normalise_records (region, provider, specialty, metric) and classifies every key as added / removed / changed / unchanged. Pure and deterministic (sorted-key output).
  • RecordChangekind, key, old, new, and changed_fields with field-level detail over source_url, average_wait_weeks, patients_seen_within_weeks, page_last_updated. A None → value transition counts as a change.
  • ParityReportis_identical, total_changes, summary().
  • format_change — one human-readable line per change (+/-/~ with old → new values).

cli.py — new nhs-scraper-diff OLD NEW entry point: prints the summary and per-change lines; exit 0 when identical, 1 when changes found, so a scheduled workflow can gate on waiting-time movement.

io/csv_handler.py

  • read_records_csv — strict header validation against the canonical COLUMNS (loud failure on foreign or moved schemas), None round-trip, tolerates legacy 8.0 float formatting, rejects non-integral week values.
  • Bug fix found during sandbox verification: week columns mixed int/None previously became float64 and serialised as 8.0 in the CSV — now cast to pandas nullable Int64, so output is always plain integers. Regression test added (test_integer_columns_never_serialise_as_floats).

Tests (12 new diff + 6 new CSV) — identical/added/removed/changed/None-transition/mixed scenarios, format_change rendering, CLI end-to-end via capsys (exit codes + output), reader round-trip/None/legacy/header/non-integral cases.

Verification

Diff engine (all classifications, field detail, None transitions), the Int64 writer fix, and the reader were all executed in the sandbox — green. The float-serialisation bug was caught by that verification, which is exactly what it's for.

Design decisions to review

  1. Same identity for diff and normalise — one definition of "same record" across the codebase; changing it changes both.
  2. Exit-code contract (0 identical / 1 changes) — makes nhs-scraper-diff composable in CI without parsing output.
  3. Strict header validation on read — a schema move or foreign file fails loudly rather than silently mis-parsing.

- diff_records: key on (region, provider, specialty, metric) — same
  identity as normalise — and classify added/removed/changed with
  field-level detail; frozen ParityReport with is_identical,
  total_changes and summary()
- nhs-scraper-diff CLI: exit 0 when identical, 1 when changes found,
  so it can gate CI comparisons
- read_records_csv: strict header validation, None round-trip
- fix: week columns now write as pandas Int64 — mixed int/None columns
  previously serialised as floats (8.0); reader tolerates legacy float
  formatting and rejects non-integral values
- diff engine, writer fix and reader all verified in the sandbox
@Hydaspex
Hydaspex merged commit febe080 into main Aug 8, 2026
3 checks passed
@Hydaspex
Hydaspex deleted the feat/parity-diff branch August 8, 2026 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant