feat: row-level parity diff tooling - #6
Merged
Merged
Conversation
- diff_records: key on (region, provider, specialty, metric) — same identity as normalise — and classify added/removed/changed with field-level detail; frozen ParityReport with is_identical, total_changes and summary() - nhs-scraper-diff CLI: exit 0 when identical, 1 when changes found, so it can gate CI comparisons - read_records_csv: strict header validation, None round-trip - fix: week columns now write as pandas Int64 — mixed int/None columns previously serialised as floats (8.0); reader tolerates legacy float formatting and rejects non-integral values - diff engine, writer fix and reader all verified in the sandbox
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Final roadmap increment: row-level added/removed/changed reporting between record sets — usable for run-over-run weekly comparisons and, later, backend parity checks. Follows on from #5.
What's included
pipeline/diff.pydiff_records(old, new) -> ParityReport— keys records on the same identity asnormalise_records(region, provider, specialty, metric)and classifies every key as added / removed / changed / unchanged. Pure and deterministic (sorted-key output).RecordChange—kind,key,old,new, andchanged_fieldswith field-level detail oversource_url,average_wait_weeks,patients_seen_within_weeks,page_last_updated. ANone→ value transition counts as a change.ParityReport—is_identical,total_changes,summary().format_change— one human-readable line per change (+/-/~with old → new values).cli.py— newnhs-scraper-diff OLD NEWentry point: prints the summary and per-change lines; exit 0 when identical, 1 when changes found, so a scheduled workflow can gate on waiting-time movement.io/csv_handler.pyread_records_csv— strict header validation against the canonicalCOLUMNS(loud failure on foreign or moved schemas),Noneround-trip, tolerates legacy8.0float formatting, rejects non-integral week values.int/Nonepreviously became float64 and serialised as8.0in the CSV — now cast to pandas nullableInt64, so output is always plain integers. Regression test added (test_integer_columns_never_serialise_as_floats).Tests (12 new diff + 6 new CSV) — identical/added/removed/changed/None-transition/mixed scenarios,
format_changerendering, CLI end-to-end viacapsys(exit codes + output), reader round-trip/None/legacy/header/non-integral cases.Verification
Diff engine (all classifications, field detail, None transitions), the Int64 writer fix, and the reader were all executed in the sandbox — green. The float-serialisation bug was caught by that verification, which is exactly what it's for.
Design decisions to review
nhs-scraper-diffcomposable in CI without parsing output.