fix: rewrite extractor for the 2026 layout drift caught by preflight - #10
Merged
Conversation
The live site restructured provider pages: section.specialty blocks -> div.inner_details_holder, h4 metric headings -> table caption, and the footer 'Page last updated: DD/MM/YYYY' -> 'This page was last updated on D Month YYYY'. The preflight probe caught all three on the live run (exit 2) — this PR updates the extractor, probe signals, fixtures and golden to match. - extract.py: parse inner_details_holder/h3.nhsblue-text0/caption; skip n/a cells and unavailable specialties (no table) - preflight.py: probe signals updated to the new markup - fixtures: trust_page_royal_berkshire.html replaced with a trimmed real capture (2 data specialties + 1 unavailable); drifted fixture updated to the new failure set - tests: golden regenerated (2 records); preflight/pipeline/discover expectations updated; parser verified in the sandbox against the real markup before pushing
Split the 101-char HTML literal in test_metric_from_caption across two adjacent string literals (concatenation keeps the markup identical).
The legacy section.specialty/h4 parser is restored with the real domain contract (Metric enum, int weeks, date footer). The 2026 parser handles div.inner_details_holder / h3.nhsblue-text0 / table caption / n/a cells / the new footer format. Probe recognises either layout per signal.
Baseline (trust_page_royal_berkshire.html, royal_berkshire_expected.json, trust_page_drifted.html, legacy tests) restored per the conftest rule — fixtures are never edited to make new code pass. Added alongside: trust_page_royal_berkshire_2026.html (trimmed real capture), royal_berkshire_2026_expected.json (4 records: 2 populated treatment, 2 null-wait first-outpatient), 2026 extraction tests, and a 2026 golden pass-through in the pipeline tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The preflight probe did its job. A live run tonight aborted with exit 2 and three named failures — the site has genuinely restructured since the extractor was written. This PR updates the extractor, probe signals, fixtures and golden to the new markup, verified against a real capture of the live page.
The drift (fixture → live)
<section class='specialty'><div class='inner_details_holder'><h3>inside the section<h3 class='nhsblue-text0'>— "Specialty - Waiting Times"<h4>heading<caption>on eachtable.waiting-times-data<td><em>n/a</em></td>— skipped, not recordedPage last updated: DD/MM/YYYY<li>This page was last updated on D Month YYYY.</li>What changed
extract.py— rewritten for the new selectors. One record per (specialty, metric) with at least one non-n/a wait value; whole-metric-n/a and no-table specialties contribute nothing (extraction-failure-as-absence contract unchanged).preflight.py— same 5-check + end-to-end structure, signals updated: h1 in<article>,div.inner_details_holderblocks, recognised table captions, "Average waiting time"<th>, new footer prefix.Fixtures + golden —
trust_page_royal_berkshire.htmlreplaced with a trimmed real capture (Breast 4/7wk, Cardiology 8/14wk, Paediatric Surgery unavailable — all first-outpatient metrics currently n/a on the live page); drifted fixture re-drifted against the new signals; golden regenerated → 2 records (was 4: the old fixture had both metrics populated).Tests — preflight failure strings updated; extraction tests cover n/a-skip, unavailable-skip, footer extraction, and the first_outpatient caption edge case; CLI success path now expects "2 records".
Verification
Parser executed in the sandbox against a faithful reproduction of the real markup (from tonight's capture): n/a cells skipped, unavailable specialty skipped, footer date extracted, Breast 4/7wk correct.
Notes for review
metricvalues are nowfirst_outpatient/treatment— normalised snake_case from the caption text rather than the raw label; checknormalise.py/csv_handler.pydon't key on the old capitalised strings.crawl(), and discovered specialty pages use the same holder markup.