Make wastewater and hospital sample fallbacks fail-soft on aged samples#18
Merged
Conversation
The weather fix (keep the sample fallback non-empty once its frozen window ages past a today-relative request) applies verbatim to the wastewater and hospital sample tiers, which clip the same way. They aren't failing CI yet only because their fallback tests use wide windows that still overlap the sample — a latent time-bomb that fires once the committed sample ages ~365 days further. Extract the whole-year shift into src/ingestion/sample_window.py so all three fetchers share one tested definition, and route weather through it too. Gate it to the synthetic sample tier only: hospital._load_csv gains a shift_to_window flag (True for the sample, False/default for the real MA DPH cache), and wastewater shifts in _load_sample before the shared _localize_and_window, so real-source parsers still clip out-of-window data instead of sliding it into range. Adds unit tests for the helper (including the leap-year month/day-preservation case) plus aged-window regression tests for both loaders, and a guard test that real hospital data is never shifted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011FPpSio8V9rxWMCG5RnYxB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to the weather fix (#17): the same aged-sample failure mode lives in the wastewater and hospital sample tiers, which clip to
[start, end]the same way. They aren't failing CI yet only because their fallback tests use wide windows that still overlap the frozen sample — a latent time-bomb that fires once the committed sample ages ~365 days further and a today-relative window no longer overlaps it, silently dropping the signal to zero rows.src/ingestion/sample_window.py::shift_sample_to_windowso all three fetchers share one tested definition; routeweatherthrough it too (removing its inline copy).hospital._load_csvgains ashift_to_windowflag —Truefor the sample,False/default for the real MA DPH cache (Tier 1).wastewatershifts in_load_samplebefore the shared_localize_and_window, so the real WastewaterSCAN/MWRA/CDC parsers that also use that helper keep clipping out-of-window data.Test plan
pytest tests/ -q— 112 passed (+8 new)shift_to_window=False) is still clipped, never shiftedpython -m src.ingestion.run --city boston(default trailing-365 window, as the daily cron does) with network restricted so every fetcher hits its sample tier — all signals produced non-empty output, accumulatedweather.parquethad 0 duplicate timestamps, no crasheshttps://claude.ai/code/session_011FPpSio8V9rxWMCG5RnYxB
Generated by Claude Code