Skip to content

Make wastewater and hospital sample fallbacks fail-soft on aged samples#18

Merged
Hefrock merged 1 commit into
mainfrom
claude/kind-planck-vX3h2
Jul 7, 2026
Merged

Make wastewater and hospital sample fallbacks fail-soft on aged samples#18
Hefrock merged 1 commit into
mainfrom
claude/kind-planck-vX3h2

Conversation

@Hefrock

@Hefrock Hefrock commented Jul 7, 2026

Copy link
Copy Markdown
Owner

Summary

Follow-up to the weather fix (#17): the same aged-sample failure mode lives in the wastewater and hospital sample tiers, which clip to [start, end] the same way. They aren't failing CI yet only because their fallback tests use wide windows that still overlap the frozen sample — a latent time-bomb that fires once the committed sample ages ~365 days further and a today-relative window no longer overlaps it, silently dropping the signal to zero rows.

  • Extract the whole-year shift into src/ingestion/sample_window.py::shift_sample_to_window so all three fetchers share one tested definition; route weather through it too (removing its inline copy).
  • Gate the shift to the synthetic sample tier only, never real data:
    • hospital._load_csv gains a shift_to_window flag — True for the sample, False/default for the real MA DPH cache (Tier 1).
    • wastewater shifts in _load_sample before the shared _localize_and_window, so the real WastewaterSCAN/MWRA/CDC parsers that also use that helper keep clipping out-of-window data.
  • The planted signal is day-of-year seasonal, so a whole-year shift preserves its seasonal position; shifting naive timestamps before localizing keeps each caller's DST guards in charge.

Test plan

  • pytest tests/ -q — 112 passed (+8 new)
  • New aged-window regression tests for weather, wastewater, and hospital sample tiers (window years past the frozen sample → still non-empty and in-range)
  • Guard test that real MA DPH data (shift_to_window=False) is still clipped, never shifted
  • Helper unit tests: no-op when overlapping, shift forward/back, empty-frame safe, leap-year month/day preservation
  • End-to-end: ran python -m src.ingestion.run --city boston (default trailing-365 window, as the daily cron does) with network restricted so every fetcher hits its sample tier — all signals produced non-empty output, accumulated weather.parquet had 0 duplicate timestamps, no crashes

https://claude.ai/code/session_011FPpSio8V9rxWMCG5RnYxB


Generated by Claude Code

The weather fix (keep the sample fallback non-empty once its frozen window
ages past a today-relative request) applies verbatim to the wastewater and
hospital sample tiers, which clip the same way. They aren't failing CI yet
only because their fallback tests use wide windows that still overlap the
sample — a latent time-bomb that fires once the committed sample ages ~365
days further.

Extract the whole-year shift into src/ingestion/sample_window.py so all three
fetchers share one tested definition, and route weather through it too.
Gate it to the synthetic sample tier only: hospital._load_csv gains a
shift_to_window flag (True for the sample, False/default for the real MA DPH
cache), and wastewater shifts in _load_sample before the shared
_localize_and_window, so real-source parsers still clip out-of-window data
instead of sliding it into range.

Adds unit tests for the helper (including the leap-year month/day-preservation
case) plus aged-window regression tests for both loaders, and a guard test
that real hospital data is never shifted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011FPpSio8V9rxWMCG5RnYxB
@Hefrock
Hefrock merged commit a86dc6d into main Jul 7, 2026
1 check passed
@Hefrock
Hefrock deleted the claude/kind-planck-vX3h2 branch July 7, 2026 04:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants