From 8140f0d70147367e4fb80fd7521388fe2c77c9ff Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Sun, 9 Aug 2026 18:28:38 -0400 Subject: [PATCH 1/2] docs(tau): comment why the slice excludes the last day Constant one-day trim (front tau + back 24-tau) keeps the same T_days in the training/testing set at every tau: identical day sample across sweeps, tau-independent tensor shapes, and exact 24-hour block pooling. Scoring all days would need one extra routed day of q' per window (~1% signal), deliberately deferred. Co-Authored-By: Claude Fable 5 --- src/training/loss.rs | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/src/training/loss.rs b/src/training/loss.rs index 3938340..d8a03a9 100644 --- a/src/training/loss.rs +++ b/src/training/loss.rs @@ -29,6 +29,19 @@ use crate::config::{LossConfig, LossKind}; /// `compute_daily_runoff` still uses the legacy form. The total trim is /// 24 h under both conventions, so `T_days = T_hours/24 - 1` is unchanged. /// +/// Why the last day is excluded: a window advanced by `tau` needs `tau` +/// hours of routed output past the final store day, which were never +/// routed (the q' feeding them is outside the window). Dropping exactly +/// one day — front trim `tau` + back trim `24 - tau` — makes the SAME +/// `T_days` days part of the training/testing set at every tau: tensor +/// shapes, the obs pairing, and NSE(tau) sweeps all score an identical +/// day sample regardless of the shift, and the trimmed length stays a +/// multiple of 24 so pooling is an exact block mean. The alternative +/// (score all days by routing one extra day of q' per window) is ~1% more +/// training signal and needs +1-day window plumbing; deliberately not +/// done — see the 2026-08-09 discussion in +/// `docs/2026-08-06-tau-sweep-pilot-findings.md` §5i. +/// /// Returns `(G, T_days)` where `T_days = T_hours_trimmed // 24`. pub fn tau_trim_and_downsample( predictions_hourly: Tensor, From c471130d6cb1a2cf4032c0e584ad2aa9e44f512e Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Mon, 10 Aug 2026 03:21:06 -0400 Subject: [PATCH 2/2] =?UTF-8?q?docs(tau):=20final=20five-arm=20tau=3D9=20r?= =?UTF-8?q?etrain=20table=20=E2=80=94=20negative=20control=20confirms?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All arms complete: aorc2f distributed 0.706, UH retro 0.707, daily LSTM 0.578, hourly LSTM 0.564 (legacy-eval path), aorc2f lumped 0.483. The lumped arm regressing at tau=9 is the pre-registered prediction (its optimum is ~-8 on the new scale) — the one arm the fix should hurt, hurt. Adds the Phase-2 resume gotcha to the skill reference. Co-Authored-By: Claude Fable 5 --- .../ddrs-dev/references/research-status.md | 11 +++++-- docs/2026-08-06-tau-sweep-pilot-findings.md | 29 ++++++++++++------- 2 files changed, 28 insertions(+), 12 deletions(-) diff --git a/.claude/skills/ddrs-dev/references/research-status.md b/.claude/skills/ddrs-dev/references/research-status.md index c812ed9..5a24013 100644 --- a/.claude/skills/ddrs-dev/references/research-status.md +++ b/.claude/skills/ddrs-dev/references/research-status.md @@ -279,8 +279,15 @@ was invalidated by a stale binary and the manifest did not reveal it. **Convention change SHIPPED (2026-08-08, findings §5i):** tau is now signed-at-zero hours of advance (`[tau : -(24-tau)]`, day i ↔ obs day i, default 9 ≡ old 20); old scale = new + 11. Old checkpoints trained at - old-3 ≡ new −8. tau=9 CPU retrain across all five stores launched same - day (`scripts/run_tau9_source_trains.sh`). + old-3 ≡ new −8. **Retrain COMPLETE (2026-08-10, all five stores, 30 + epochs, CPU, 1,841 gauges, full test window): aorc2f distributed + 0.620 → 0.706 median NSE (+0.086 from timing alone; now beats the + summed-q' baseline 0.642 by +0.064), UH retro 0.707, daily LSTM 0.578, + hourly LSTM 0.564, aorc2f lumped 0.483 (WORSE, as pre-registered — its + optimum is ≈ −8; the negative control behaved).** Merged to master in + PR #33. Gotcha: a train-and-test resume from a final checkpoint cannot + re-enter Phase 2 (needs checkpoints from its own Phase 1) — finish a + killed eval with the legacy eval binary instead. **Gamma-UH params pulled (§5h):** the distributed aorc2f store's q' was exported (2026-07-29, water_loss) with each divide routed through its own learned gamma UH; `scripts/dump_gamma_uh_params.py` (water_loss venv) diff --git a/docs/2026-08-06-tau-sweep-pilot-findings.md b/docs/2026-08-06-tau-sweep-pilot-findings.md index 1b09649..6c64c37 100644 --- a/docs/2026-08-06-tau-sweep-pilot-findings.md +++ b/docs/2026-08-06-tau-sweep-pilot-findings.md @@ -509,8 +509,8 @@ Caveat noted at launch: the aorc2f_lumped store's measured optimum is new-convention ≈ −8 (§5e), so tau=9 is expected to HURT that arm; it runs anyway as the consistency control. -**Retrain results (2026-08-09, full test window 1995-10..2010-09, 1,841 -gauges, 30 epochs each):** +**Retrain results — FINAL, all five arms (2026-08-09/10, full test window +1995-10..2010-09, 1,841 gauges, 30 epochs each, CPU):** | arm | median NSE | median KGE | |---|---|---| @@ -519,17 +519,26 @@ gauges, 30 epochs each):** | **tau=9 aorc2f distributed** | **0.706** | **0.730** | | **tau=9 UH retrospective** | **0.707** | **0.738** | | tau=9 daily LSTM | 0.578 | 0.616 | +| tau=9 hourly LSTM (native hourly) | 0.564 | 0.500 | +| tau=9 aorc2f lumped (consistency control) | 0.483 | 0.504 | The timing fix alone is worth **+0.086 median NSE** on the flagship arm and flips routing from LOSING to the summed-q' baseline (0.620 vs 0.642) -to beating it by +0.064. Run IDs: `2026-08-09T{03-05-54,09-30-39,12-05-08}Z- -train-and-test`. Operational note: the hourly_lstm arm finished training -(run `2026-08-09T14-55-05Z`, 60 checkpoints) but its eval phase was killed -with the parent driver at chunk 49/366; a train-and-test resume from the -final checkpoint cannot re-enter Phase 2 (it requires checkpoints written -by its own Phase 1 — `tau9_train_hourly_lstm_evalresume.yaml` records the -failed attempt), so the arm completes via the legacy eval binary -(`scripts/run_tau9_hourly_eval_chain.sh`), chained behind the +to beating it by +0.064. Every arm ordered exactly as the eval-only +sweeps predicted (§5e), including the negative control: the lumped store +(measured optimum ≈ −8 on the new scale) got WORSE at tau=9 (0.510 +eval-only at old tau → 0.483 trained at tau=9) — the one arm the theory +said the fix should hurt, hurt. Run IDs: +`2026-08-09T{03-05-54,09-30-39,12-05-08,14-55-05,22-13-43}Z-train-and-test`. + +Operational note: the hourly_lstm arm finished training (run +`2026-08-09T14-55-05Z`, 60 checkpoints) but its eval phase was killed with +the parent driver at chunk 49/366; a train-and-test resume from the final +checkpoint cannot re-enter Phase 2 (it requires checkpoints written by its +own Phase 1 — `tau9_train_hourly_lstm_evalresume.yaml` records the failed +attempt), so its numbers above come from the legacy eval binary against +the epoch-30 checkpoint (`scripts/run_tau9_hourly_eval_chain.sh`, +`output/tau_sweep/train9_hourly_lstm_eval.{log,zarr}`), chained behind the aorc2f_lumped run (`scripts/run_tau9_remaining.sh`). ## 6. Raw output