Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions .claude/skills/ddrs-dev/references/research-status.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,8 +279,15 @@ was invalidated by a stale binary and the manifest did not reveal it.
**Convention change SHIPPED (2026-08-08, findings §5i):** tau is now
signed-at-zero hours of advance (`[tau : -(24-tau)]`, day i ↔ obs day i,
default 9 ≡ old 20); old scale = new + 11. Old checkpoints trained at
old-3 ≡ new −8. tau=9 CPU retrain across all five stores launched same
day (`scripts/run_tau9_source_trains.sh`).
old-3 ≡ new −8. **Retrain COMPLETE (2026-08-10, all five stores, 30
epochs, CPU, 1,841 gauges, full test window): aorc2f distributed
0.620 → 0.706 median NSE (+0.086 from timing alone; now beats the
summed-q' baseline 0.642 by +0.064), UH retro 0.707, daily LSTM 0.578,
hourly LSTM 0.564, aorc2f lumped 0.483 (WORSE, as pre-registered — its
optimum is ≈ −8; the negative control behaved).** Merged to master in
PR #33. Gotcha: a train-and-test resume from a final checkpoint cannot
re-enter Phase 2 (needs checkpoints from its own Phase 1) — finish a
killed eval with the legacy eval binary instead.
**Gamma-UH params pulled (§5h):** the distributed aorc2f store's q' was
exported (2026-07-29, water_loss) with each divide routed through its own
learned gamma UH; `scripts/dump_gamma_uh_params.py` (water_loss venv)
Expand Down
29 changes: 19 additions & 10 deletions docs/2026-08-06-tau-sweep-pilot-findings.md
Original file line number Diff line number Diff line change
Expand Up @@ -509,8 +509,8 @@ Caveat noted at launch: the aorc2f_lumped store's measured optimum is
new-convention ≈ −8 (§5e), so tau=9 is expected to HURT that arm; it runs
anyway as the consistency control.

**Retrain results (2026-08-09, full test window 1995-10..2010-09, 1,841
gauges, 30 epochs each):**
**Retrain results — FINAL, all five arms (2026-08-09/10, full test window
1995-10..2010-09, 1,841 gauges, 30 epochs each, CPU):**

| arm | median NSE | median KGE |
|---|---|---|
Expand All @@ -519,17 +519,26 @@ gauges, 30 epochs each):**
| **tau=9 aorc2f distributed** | **0.706** | **0.730** |
| **tau=9 UH retrospective** | **0.707** | **0.738** |
| tau=9 daily LSTM | 0.578 | 0.616 |
| tau=9 hourly LSTM (native hourly) | 0.564 | 0.500 |
| tau=9 aorc2f lumped (consistency control) | 0.483 | 0.504 |

The timing fix alone is worth **+0.086 median NSE** on the flagship arm
and flips routing from LOSING to the summed-q' baseline (0.620 vs 0.642)
to beating it by +0.064. Run IDs: `2026-08-09T{03-05-54,09-30-39,12-05-08}Z-
train-and-test`. Operational note: the hourly_lstm arm finished training
(run `2026-08-09T14-55-05Z`, 60 checkpoints) but its eval phase was killed
with the parent driver at chunk 49/366; a train-and-test resume from the
final checkpoint cannot re-enter Phase 2 (it requires checkpoints written
by its own Phase 1 — `tau9_train_hourly_lstm_evalresume.yaml` records the
failed attempt), so the arm completes via the legacy eval binary
(`scripts/run_tau9_hourly_eval_chain.sh`), chained behind the
to beating it by +0.064. Every arm ordered exactly as the eval-only
sweeps predicted (§5e), including the negative control: the lumped store
(measured optimum ≈ −8 on the new scale) got WORSE at tau=9 (0.510
eval-only at old tau → 0.483 trained at tau=9) — the one arm the theory
said the fix should hurt, hurt. Run IDs:
`2026-08-09T{03-05-54,09-30-39,12-05-08,14-55-05,22-13-43}Z-train-and-test`.

Operational note: the hourly_lstm arm finished training (run
`2026-08-09T14-55-05Z`, 60 checkpoints) but its eval phase was killed with
the parent driver at chunk 49/366; a train-and-test resume from the final
checkpoint cannot re-enter Phase 2 (it requires checkpoints written by its
own Phase 1 — `tau9_train_hourly_lstm_evalresume.yaml` records the failed
attempt), so its numbers above come from the legacy eval binary against
the epoch-30 checkpoint (`scripts/run_tau9_hourly_eval_chain.sh`,
`output/tau_sweep/train9_hourly_lstm_eval.{log,zarr}`), chained behind the
aorc2f_lumped run (`scripts/run_tau9_remaining.sh`).

## 6. Raw output
Expand Down
13 changes: 13 additions & 0 deletions src/training/loss.rs
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,19 @@ use crate::config::{LossConfig, LossKind};
/// `compute_daily_runoff` still uses the legacy form. The total trim is
/// 24 h under both conventions, so `T_days = T_hours/24 - 1` is unchanged.
///
/// Why the last day is excluded: a window advanced by `tau` needs `tau`
/// hours of routed output past the final store day, which were never
/// routed (the q' feeding them is outside the window). Dropping exactly
/// one day — front trim `tau` + back trim `24 - tau` — makes the SAME
/// `T_days` days part of the training/testing set at every tau: tensor
/// shapes, the obs pairing, and NSE(tau) sweeps all score an identical
/// day sample regardless of the shift, and the trimmed length stays a
/// multiple of 24 so pooling is an exact block mean. The alternative
/// (score all days by routing one extra day of q' per window) is ~1% more
/// training signal and needs +1-day window plumbing; deliberately not
/// done — see the 2026-08-09 discussion in
/// `docs/2026-08-06-tau-sweep-pilot-findings.md` §5i.
///
/// Returns `(G, T_days)` where `T_days = T_hours_trimmed // 24`.
pub fn tau_trim_and_downsample<B: Backend>(
predictions_hourly: Tensor<B, 2>,
Expand Down
Loading