From b5c302b6fe0367e808429c4ccf5122686698b44c Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Wed, 1 Jul 2026 21:05:50 -0400 Subject: [PATCH 01/41] docs: spec for NH Q' store contract, hourly-native reading, ddrs import Co-Authored-By: Claude Fable 5 --- .../2026-07-01-nh-qprime-import-design.md | 185 ++++++++++++++++++ 1 file changed, 185 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-01-nh-qprime-import-design.md diff --git a/docs/superpowers/specs/2026-07-01-nh-qprime-import-design.md b/docs/superpowers/specs/2026-07-01-nh-qprime-import-design.md new file mode 100644 index 0000000..e854da9 --- /dev/null +++ b/docs/superpowers/specs/2026-07-01-nh-qprime-import-design.md @@ -0,0 +1,185 @@ +# NH → icechunk → route: Q' store contract, hourly-native reading, and `ddrs import` + +**Date:** 2026-07-01 +**Status:** Approved (brainstorming session) +**Scope decision trail:** full pipeline (NH outputs → icechunk distributed Q' → route); +both existing LSTM stores in scope including hourly-native routing; NH-side forward +step stays in `~/projects/neuralhydrology` (it runs NH); ddrs gains an `import` +module; success = plumbing verified + short smoke train on each store; +hourly reading via resolution sniffing (approach A). + +## Problem + +ddr trained routing on top of neural hydrology (NH) LSTM outputs by an offline +pattern: forward the trained LSTM once over ~288K MERIT unit catchments +(`~/projects/neuralhydrology/examples/merit_hydro/forward_merit.py`), write +`Qr(divide_id, time)` in m³/s to an icechunk store, and point +`data_sources.streamflow` at it. ddrs must support the same workflow so each +new NH dataset becomes routable. + +What exists today — `/mnt/ssd1/data/icechunk/` holds **all** unit-catchment +forward outputs, and every one of them is an import target: + +| Store | Size | Resolution | Producer | +|---|---|---|---| +| `daily_lstm_merit_unit_catchments.ic` | 26 GB | daily | CudaLSTM run `merit_hydro_daily_cudalstm_1004_213620`, `forward_merit.py --mode daily` | +| `hourly_lstm_merit_unit_catchments.ic` | 257 GB | hourly | MTS-LSTM run `merit_hydro_mtslstm_1104_163854`, `forward_merit.py --mode hourly` | +| `daily_dhbv2_merit_unit_catchments.ic` | 17 GB | daily | dHBV2 unit-catchment forward | +| `merit_dhbv2_UH_retrospective.ic` | 9.5 GB | daily | dHBV2 UH retrospective (current ddrs training source) | + +Gaps in ddrs: + +1. `StreamflowStore` (icechunk reader) assumes a **daily** CF time axis and + upsamples via repeat-24 or the disaggregation head. The hourly store has an + hourly time axis and no read path. +2. No first-class way to validate + register a new Q' store; today it means + hand-editing `ddrs.yaml` or a source-group file. +3. The producer/consumer interface is implicit (whatever `forward_merit.py` + happens to write). + +## Design + +### 1. Store contract (`docs/nh-qprime-store-contract.md`) + +A new doc codifies what any NH forward script must emit for ddrs to route it: + +- icechunk repo, `main` branch, root group, one variable **`Qr(divide_id, time)`**, + f32, attr `units: m^3/s`. +- `divide_id`: int64 MERIT COMIDs. Values are the **local** lateral inflow per + unit catchment (no upstream accumulation — routing does that). +- `time`: CF-encoded int, **`days since …`** (daily) or **`hours since …`** + (hourly), contiguous, no gaps. +- Values strictly positive; producer floors NaN/negatives to 1e-6 + (as `forward_merit.py` already does). +- COMIDs absent from the store are handled by ddrs (0.001 fill at read), + never an error. + +The contract is written from `forward_merit.py`'s output, so both in-scope +stores conform and **the NH repo needs no changes** for them. A future NH +dataset = adapt/write a forward script in the NH repo to emit a conforming +store, then `ddrs import` it here. + +### 2. Resolution-aware icechunk reader (`src/data/store/icechunk.rs`) + +`StreamflowStore` gains `resolution: data::dates::Frequency`, sniffed at open +from the CF units string: + +- `days since …` → `Daily` (today's behavior, unchanged) +- `hours since …` → `Hourly` (new) +- anything else → hard `DataError` carrying the store path and units string. + No silent fallback. + +The three-method contract is unchanged for callers (`StreamflowSource` enum +and dispatch in `src/data/store/mod.rs` untouched apart from plumbing): + +| method | daily store | hourly store (new) | +|---|---|---| +| `read_window_daily(start, n_days, comids)` | unchanged | read `n_days*24` rows, mean over each 24-block → `(n_days, N)` | +| `read_window(&RhoWindow, comids)` | unchanged (`daily_to_hourly_trim` repeat-24) | direct slice of `window.hourly_range()` → `(n_hourly, N)` | +| `read_test_window(&TestWindow, comids)` | unchanged | direct `n_days*24` contiguous rows | + +Rationale for the 24-block mean: Q' is a rate (m³/s), so the daily value is +the day's average flow; this keeps the **summed-Q' baseline** and any other +`read_window_daily` caller working on hourly stores unmodified. + +Time offsets are computed in hours for hourly stores. A requested window +falling outside the store's time range is a **hard error** — the hourly store +starts 1981-01-01, and an experiment configured from 1980 must fail loudly, +not clamp. If the current daily path silently tolerates out-of-range windows, +that latent bug is fixed as part of this work. + +**Guardrail:** `MeritGagesDataset::open` (`src/data/dataset.rs`) rejects +`kan_head.disaggregation` when the streamflow source is hourly-native — +disaggregating an already-hourly signal is a config contradiction. Same +enforcement style as the existing missing-`aorc_precip` error +(`dataset.rs:355`). `flow_scale` applies unchanged (per-column constant, +resolution-independent). + +**Regression guard:** the daily path must stay byte-identical; a +`leakance_off_parity`-style test enforces it. + +### 3. `ddrs import` module (`src/cli/import.rs`) + +``` +ddrs import --name [--dry-run] +``` + +1. **Open & detect** — `StreamflowSource::open` (existing icechunk-vs-zarr + sniff), then report format, detected resolution, time range, basin count. +2. **Validate the contract** — `Qr` present with expected dims/dtype; time + axis parses and is contiguous; sample read of a few COMIDs is finite and + positive. +3. **Coverage report** — intersect store `divide_id`s against the workspace + adjacency when `.ddrs/` exists: "N of M fabric COMIDs covered; X% get the + 0.001 fill." No workspace → skip with a warning, don't fail. +4. **Register** — copy the current `ddrs.yaml` `data_sources:` block, swap + `streamflow:` to the store path, save as `config/sources/.yaml` via + the existing `sources::save` machinery, re-lock. `--dry-run` stops after + step 3. + +Precondition: an existing `ddrs.yaml` supplies the non-streamflow source keys +(same precondition as `ddrs sources save`). Post-import flow: +`ddrs sources use && ddrs plan && ddrs run --workflow train`. + +### 4. Verification (success criterion: plumbing + short smoke train) + +- **Verify-first step:** before building the sniff, check the real stores' + actual CF time encodings (a one-line xarray inspection under the ddr venv — + was classifier-blocked during brainstorming). If xarray encoded the hourly + axis as something other than `hours since` (e.g. minutes/seconds since), + the sniff grammar adapts before anything else is built. +- **Tests:** CF `hours since` parsing; hourly read shapes + alignment (hour + *h* of day *d* returns the stored value) against a small fixture store; + 24-block-mean correctness; daily-path byte-parity regression; the + disagg-rejection error. +- **Import validation:** `ddrs import --dry-run` must pass on all four + unit-catchment stores in `/mnt/ssd1/data/icechunk/` (the two LSTM stores, + `daily_dhbv2_merit_unit_catchments.ic`, and + `merit_dhbv2_UH_retrospective.ic` — the latter doubles as a known-good + control since ddrs already trains on it). +- **Smoke trains:** `ddrs import` both LSTM stores, then a few-epoch small-batch + train on `daily-lstm` (unchanged path, new store) and on `hourly-lstm` + (new path): finite loss, directory-style checkpoints, and a log line + confirming `resolution: hourly` actually executed (stale-binary lesson — + reinstall or `cargo run` the working-tree binary). + +### 5. Concerns and assumptions + +**Concerns (what could go wrong, why):** + +- *Time-encoding surprise* — xarray auto-picks CF units; the hourly axis may + not literally be `hours since`. Low risk; the verify-first step catches it + before any reader code is written. +- *Hourly read cost* — 24× the rows per window from a 257 GB store (chunks + ~3080 basins × 11232 hours); random rho-windows could make epochs + I/O-bound. The smoke train measures wall-clock; chunk-aligned reads are a + possible follow-up, explicitly out of scope now. +- *Store starts 1981* — out-of-range windows must hard-error (see §2); this + may surface (and fix) a latent tolerance in the daily path. +- *Coverage gaps* — the hourly store was filtered to the UH store's + divide_ids; 0.001 fill for missing reaches is existing intended behavior, + but the import report makes the magnitude visible instead of silent. + +**Assumptions (and why):** + +- The two existing stores conform to the contract — it was written from their + producer's code. +- Source groups are the right registration target — they are already how + datasets are switched in ddrs. +- The NH-side forward script needs no changes for the in-scope datasets; + generalizing it for future forcing families is NH-repo work, out of scope. + +**Benefit:** every NH-trained model becomes a routable ddrs dataset via one +command, and hourly-native routing is unlocked — real MTS-LSTM hourly forcing +instead of the disagg approximation — slotting a third forcing option into +the leakance × forcing experiment line. + +## Out of scope + +- Generalizing `forward_merit.py` for new forcing families (NH-repo work). +- Chunk-aligned / performance-tuned hourly reads (follow-up if smoke train + shows I/O-bound epochs). +- Full science-quality training runs and baseline comparisons (experiments, + not plumbing). +- Any change to the routing core, sparse solver, or KAN head (invariants 1–7 + in CLAUDE.md untouched). From 55ca424e0249b51234972efc4b64d5937ac5a47d Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Wed, 1 Jul 2026 21:15:09 -0400 Subject: [PATCH 02/41] docs: implementation plan for NH Q' import + hourly-native reading Co-Authored-By: Claude Fable 5 --- .../plans/2026-07-01-nh-qprime-import.md | 1656 +++++++++++++++++ 1 file changed, 1656 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-01-nh-qprime-import.md diff --git a/docs/superpowers/plans/2026-07-01-nh-qprime-import.md b/docs/superpowers/plans/2026-07-01-nh-qprime-import.md new file mode 100644 index 0000000..638696a --- /dev/null +++ b/docs/superpowers/plans/2026-07-01-nh-qprime-import.md @@ -0,0 +1,1656 @@ +# NH Q' Store Contract + Hourly-Native Reading + `ddrs import` Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Route neural-hydrology LSTM outputs in ddrs: sniff daily vs hourly icechunk Q' stores from their CF time axis, read hourly stores natively (no repeat-24/disagg), and add `ddrs import` to validate + register any conforming store as a data-source group. + +**Architecture:** `StreamflowStore` (icechunk reader) gains a `resolution: Frequency` field parsed from the time `units` attribute; its three read methods branch on it, sharing one generic slab reader. A new `src/cli/import.rs` opens a store via the existing `StreamflowSource::open` sniff, validates the contract, reports COMID coverage against the resolved adjacency, and writes a `config/sources/.yaml` group with the `streamflow:` path swapped in. The NH-side producer (`~/projects/neuralhydrology/examples/merit_hydro/forward_merit.py`) is unchanged. + +**Tech Stack:** Rust (zarrs + icechunk crates, ndarray, clap, serde_yaml); one Python fixture script run under DDR's uv venv. + +**Spec:** `docs/superpowers/specs/2026-07-01-nh-qprime-import-design.md` +**Branch:** work on the current branch `unit_catchments`. + +**Confirmed store facts (probed 2026-07-01, drive the sniff grammar and tests):** + +| Store (`/mnt/ssd1/data/icechunk/`) | time units | range | n_time | divides | +|---|---|---|---|---| +| `daily_lstm_merit_unit_catchments.ic` | `days since 1981-01-01 00:00:00` | 1981-01-01 → 2020-12-30 | 14,609 | 288,421 | +| `hourly_lstm_merit_unit_catchments.ic` | `hours since 1981-01-01 00:00:00` | 1981-01-01T00 → 2020-12-31T23 | 350,640 | 197,088 | +| `daily_dhbv2_merit_unit_catchments.ic` | `days since 1980-01-01 00:00:00` | 1980-01-01 → 2020-12-30 | 14,975 | 288,421 | +| `merit_dhbv2_UH_retrospective.ic` | `days since 1980-01-01` | 1980-01-01 → 2020-12-31 | 14,976 | 197,088 | + +All four: `Qr(divide_id, time)` float32 with `units: m^3/s`, `divide_id` int64, time int64 on disk. + +**Known accepted cost (do NOT "fix" in this plan):** `MeritGagesDataset::collate` reads `q_prime_daily` unconditionally (`src/data/dataset.rs:496-500`). On an hourly store that re-reads the same chunks a second time (aggregated). The smoke train measures it; changing collate's read pattern is out of scope. + +--- + +### Task 1: Test fixture stores (Python, run under DDR's venv) + +Tiny icechunk fixtures with deterministic values so hourly read alignment is assertable to the exact element. Checked into git (a few KB each). + +**Files:** +- Create: `scripts/make_streamflow_fixtures.py` +- Create (generated): `tests/fixtures/qr_daily.ic/`, `tests/fixtures/qr_hourly.ic/`, `tests/fixtures/qr_minutes.ic/` + +- [ ] **Step 1: Write the fixture generator** + +```python +"""Write tiny icechunk Qr fixture stores for ddrs integration tests. + +Run under DDR's uv venv (it has icechunk + xarray): + + cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/make_streamflow_fixtures.py + +Layout matches the DDR Q' store contract (docs/nh-qprime-store-contract.md): +Qr(divide_id, time) f32 m^3/s, divide_id int64, CF int64 time axis. + +Deterministic values so tests can assert exact elements: + qr_daily.ic : 4 divides x 10 days, Qr[j, t] = (j+1)*100 + t + qr_hourly.ic : 4 divides x 240 hours, Qr[j, h] = (j+1)*1000 + h + qr_minutes.ic : sniff-rejection fixture (units "minutes since ...") +""" +from pathlib import Path +import shutil + +import icechunk +import numpy as np +import xarray as xr + +FIXTURES = Path(__file__).resolve().parent.parent / "tests" / "fixtures" +DIVIDES = np.array([101, 102, 103, 104], dtype=np.int64) + + +def write_store(path: Path, times: np.ndarray, qr: np.ndarray, time_units: str) -> None: + shutil.rmtree(path, ignore_errors=True) + storage = icechunk.local_filesystem_storage(str(path)) + repo = icechunk.Repository.create(storage) + session = repo.writable_session("main") + ds = xr.Dataset( + data_vars={ + "Qr": (["divide_id", "time"], qr.astype(np.float32), {"units": "m^3/s"}), + }, + coords={ + "divide_id": ("divide_id", DIVIDES), + "time": ("time", times), + }, + attrs={"units": "m^3/s", "source": "ddrs test fixture"}, + ) + ds.to_zarr( + session.store, + mode="w", + encoding={"time": {"units": time_units, "dtype": "int64"}}, + ) + session.commit("fixture") + print(f"wrote {path}") + + +def main() -> None: + n_days = 10 + daily_times = np.datetime64("1981-01-01") + np.arange(n_days).astype("timedelta64[D]") + daily = (np.arange(4)[:, None] + 1) * 100 + np.arange(n_days)[None, :] + write_store(FIXTURES / "qr_daily.ic", daily_times, daily, "days since 1981-01-01") + + n_hours = n_days * 24 + hourly_times = np.datetime64("1981-01-01T00") + np.arange(n_hours).astype("timedelta64[h]") + hourly = (np.arange(4)[:, None] + 1) * 1000 + np.arange(n_hours)[None, :] + write_store( + FIXTURES / "qr_hourly.ic", hourly_times, hourly, + "hours since 1981-01-01 00:00:00", + ) + + # Same data, unsupported units string — exercises the sniff hard-error. + write_store( + FIXTURES / "qr_minutes.ic", hourly_times[:48], hourly[:, :48], + "minutes since 1981-01-01", + ) + + +if __name__ == "__main__": + main() +``` + +- [ ] **Step 2: Run it and sanity-check the output** + +Run: `cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/make_streamflow_fixtures.py` +Expected: three `wrote .../tests/fixtures/qr_*.ic` lines. + +Run: `du -sh ~/projects/ddrs/tests/fixtures/qr_*.ic` +Expected: each well under 1 MB. + +Verify the encoding took (this is what the Rust sniff will read): + +```bash +cd ~/projects/ddr && uv run python -c " +import icechunk as ic, xarray as xr +for n in ['qr_daily.ic', 'qr_hourly.ic', 'qr_minutes.ic']: + p = '/home/tbindas/projects/ddrs/tests/fixtures/' + n + repo = ic.Repository.open(ic.local_filesystem_storage(p)) + ds = xr.open_zarr(repo.readonly_session('main').store, consolidated=False) + print(n, ds.time.encoding.get('units'), ds.Qr.shape) +" +``` +Expected: `days since 1981-01-01` / `hours since 1981-01-01 00:00:00` / `minutes since 1981-01-01`, shapes `(4, 10)`, `(4, 240)`, `(4, 48)`. + +- [ ] **Step 3: Commit** + +```bash +cd ~/projects/ddrs +git add scripts/make_streamflow_fixtures.py tests/fixtures/qr_daily.ic tests/fixtures/qr_hourly.ic tests/fixtures/qr_minutes.ic +git commit -m "test: icechunk Qr fixture stores (daily/hourly/bad-units)" +``` + +--- + +### Task 2: `parse_cf_units` — resolution sniff (TDD) + +**Files:** +- Modify: `src/data/store/icechunk.rs:234-262` (replace `parse_cf_epoch` internals; keep a daily-only wrapper for the USGS obs store) + +- [ ] **Step 1: Write the failing unit tests** + +Append inside the existing `mod tests` in `src/data/store/icechunk.rs`: + +```rust + fn attrs_with_units(u: &str) -> serde_json::Map { + let mut m = serde_json::Map::new(); + m.insert("units".into(), serde_json::Value::String(u.into())); + m + } + + #[test] + fn parse_cf_units_daily() { + let (epoch, res) = + parse_cf_units(&attrs_with_units("days since 1980-01-01"), Path::new("/t")).unwrap(); + assert_eq!(epoch, chrono::NaiveDate::from_ymd_opt(1980, 1, 1).unwrap()); + assert_eq!(res, crate::data::dates::Frequency::Daily); + } + + #[test] + fn parse_cf_units_daily_with_time_of_day() { + // daily_lstm store encodes "days since 1981-01-01 00:00:00". + let (epoch, res) = + parse_cf_units(&attrs_with_units("days since 1981-01-01 00:00:00"), Path::new("/t")) + .unwrap(); + assert_eq!(epoch, chrono::NaiveDate::from_ymd_opt(1981, 1, 1).unwrap()); + assert_eq!(res, crate::data::dates::Frequency::Daily); + } + + #[test] + fn parse_cf_units_hourly() { + // hourly_lstm store encodes "hours since 1981-01-01 00:00:00". + let (epoch, res) = + parse_cf_units(&attrs_with_units("hours since 1981-01-01 00:00:00"), Path::new("/t")) + .unwrap(); + assert_eq!(epoch, chrono::NaiveDate::from_ymd_opt(1981, 1, 1).unwrap()); + assert_eq!(res, crate::data::dates::Frequency::Hourly); + } + + #[test] + fn parse_cf_units_rejects_other_resolutions() { + let err = parse_cf_units(&attrs_with_units("minutes since 1981-01-01"), Path::new("/t")) + .unwrap_err(); + let msg = err.to_string(); + assert!(msg.contains("minutes since"), "error must name the units: {msg}"); + assert!(msg.contains("days since"), "error must name what IS supported: {msg}"); + } + + #[test] + fn parse_cf_epoch_rejects_hourly_axis() { + // The daily-only wrapper (used by the USGS observations store) must + // refuse an hourly axis rather than silently mis-scaling. + let err = parse_cf_epoch(&attrs_with_units("hours since 1980-01-01"), Path::new("/t")) + .unwrap_err(); + assert!(err.to_string().contains("daily"), "got: {err}"); + } +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cargo test --lib parse_cf` +Expected: FAIL — `parse_cf_units` not found (compile error). + +- [ ] **Step 3: Implement `parse_cf_units`, keep `parse_cf_epoch` as the daily-only wrapper** + +Replace the whole `parse_cf_epoch` function (`src/data/store/icechunk.rs:234-262`) with: + +```rust +/// Parse the CF `units` attribute of a time coordinate and return the epoch +/// plus the native axis resolution. Supported forms (see +/// docs/nh-qprime-store-contract.md): +/// "days since YYYY-MM-DD[ HH:MM:SS]" → Daily +/// "hours since YYYY-MM-DD[ HH:MM:SS]" → Hourly +/// Anything else is a hard error naming the store and the units string — a +/// mis-scaled time axis must never be silently accepted. +pub(crate) fn parse_cf_units( + attrs: &serde_json::Map, + path: &Path, +) -> Result<(NaiveDate, crate::data::dates::Frequency)> { + use crate::data::dates::Frequency; + + let units = attrs + .get("units") + .and_then(|v| v.as_str()) + .ok_or_else(|| DataError::Malformed { + path: path.to_path_buf(), + message: "time array missing 'units' attribute".into(), + })?; + let (date_str, resolution) = if let Some(rest) = units.strip_prefix("days since ") { + (rest, Frequency::Daily) + } else if let Some(rest) = units.strip_prefix("hours since ") { + (rest, Frequency::Hourly) + } else { + return Err(DataError::Malformed { + path: path.to_path_buf(), + message: format!( + "unsupported time units {units:?}: expected \"days since …\" \ + or \"hours since …\"" + ), + }); + }; + // The date portion may be followed by a time-of-day component, e.g. + // "1981-01-01 00:00:00" — take only the first token. + let date_part = date_str.split_whitespace().next().unwrap_or(""); + let epoch = + NaiveDate::parse_from_str(date_part, "%Y-%m-%d").map_err(|e| DataError::Malformed { + path: path.to_path_buf(), + message: format!("cannot parse epoch from units {units:?}: {e}"), + })?; + Ok((epoch, resolution)) +} + +/// Daily-only wrapper for stores whose axis MUST be daily (USGS observations). +pub(crate) fn parse_cf_epoch( + attrs: &serde_json::Map, + path: &Path, +) -> Result { + match parse_cf_units(attrs, path)? { + (epoch, crate::data::dates::Frequency::Daily) => Ok(epoch), + (_, crate::data::dates::Frequency::Hourly) => Err(DataError::Malformed { + path: path.to_path_buf(), + message: "expected a daily time axis (\"days since …\"), got hourly".into(), + }), + } +} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cargo test --lib parse_cf` +Expected: 5 tests PASS. Also run `cargo test --lib` — no other lib test may break (`StreamflowStore::open` still calls `parse_cf_epoch`, unchanged behavior until Task 3). + +- [ ] **Step 5: Commit** + +```bash +git add src/data/store/icechunk.rs +git commit -m "feat(data): parse_cf_units sniffs daily vs hourly CF time axes" +``` + +--- + +### Task 3: Resolution-aware `StreamflowStore` (TDD, fixture-backed) + +The core change. `StreamflowStore` gains `resolution`; the existing `read_window_daily` body becomes a resolution-agnostic `read_slab` over native time steps; the three public methods branch on resolution. Daily-path behavior must stay byte-identical. + +**Files:** +- Modify: `src/data/store/icechunk.rs:177-231` (struct + open), `:286-408` (read methods) +- Create: `tests/hourly_streamflow.rs` + +- [ ] **Step 1: Write the failing integration tests** + +Create `tests/hourly_streamflow.rs`: + +```rust +//! Fixture-backed tests for resolution-aware Q' reading. +//! +//! Fixtures are generated by `scripts/make_streamflow_fixtures.py` (run under +//! DDR's uv venv) and checked into tests/fixtures/. Deterministic values: +//! qr_daily.ic : 4 divides [101..104] x 10 days, Qr[j, t] = (j+1)*100 + t +//! qr_hourly.ic : 4 divides [101..104] x 240 hours, Qr[j, h] = (j+1)*1000 + h +//! Both axes start 1981-01-01. + +use chrono::NaiveDate; + +use ddrs::data::dates::{Frequency, RhoWindow, TimeAxis}; +use ddrs::data::ids::Comid; +use ddrs::data::store::{StreamflowSource, StreamflowStore}; +use ddrs::data::TestWindow; + +fn fixture(name: &str) -> std::path::PathBuf { + std::path::Path::new(env!("CARGO_MANIFEST_DIR")) + .join("tests/fixtures") + .join(name) +} + +fn d(y: i32, m: u32, day: u32) -> NaiveDate { + NaiveDate::from_ymd_opt(y, m, day).unwrap() +} + +const COMIDS: [Comid; 4] = [Comid(101), Comid(102), Comid(103), Comid(104)]; + +#[test] +fn hourly_store_opens_with_hourly_resolution() { + let s = StreamflowStore::open(fixture("qr_hourly.ic")).expect("open"); + assert_eq!(s.resolution, Frequency::Hourly); + assert_eq!(s.time_start, d(1981, 1, 1)); + assert_eq!(s.n_time, 240); + assert_eq!(s.index.len(), 4); +} + +#[test] +fn daily_store_opens_with_daily_resolution() { + let s = StreamflowStore::open(fixture("qr_daily.ic")).expect("open"); + assert_eq!(s.resolution, Frequency::Daily); + assert_eq!(s.time_start, d(1981, 1, 1)); + assert_eq!(s.n_time, 10); +} + +#[test] +fn minutes_axis_is_rejected_at_open() { + let err = StreamflowSource::open(fixture("qr_minutes.ic")).unwrap_err(); + assert!( + err.to_string().contains("unsupported time units"), + "got: {err}" + ); +} + +#[test] +fn hourly_read_window_slices_natively() { + let s = StreamflowStore::open(fixture("qr_hourly.ic")).expect("open"); + // Window: days [2, 6) of the axis → hours [48, 120); n_hourly = 3*24 = 72. + let w = RhoWindow { + start_day_idx: 2, + rho_days: 4, + window_start: d(1981, 1, 3), + }; + let q = s.read_window(&w, &COMIDS).expect("read_window"); + assert_eq!(q.shape(), &[72, 4]); + for h in 0..72 { + for j in 0..4 { + let expect = (j as f32 + 1.0) * 1000.0 + (48 + h) as f32; + assert_eq!(q[(h, j)], expect, "mismatch at hour {h}, divide {j}"); + } + } +} + +#[test] +fn hourly_read_window_daily_is_24h_mean() { + let s = StreamflowStore::open(fixture("qr_hourly.ic")).expect("open"); + let q = s + .read_window_daily(d(1981, 1, 3), 4, &COMIDS) + .expect("read_window_daily"); + assert_eq!(q.shape(), &[4, 4]); + // Day d of the window covers hours 48+24d .. 48+24d+24; the mean of a + // 24-term arithmetic ramp k..k+23 is k + 11.5. + for day in 0..4 { + for j in 0..4 { + let expect = (j as f32 + 1.0) * 1000.0 + (48 + 24 * day) as f32 + 11.5; + assert_eq!(q[(day, j)], expect, "mismatch at day {day}, divide {j}"); + } + } +} + +#[test] +fn hourly_read_test_window_is_contiguous() { + let s = StreamflowStore::open(fixture("qr_hourly.ic")).expect("open"); + let axis = TimeAxis::new(d(1981, 1, 1), d(1981, 1, 10)); + let w = TestWindow::new(&axis, 2, 4); // hours [48, 144), no trailing trim + let q = s.read_test_window(&w, &COMIDS).expect("read_test_window"); + assert_eq!(q.shape(), &[96, 4]); + assert_eq!(q[(0, 0)], 1000.0 + 48.0); + assert_eq!(q[(95, 3)], 4000.0 + 143.0); +} + +#[test] +fn hourly_missing_comid_gets_fill() { + let s = StreamflowStore::open(fixture("qr_hourly.ic")).expect("open"); + let w = RhoWindow { + start_day_idx: 0, + rho_days: 2, + window_start: d(1981, 1, 1), + }; + let q = s + .read_window(&w, &[Comid(101), Comid(999)]) + .expect("read_window"); + assert_eq!(q.shape(), &[24, 2]); + assert_eq!(q[(5, 0)], 1000.0 + 5.0); + assert_eq!(q[(5, 1)], 0.001, "missing COMID must fill with 0.001"); +} + +#[test] +fn hourly_out_of_range_windows_hard_error() { + let s = StreamflowStore::open(fixture("qr_hourly.ic")).expect("open"); + // Before store start. + let before = RhoWindow { + start_day_idx: 0, + rho_days: 2, + window_start: d(1980, 12, 1), + }; + let err = s.read_window(&before, &COMIDS).unwrap_err(); + assert!(err.to_string().contains("before store start"), "got: {err}"); + // Past store end (store holds 10 days). + let past = RhoWindow { + start_day_idx: 8, + rho_days: 5, + window_start: d(1981, 1, 9), + }; + assert!(s.read_window(&past, &COMIDS).is_err()); +} + +#[test] +fn daily_fixture_read_window_keeps_repeat24_semantics() { + // Pins the daily path: values repeat 24x per day with the trailing-day trim. + let s = StreamflowStore::open(fixture("qr_daily.ic")).expect("open"); + let w = RhoWindow { + start_day_idx: 2, + rho_days: 4, + window_start: d(1981, 1, 3), + }; + let q = s.read_window(&w, &COMIDS).expect("read_window"); + assert_eq!(q.shape(), &[72, 4]); + for h in 0..72 { + for j in 0..4 { + let expect = (j as f32 + 1.0) * 100.0 + (2 + h / 24) as f32; + assert_eq!(q[(h, j)], expect, "mismatch at hour {h}, divide {j}"); + } + } +} +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cargo test --test hourly_streamflow` +Expected: compile error — `StreamflowStore` has no field `resolution`. + +- [ ] **Step 3: Add `resolution` to the struct and open()** + +In `src/data/store/icechunk.rs`, change the struct (`:177-187`): + +```rust +pub struct StreamflowStore { + pub path: PathBuf, + pub index: IdIndex, + /// First calendar day covered by the store (for hourly stores, the day + /// containing the first hour — open() enforces hour-0 alignment). + pub time_start: NaiveDate, + /// Length of the NATIVE time axis: days for daily stores, hours for + /// hourly stores. + pub n_time: usize, + /// Native axis resolution, sniffed from the CF `units` attribute. + pub resolution: crate::data::dates::Frequency, + // SP-3 may consolidate to a shared runtime; keep the Arc alive so the + // icechunk Store is not dropped while `qr` is in use. + #[allow(dead_code)] + storage: Arc, + qr: ZarrArray, +} +``` + +In `open()` (`:190-231`), replace the epoch/time_start block: + +```rust + // 1. Read `time` coord: shape (n_time,), dtype int64. CF units are + // "days since …" (daily) or "hours since …" (hourly) — the sniff + // that decides this store's native resolution. + let time_arr = ZarrArray::open(readable.clone(), "/time") + .map_err(|e| ic_err(&path, e))?; + let (time_epoch, resolution) = parse_cf_units(time_arr.attributes(), &path)?; + let time_subset = time_arr.subset_all(); + let time_i64: Vec = time_arr + .retrieve_array_subset(&time_subset) + .map_err(|e| ic_err(&path, e))?; + let n_time = time_i64.len(); + if n_time == 0 { + return Err(DataError::Malformed { + path: path.clone(), + message: "time axis is empty".into(), + }); + } + let time_start = match resolution { + crate::data::dates::Frequency::Daily => { + time_epoch + chrono::Duration::days(time_i64[0]) + } + crate::data::dates::Frequency::Hourly => { + // Contract: hourly axes start at hour 0 of a day and are + // contiguous (docs/nh-qprime-store-contract.md). The full + // scan is cheap (~2.8 MB of i64 for 40 years of hours). + if time_i64[0] % 24 != 0 { + return Err(DataError::Malformed { + path: path.clone(), + message: format!( + "hourly store must start at hour 0 of a day; \ + first time value is {}", + time_i64[0] + ), + }); + } + if let Some(i) = + (1..time_i64.len()).find(|&i| time_i64[i] - time_i64[i - 1] != 1) + { + return Err(DataError::Malformed { + path: path.clone(), + message: format!("hourly time axis has a gap at index {i}"), + }); + } + time_epoch + chrono::Duration::days(time_i64[0] / 24) + } + }; +``` + +and the constructor return: + +```rust + Ok(Self { path, index, time_start, n_time, resolution, storage, qr }) +``` + +- [ ] **Step 4: Refactor reads — `native_start_index` + `read_slab` + branching public methods** + +Replace the whole second `impl StreamflowStore` block (`:286-409`) with: + +```rust +impl StreamflowStore { + /// Store-local index of `window_start` on the NATIVE time axis + /// (day index for daily stores, hour index for hourly stores). + fn native_start_index(&self, window_start: NaiveDate) -> Result { + let days = (window_start - self.time_start).num_days(); + if days < 0 { + return Err(DataError::Malformed { + path: self.path.clone(), + message: format!( + "window starts {} before store start {}", + window_start, self.time_start + ), + }); + } + Ok(match self.resolution { + crate::data::dates::Frequency::Daily => days as usize, + crate::data::dates::Frequency::Hourly => days as usize * 24, + }) + } + + /// Read `(n_steps, N)` from native time-axis positions + /// `[start_step, start_step + n_steps)` for `comids`. Missing COMIDs are + /// filled with `0.001` (discharge minimum, mirrors DDR's + /// `torch.full(..., fill_value=0.001)` in `readers.py:464-468`). + fn read_slab( + &self, + start_step: usize, + n_steps: usize, + comids: &[Comid], + ) -> Result> { + let end_step = start_step + n_steps; + if end_step > self.n_time { + return Err(DataError::Malformed { + path: self.path.clone(), + message: format!( + "window extends to store step {end_step} but n_time={} \ + ({:?} axis)", + self.n_time, self.resolution + ), + }); + } + + // Resolve COMIDs → divide-axis positions. + // `positions_of` returns positions in the order of non-missing inputs, + // plus a list of indices (into `comids`) that were missing. + let (positions, missing_indices) = self.index.positions_of(comids); + let missing_set: std::collections::HashSet = + missing_indices.iter().copied().collect(); + let n_out = comids.len(); + + // Pre-fill with the discharge minimum; missing COMIDs keep this value. + let mut out = Array2::::from_elem((n_steps, n_out), 0.001); + + if positions.is_empty() { + return Ok(out); + } + + // Contiguous divide-axis read covering [min_pos, max_pos]. + let min_pos = *positions.iter().min().unwrap(); + let max_pos = *positions.iter().max().unwrap(); + let div_range_end = max_pos + 1; + let div_count = div_range_end - min_pos; + + // Qr is stored as (divide_id, time). Subset: axis 0 = divide, axis 1 = time. + let subset = zarrs::array::ArraySubset::new_with_ranges(&[ + (min_pos as u64)..(div_range_end as u64), + (start_step as u64)..(end_step as u64), + ]); + let raw_f32: Vec = self + .qr + .retrieve_array_subset(&subset) + .map_err(|e| ic_err(&self.path, e))?; + // raw_f32 is row-major: shape (div_count, n_steps). + debug_assert_eq!(raw_f32.len(), div_count * n_steps); + + // Scatter into the output. Walk `comids` in order; for each + // non-missing entry consume the next element of `positions`. + let mut next_present = 0usize; + for (out_col, _) in comids.iter().enumerate() { + if missing_set.contains(&out_col) { + continue; + } + let div_pos = positions[next_present]; + next_present += 1; + let local_div = div_pos - min_pos; + for t in 0..n_steps { + let raw_idx = local_div * n_steps + t; + out[(t, out_col)] = raw_f32[raw_idx]; + } + } + + debug_assert_eq!( + next_present, + positions.len(), + "scatter walked past `positions` — IdIndex::positions_of invariant broken" + ); + + Ok(out) + } + + /// Read `Qr` daily for `[window_start, window_start + n_days)` and + /// `comids`. Returns `(n_days, N)` f32 matrix. On hourly-native stores + /// each day is the mean of its 24 hours (Q' is a rate in m³/s, so the + /// daily value is the day's average flow — keeps the summed-Q' baseline + /// meaningful on hourly stores). + pub fn read_window_daily( + &self, + window_start: NaiveDate, + n_days: usize, + comids: &[Comid], + ) -> Result> { + let start = self.native_start_index(window_start)?; + match self.resolution { + crate::data::dates::Frequency::Daily => self.read_slab(start, n_days, comids), + crate::data::dates::Frequency::Hourly => { + let hourly = self.read_slab(start, n_days * 24, comids)?; + Ok(hourly_to_daily_mean(&hourly)) + } + } + } + + /// Read `Qr` for `window` and `comids`. Returns `(n_hourly, N)` f32. + /// Daily stores upsample via repeat-24 + trailing-day trim (unchanged); + /// hourly stores slice the native axis directly — no upsampling. + pub fn read_window(&self, window: &RhoWindow, comids: &[Comid]) -> Result> { + match self.resolution { + crate::data::dates::Frequency::Daily => { + let daily = + self.read_window_daily(window.window_start, window.rho_days, comids)?; + Ok(daily_to_hourly_trim(&daily, window.n_hourly())) + } + crate::data::dates::Frequency::Hourly => { + let start = self.native_start_index(window.window_start)?; + self.read_slab(start, window.n_hourly(), comids) + } + } + } + + /// Same as `read_window` but for `TestWindow` — `n_days * 24` hours + /// (no trailing-day trim) so chunks tile cleanly. + pub fn read_test_window( + &self, + window: &crate::data::TestWindow, + comids: &[Comid], + ) -> Result> { + match self.resolution { + crate::data::dates::Frequency::Daily => { + let daily = + self.read_window_daily(window.window_start, window.n_days, comids)?; + Ok(daily_to_hourly_trim(&daily, window.n_hourly())) + } + crate::data::dates::Frequency::Hourly => { + let start = self.native_start_index(window.window_start)?; + self.read_slab(start, window.n_hourly(), comids) + } + } + } + + /// `units` attribute of the `/Qr` variable, if present. Used by + /// `ddrs import` to check the m³/s contract. + pub fn qr_units(&self) -> Option { + self.qr + .attributes() + .get("units") + .and_then(|v| v.as_str()) + .map(str::to_string) + } +} +``` + +Then add next to `daily_to_hourly_trim` (after `:284`): + +```rust +/// Collapse a `(n_days * 24, N)` hourly slab to `(n_days, N)` by averaging +/// each 24-hour block. Q' is a rate (m³/s): the daily value is the day's +/// mean flow, so total daily volume is preserved. +pub(crate) fn hourly_to_daily_mean(hourly: &Array2) -> Array2 { + let (n_hours, n_div) = hourly.dim(); + debug_assert_eq!(n_hours % 24, 0, "hourly slab length {n_hours} not a multiple of 24"); + let n_days = n_hours / 24; + let mut daily = Array2::::zeros((n_days, n_div)); + for d in 0..n_days { + for j in 0..n_div { + let mut acc = 0.0f32; + for h in 0..24 { + acc += hourly[(d * 24 + h, j)]; + } + daily[(d, j)] = acc / 24.0; + } + } + daily +} +``` + +Note: the old `read_window_daily` body moves into `read_slab` with only renames (`n_days`→`n_steps`, `store_start_day`→`start_step`, `end_day`→`end_step`, `daily`→`out`, loop var `d`→`t`) — the window-start validation moves to `native_start_index`. Do not otherwise change the read/scatter logic: the daily path must stay behaviorally identical. + +- [ ] **Step 5: Run the new tests and the daily regression tests** + +Run: `cargo test --test hourly_streamflow` +Expected: 9 tests PASS. + +Run: `cargo test --lib` and `cargo test --test data_zarr_store 2>/dev/null; cargo test streamflow` +Expected: all PASS — in particular the existing real-store tests `streamflow_read_window_returns_expected_shape` and `streamflow_store_open_sees_expected_axes` (daily path unchanged against `merit_dhbv2_UH_retrospective.ic`). + +- [ ] **Step 6: Check `n_time` consumers survive the semantics note** + +`n_time` now means "native steps" (hours for hourly stores). + +Run: `grep -rn "\.n_time" src/ examples/ | grep -v "store/icechunk.rs"` +Expected: only uses on observation stores or daily streamflow contexts. If any call site assumes `StreamflowStore::n_time` is days, fix it to use the new doc'd semantics (divide by 24 on hourly) and note it in the commit message. (As of plan-writing, the baseline and dataset go through `read_window*` only.) + +- [ ] **Step 7: Commit** + +```bash +git add src/data/store/icechunk.rs tests/hourly_streamflow.rs +git commit -m "feat(data): hourly-native Q' reading in StreamflowStore + +Resolution sniffed from CF time units at open. Daily path unchanged +(read_slab is the old read_window_daily body, renames only). Hourly +stores slice the native axis in read_window/read_test_window and +24h-mean in read_window_daily." +``` + +--- + +### Task 4: `StreamflowSource::resolution()` + disagg guard in the dataset (TDD) + +**Files:** +- Modify: `src/data/store/mod.rs:99-141` +- Modify: `src/data/dataset.rs:332` area (guard + log line) + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/hourly_streamflow.rs`: + +```rust +#[test] +fn streamflow_source_reports_resolution() { + let daily = StreamflowSource::open(fixture("qr_daily.ic")).expect("open daily"); + assert_eq!(daily.resolution(), Frequency::Daily); + let hourly = StreamflowSource::open(fixture("qr_hourly.ic")).expect("open hourly"); + assert_eq!(hourly.resolution(), Frequency::Hourly); +} +``` + +And a unit test for the guard — append inside `mod tests` at the bottom of `src/data/dataset.rs` (create the module if the file has none; check first with `grep -n "mod tests" src/data/dataset.rs`): + +```rust + #[test] + fn disagg_rejected_on_hourly_native_source() { + use crate::data::dates::Frequency; + let p = std::path::Path::new("/mnt/fake/qr_hourly.ic"); + // hourly + disagg block → config contradiction + let err = validate_disagg_vs_resolution(Frequency::Hourly, true, p).unwrap_err(); + assert!(err.to_string().contains("hourly-native"), "got: {err}"); + // every other combination is fine + assert!(validate_disagg_vs_resolution(Frequency::Hourly, false, p).is_ok()); + assert!(validate_disagg_vs_resolution(Frequency::Daily, true, p).is_ok()); + assert!(validate_disagg_vs_resolution(Frequency::Daily, false, p).is_ok()); + } +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cargo test --test hourly_streamflow streamflow_source_reports_resolution; cargo test --lib disagg_rejected` +Expected: compile errors — `resolution()` and `validate_disagg_vs_resolution` not defined. + +- [ ] **Step 3: Implement** + +In `src/data/store/mod.rs`, inside `impl StreamflowSource` (after `open`, `:110`): + +```rust + /// Native time-axis resolution of the underlying store. The global + /// zarr v2 layout is daily by construction. + pub fn resolution(&self) -> crate::data::dates::Frequency { + match self { + Self::Icechunk(s) => s.resolution, + Self::GlobalZarr(_) => crate::data::dates::Frequency::Daily, + } + } +``` + +In `src/data/dataset.rs`, add a free function near the other helpers at module level (e.g. directly above `impl MeritGagesDataset` — find it with `grep -n "^impl MeritGagesDataset" src/data/dataset.rs`): + +```rust +/// Reject the disaggregation head when the streamflow store is hourly-native: +/// disaggregating an already-hourly signal is a config contradiction, and +/// after the 2026-07-01 stale-binary incident nothing in the forcing path is +/// allowed to silently degrade. +fn validate_disagg_vs_resolution( + resolution: Frequency, + has_disagg: bool, + streamflow_path: &std::path::Path, +) -> Result<()> { + if resolution == Frequency::Hourly && has_disagg { + return Err(DataError::Malformed { + path: streamflow_path.to_path_buf(), + message: "kan_head.disaggregation is set but the streamflow store is \ + hourly-native; remove the disaggregation block (an hourly \ + store needs no daily→hourly head)" + .into(), + }); + } + Ok(()) +} +``` + +(`Frequency` may need adding to the existing `use crate::data::dates::…` import at the top of `dataset.rs` — check with `grep -n "use crate::data::dates" src/data/dataset.rs`.) + +Wire it in `MeritGagesDataset::open`, immediately after the streamflow open at `src/data/dataset.rs:332`: + +```rust + let streamflow = Arc::new(StreamflowSource::open(&ds.streamflow)?); + // The smoke-train self-check line: proves which read path executed. + eprintln!("streamflow resolution: {:?}", streamflow.resolution()); + validate_disagg_vs_resolution( + streamflow.resolution(), + head_cfg.disaggregation.is_some(), + &ds.streamflow, + )?; + let observations = Arc::new(ObservationsStore::open(&ds.observations)?); +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cargo test --test hourly_streamflow; cargo test --lib disagg_rejected` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/data/store/mod.rs src/data/dataset.rs tests/hourly_streamflow.rs +git commit -m "feat(data): expose Q' resolution; reject disagg head on hourly-native source" +``` + +--- + +### Task 5: Store contract doc + +**Files:** +- Create: `docs/nh-qprime-store-contract.md` + +- [ ] **Step 1: Write the doc** + +```markdown +# DDR Q' store contract + +The interface between runoff producers (neural-hydrology LSTMs, dHBV2, …) +and ddrs routing. Any store meeting this contract can be validated and +registered with `ddrs import --name ` and then routed. + +The reference producer is +`~/projects/neuralhydrology/examples/merit_hydro/forward_merit.py` +(`--mode daily|hourly`), which runs a trained NH model over the MERIT unit +catchments and writes a conforming store. Producers that RUN neural +hydrology live in the NH repo; everything downstream of the written store +lives here. + +## Contract + +- An **icechunk repository** (`main` branch, local filesystem), root group. +- One data variable **`Qr(divide_id, time)`**, dtype **float32**, attr + `units: m^3/s`. +- `Qr` values are the **local lateral inflow per MERIT unit catchment** — + no upstream accumulation (routing does that). +- `divide_id`: int64 MERIT COMIDs. +- `time`: int64, CF-encoded as either + - `days since YYYY-MM-DD[ HH:MM:SS]` — a **daily** store, or + - `hours since YYYY-MM-DD[ HH:MM:SS]` — an **hourly** store. + The axis must be contiguous (no gaps); an hourly axis must start at + hour 0 of a calendar day. Any other units string is rejected at open. +- Values strictly positive: producers floor NaN/negative predictions to + `1e-6` (as `forward_merit.py::mm_day_to_m3s` does). +- COMIDs **absent** from the store are ddrs's concern, not the producer's: + reads fill them with `0.001` m³/s, never error. + +## How ddrs reads each resolution + +| ddrs read | daily store | hourly store | +|---|---|---| +| `read_window` (training) | repeat-24 + trailing-day trim (or disagg head) | native hourly slice | +| `read_test_window` (eval) | repeat-24, `n_days*24` | native hourly slice | +| `read_window_daily` (baseline, disagg input) | direct | mean of each 24-h block | + +`kan_head.disaggregation` is **rejected** when the streamflow source is +hourly-native — disaggregating an already-hourly signal is a config +contradiction (`src/data/dataset.rs::validate_disagg_vs_resolution`). + +## Conforming stores (2026-07-01) + +| Store (`/mnt/ssd1/data/icechunk/`) | resolution | range | divides | +|---|---|---|---| +| `daily_lstm_merit_unit_catchments.ic` | daily | 1981-01-01 → 2020-12-30 | 288,421 | +| `hourly_lstm_merit_unit_catchments.ic` | hourly | 1981-01-01 → 2020-12-31T23 | 197,088 | +| `daily_dhbv2_merit_unit_catchments.ic` | daily | 1980-01-01 → 2020-12-30 | 288,421 | +| `merit_dhbv2_UH_retrospective.ic` | daily | 1980-01-01 → 2020-12-31 | 197,088 | + +Note the hourly store starts **1981-01-01** (1980 was LSTM warmup): an +experiment window reaching into 1980 hard-errors rather than clamping. + +## Onboarding a new NH dataset + +1. In `~/projects/neuralhydrology`, write/adapt a forward script that emits + a conforming store (start from `forward_merit.py`). +2. `ddrs import --dry-run` — validates the contract + prints a + COMID-coverage report. +3. `ddrs import --name ` — registers it under + `config/sources/.yaml`. +4. `ddrs sources use && ddrs plan && ddrs run --workflow train`. + +Design history: `docs/superpowers/specs/2026-07-01-nh-qprime-import-design.md`. +``` + +- [ ] **Step 2: Commit** + +```bash +git add docs/nh-qprime-store-contract.md +git commit -m "docs: DDR Q' store contract (producer/consumer interface)" +``` + +--- + +### Task 6: `ddrs import` module (TDD) + +**Files:** +- Create: `src/cli/import.rs` +- Modify: `src/cli/mod.rs:3-17` (register module) +- Modify: `src/cli/sources.rs:52,96,108-130` (make `validate_name` + `extract_block` `pub(crate)`; extract `save_block` from `run_save`) +- Modify: `src/cli/plan.rs:288` (`fn resolve_adjacency` → `pub(crate) fn resolve_adjacency`) +- Create: `tests/import_cmd.rs` + +- [ ] **Step 1: Write the failing tests** + +Create `tests/import_cmd.rs`: + +```rust +//! `ddrs import` behavior against the checked-in fixture stores. + +use std::fs; +use std::path::{Path, PathBuf}; + +use ddrs::cli::import::{run_import, ImportInput}; +use ddrs::cli::workspace::Workspace; + +fn fixture(name: &str) -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")) + .join("tests/fixtures") + .join(name) +} + +/// Minimal parseable ddrs.yaml (mirrors src/cli/sources.rs test CFG). +const CFG: &str = "\ +mode: training +geodataset: merit +seed: 1 +np_seed: 1 +data_sources: + attributes: /dev/null/attrs.nc + conus_adjacency: /dev/null/conus.zarr + gages_adjacency: /dev/null/gages.zarr + streamflow: /dev/null/sf.ic + observations: /dev/null/obs.ic + gages: /dev/null/gages.csv +"; + +fn setup() -> (tempfile::TempDir, PathBuf, Workspace) { + let tmp = tempfile::tempdir().unwrap(); + let cfg = tmp.path().join("ddrs.yaml"); + fs::write(&cfg, CFG).unwrap(); + let ws = Workspace::with_root(tmp.path().join(".ddrs")); + (tmp, cfg, ws) +} + +#[test] +fn dry_run_validates_without_writing_a_group() { + let (_tmp, cfg, ws) = setup(); + run_import( + Some(&cfg), + &ws, + ImportInput { + store_path: fixture("qr_hourly.ic"), + name: None, + dry_run: true, + force: false, + }, + ) + .expect("dry-run import of hourly fixture"); + assert!( + !cfg.parent().unwrap().join("config/sources").exists(), + "dry-run must not create a group" + ); +} + +#[test] +fn import_registers_group_with_swapped_streamflow() { + let (_tmp, cfg, ws) = setup(); + run_import( + Some(&cfg), + &ws, + ImportInput { + store_path: fixture("qr_daily.ic"), + name: Some("test-daily".into()), + dry_run: false, + force: false, + }, + ) + .expect("import daily fixture"); + + let group = cfg.parent().unwrap().join("config/sources/test-daily.yaml"); + let text = fs::read_to_string(&group).expect("group file written"); + assert!(text.contains("qr_daily.ic"), "streamflow swapped: {text}"); + assert!( + text.contains("observations: /dev/null/obs.ic"), + "other keys carried over from ddrs.yaml: {text}" + ); + // Registering again without --force refuses; with force succeeds. + let again = ImportInput { + store_path: fixture("qr_daily.ic"), + name: Some("test-daily".into()), + dry_run: false, + force: false, + }; + assert!(run_import(Some(&cfg), &ws, again).is_err()); +} + +#[test] +fn import_rejects_nonconforming_store() { + let (_tmp, cfg, ws) = setup(); + let err = run_import( + Some(&cfg), + &ws, + ImportInput { + store_path: fixture("qr_minutes.ic"), + name: None, + dry_run: true, + force: false, + }, + ) + .unwrap_err(); + assert!( + err.to_string().contains("unsupported time units"), + "got: {err}" + ); +} + +#[test] +fn register_without_name_or_dry_run_is_an_error() { + let (_tmp, cfg, ws) = setup(); + let err = run_import( + Some(&cfg), + &ws, + ImportInput { + store_path: fixture("qr_daily.ic"), + name: None, + dry_run: false, + force: false, + }, + ) + .unwrap_err(); + assert!(err.to_string().contains("--name"), "got: {err}"); +} +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cargo test --test import_cmd` +Expected: compile error — `ddrs::cli::import` does not exist. + +- [ ] **Step 3: Open up the shared helpers** + +In `src/cli/sources.rs`: +- `:52` — `fn validate_name` → `pub(crate) fn validate_name` +- `:96` — `fn extract_block` → `pub(crate) fn extract_block` +- Split `run_save` (`:108-130`) so the persistence half is reusable: + +```rust +/// Save the current config's `data_sources:` block as group `name`. +pub fn run_save(cfg_path: &Path, name: &str, force: bool) -> Result { + let cfg_text = fs::read_to_string(cfg_path)?; + let block = extract_block(&cfg_text, cfg_path)?; + save_block(cfg_path, name, &block, force) +} + +/// Persist `block` (a full `data_sources:` block) as group `name`, after +/// validating it deserializes to `DataSources`. Shared by `ddrs sources save` +/// (verbatim block) and `ddrs import` (block with `streamflow:` swapped). +pub(crate) fn save_block( + cfg_path: &Path, + name: &str, + block: &str, + force: bool, +) -> Result { + validate_name(name)?; + serde_yaml::from_str::(block).map_err(|e| CliError::ConfigInvalid { + path: cfg_path.to_path_buf(), + source: Box::new(e), + })?; + + let dest = group_path(cfg_path, name); + if dest.exists() && !force { + return Err(CliError::Runtime(format!( + "group {name:?} already exists at {} — pass --force to overwrite", + dest.display() + ))); + } + if let Some(parent) = dest.parent() { + fs::create_dir_all(parent)?; + } + fs::write(&dest, block)?; + Ok(dest) +} +``` + +In `src/cli/plan.rs:288`: `fn resolve_adjacency(` → `pub(crate) fn resolve_adjacency(`. + +In `src/cli/mod.rs`, add `pub mod import;` to the module list (alphabetical, after `pub mod gc;`). + +- [ ] **Step 4: Write `src/cli/import.rs`** + +```rust +//! `ddrs import` — validate a Q' store against the DDR store contract and +//! register it as a named data-source group. +//! +//! One command turns a conforming store (see docs/nh-qprime-store-contract.md) +//! into a routable dataset: +//! +//! ```text +//! ddrs import /mnt/ssd1/data/icechunk/hourly_lstm_merit_unit_catchments.ic \ +//! --name hourly-lstm +//! ddrs sources use hourly-lstm && ddrs plan && ddrs run --workflow train +//! ``` +//! +//! Validation opens the store through the same `StreamflowSource::open` the +//! training loop uses, so "import succeeded" means "training will read it". +//! The coverage report is best-effort: it needs a resolvable adjacency +//! (explicit paths or a warm `.ddrs/adjacency` cache) and degrades to a +//! warning without one. + +use std::fs; +use std::path::{Path, PathBuf}; + +use crate::cli::sources; +use crate::cli::workspace::Workspace; +use crate::config::{Config, ConfigMode}; +use crate::data::dates::Frequency; +use crate::data::store::{ConusAdjacencyStore, StreamflowSource}; +use crate::error::CliError; + +pub struct ImportInput { + pub store_path: PathBuf, + /// Group name to register under `config/sources/`. `None` is only valid + /// with `dry_run`. + pub name: Option, + /// Validate + report only; write nothing. + pub dry_run: bool, + /// Overwrite an existing group of the same name. + pub force: bool, +} + +pub fn run_import( + cfg_path: Option<&Path>, + ws: &Workspace, + input: ImportInput, +) -> Result<(), CliError> { + if input.name.is_none() && !input.dry_run { + return Err(CliError::Runtime( + "pass --name to register the store, or --dry-run to \ + validate only" + .into(), + )); + } + if let Some(name) = &input.name { + // Fail on a bad name BEFORE the (possibly slow) store open. + sources::validate_name(name)?; + } + if !input.store_path.exists() { + return Err(CliError::DataSourceMissing { + path: input.store_path.clone(), + }); + } + + // ---- 1. Open & detect (same code path the training loop uses) ---- + let source = StreamflowSource::open(&input.store_path) + .map_err(|e| CliError::Runtime(format!("store failed to open: {e}")))?; + + println!("store {}", input.store_path.display()); + match &source { + StreamflowSource::Icechunk(s) => { + let (res_str, n_days) = match s.resolution { + Frequency::Daily => ("daily", s.n_time), + Frequency::Hourly => ("hourly", s.n_time / 24), + }; + let time_end = s.time_start + chrono::Duration::days(n_days as i64 - 1); + println!("format icechunk"); + println!("resolution {res_str}"); + println!( + "time {} .. {} ({} native steps)", + s.time_start, time_end, s.n_time + ); + println!("divides {}", s.index.len()); + + // ---- 2. Contract checks ---- + match s.qr_units() { + Some(u) if u == "m^3/s" => println!("Qr units m^3/s"), + Some(u) => println!( + "Qr units WARNING: {u:?} (contract expects \"m^3/s\"; \ + the solver will treat values as m³/s regardless)" + ), + None => println!( + "Qr units WARNING: no units attribute (contract expects \ + \"m^3/s\")" + ), + } + sample_read(s)?; + + // ---- 3. Coverage report (best-effort) ---- + coverage_report(cfg_path, ws, s); + } + StreamflowSource::GlobalZarr(_) => { + println!("format global zarr v2 (daily)"); + println!( + "note detailed contract validation and coverage are \ + icechunk-only; open succeeded, which exercises the same \ + reader the training loop uses" + ); + } + } + + // ---- 4. Register ---- + if input.dry_run { + println!("dry-run no group written"); + return Ok(()); + } + let name = input.name.expect("checked at entry"); + let cfg = cfg_path.ok_or_else(|| CliError::ConfigInvalid { + path: ".".into(), + source: "no ddrs.yaml found — registration copies its data_sources \ + block. Run inside a ddrs workspace or pass --config." + .into(), + })?; + let cfg_text = fs::read_to_string(cfg)?; + let block = sources::extract_block(&cfg_text, cfg)?; + let swapped = swap_streamflow_line(&block, &input.store_path)?; + let dest = sources::save_block(cfg, &name, &swapped, input.force)?; + println!("registered {}", dest.display()); + println!("activate ddrs sources use {name}"); + Ok(()) +} + +/// Read a tiny sample (first 5 divides × up to 3 days) and require finite, +/// positive values — catches unit disasters and all-NaN stores. +fn sample_read(s: &crate::data::store::StreamflowStore) -> Result<(), CliError> { + let comids: Vec<_> = s.index.ids().iter().take(5).copied().collect(); + let n_days_native = match s.resolution { + Frequency::Daily => s.n_time, + Frequency::Hourly => s.n_time / 24, + }; + let n_days = n_days_native.min(3); + let q = s + .read_window_daily(s.time_start, n_days, &comids) + .map_err(|e| CliError::Runtime(format!("sample read failed: {e}")))?; + for &v in q.iter() { + if !v.is_finite() || v <= 0.0 { + return Err(CliError::Runtime(format!( + "sample read violates the contract: value {v} (must be \ + finite and > 0; producers floor to 1e-6)" + ))); + } + } + println!( + "sample {} COMIDs × {} days: finite, positive ✓", + comids.len(), + n_days + ); + Ok(()) +} + +/// Intersect the store's divide_ids with the resolved CONUS adjacency and +/// report coverage. Best-effort: any failure (no config, unreadable +/// adjacency) prints a warning instead of failing the import. NOTE: with a +/// fabric-only config and a cold cache this triggers the managed adjacency +/// build (~10 s CONUS), same as `ddrs plan`. +fn coverage_report( + cfg_path: Option<&Path>, + ws: &Workspace, + s: &crate::data::store::StreamflowStore, +) { + let Some(cfg_path) = cfg_path else { + println!("coverage skipped (no ddrs.yaml — run inside a workspace for a report)"); + return; + }; + let resolved = Config::from_yaml_file_with_mode(cfg_path, ConfigMode::Training) + .map_err(|e| e.to_string()) + .and_then(|config| { + crate::cli::plan::resolve_adjacency(&config, cfg_path, ws) + .map_err(|e| e.to_string()) + }) + .and_then(|resolved| { + ConusAdjacencyStore::open(&resolved.conus).map_err(|e| e.to_string()) + }); + match resolved { + Ok(conus) => { + let total = conus.order.len(); + let covered = conus.order.iter().filter(|c| s.index.contains(c)).count(); + let pct = 100.0 * covered as f64 / total.max(1) as f64; + println!( + "coverage {covered}/{total} fabric COMIDs ({pct:.1}%); \ + the rest read as 0.001 m³/s fill" + ); + } + Err(e) => println!("coverage skipped ({e})"), + } +} + +/// Replace the value of the `streamflow:` key inside a `data_sources:` block, +/// preserving indentation and every other line (comments included). +fn swap_streamflow_line(block: &str, store_path: &Path) -> Result { + let mut out = String::new(); + let mut swapped = false; + for line in block.lines() { + let trimmed = line.trim_start(); + if !swapped && trimmed.starts_with("streamflow:") { + let indent = &line[..line.len() - trimmed.len()]; + out.push_str(&format!("{indent}streamflow: {}\n", store_path.display())); + swapped = true; + } else { + out.push_str(line); + out.push('\n'); + } + } + if !swapped { + return Err(CliError::Runtime( + "config's data_sources block has no `streamflow:` key".into(), + )); + } + Ok(out) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn swap_streamflow_preserves_everything_else() { + let block = "\ +data_sources: + attributes: /a.nc + # comment stays + streamflow: /old.ic + observations: /obs +"; + let out = swap_streamflow_line(block, Path::new("/new/store.ic")).unwrap(); + assert!(out.contains("streamflow: /new/store.ic")); + assert!(!out.contains("/old.ic")); + assert!(out.contains("# comment stays")); + assert!(out.contains("attributes: /a.nc")); + assert!(out.contains("observations: /obs")); + } + + #[test] + fn swap_errors_without_streamflow_key() { + let err = swap_streamflow_line("data_sources:\n gages: /g.csv\n", Path::new("/x")) + .unwrap_err(); + assert!(err.to_string().contains("streamflow")); + } +} +``` + +Check the export exists: `grep -n "pub use zarr::" src/data/store/mod.rs` — `ConusAdjacencyStore` is already re-exported (`mod.rs:23`). `StreamflowStore` is re-exported at `mod.rs:21`. + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `cargo test --test import_cmd; cargo test --lib swap_streamflow; cargo test --lib sources` +Expected: all PASS (the sources.rs tests confirm `run_save` still behaves after the `save_block` split). + +- [ ] **Step 6: Commit** + +```bash +git add src/cli/import.rs src/cli/mod.rs src/cli/sources.rs src/cli/plan.rs tests/import_cmd.rs +git commit -m "feat(cli): ddrs import — validate Q' store contract + register source group" +``` + +--- + +### Task 7: Wire the `import` subcommand into the binary + +**Files:** +- Modify: `src/bin/ddrs.rs:48-119` (Cmd enum), `:150-276` (dispatch) + +- [ ] **Step 1: Add the subcommand variant** + +In the `Cmd` enum (after `Show`, before `Sources`): + +```rust + /// Validate a Q' store against the DDR store contract + /// (docs/nh-qprime-store-contract.md) and register it as a data-source + /// group under config/sources/. + Import { + /// Path to the Q' store (icechunk repo or global zarr). + store: PathBuf, + /// Group name to register (omit together with --dry-run to validate only). + #[arg(long)] name: Option, + /// Validate and report only; don't write a source group. + #[arg(long)] dry_run: bool, + /// Overwrite an existing group with the same name. + #[arg(long)] force: bool, + }, +``` + +- [ ] **Step 2: Add the dispatch arm** + +In `dispatch()`'s match (after the `Cmd::Show` arm): + +```rust + Cmd::Import { store, name, dry_run, force } => { + ddrs::cli::import::run_import( + cfg_path.as_deref(), + &ws, + ddrs::cli::import::ImportInput { + store_path: store, + name, + dry_run, + force, + }, + ) + } +``` + +- [ ] **Step 3: Verify it builds and self-documents** + +Run: `cargo run --release --bin ddrs -- import --help` +Expected: help text showing `store`, `--name`, `--dry-run`, `--force`. + +Run: `cargo run --release --bin ddrs -- import tests/fixtures/qr_hourly.ic --dry-run` +Expected: report with `resolution hourly`, `divides 4`, `sample … ✓`, `coverage skipped (…)` (the repo ddrs.yaml, if present, points at real adjacency — either outcome of coverage is acceptable here), `dry-run no group written`. Exit 0. + +- [ ] **Step 4: Commit** + +```bash +git add src/bin/ddrs.rs +git commit -m "feat(cli): wire ddrs import subcommand" +``` + +--- + +### Task 8: Validate + register the real stores + +Everything below runs the just-built binary from the working tree — **do not use a stale `~/.cargo/bin/ddrs`** (CLAUDE.md stale-binary trap). Refresh it first. + +- [ ] **Step 1: Refresh the installed binary** + +```bash +cargo install --path . +``` + +- [ ] **Step 2: Dry-run all four unit-catchment stores** + +```bash +ddrs import /mnt/ssd1/data/icechunk/merit_dhbv2_UH_retrospective.ic --dry-run +ddrs import /mnt/ssd1/data/icechunk/daily_dhbv2_merit_unit_catchments.ic --dry-run +ddrs import /mnt/ssd1/data/icechunk/daily_lstm_merit_unit_catchments.ic --dry-run +ddrs import /mnt/ssd1/data/icechunk/hourly_lstm_merit_unit_catchments.ic --dry-run +``` + +Expected, per store: +- UH retrospective (known-good control): `resolution daily`, `divides 197088`, sample ✓. +- daily dHBV2: `resolution daily`, `divides 288421`, time starting 1980-01-01. +- daily LSTM: `resolution daily`, `divides 288421`, time starting 1981-01-01. +- hourly LSTM: `resolution hourly`, `divides 197088`, `350640 native steps`. +- Coverage line on each (the workspace has a warm adjacency cache) — expect the 288,421-divide stores to cover ~83% of the 346,321-reach CONUS fabric and the 197,088-divide stores proportionally less; any number is fine, the point is the report prints. + +The hourly open includes a full contiguity scan of 350,640 time values — expect a few seconds, not minutes. If any store FAILS validation, stop: that's a spec-vs-reality divergence to investigate, not to code around. + +- [ ] **Step 3: Register the two LSTM groups** + +```bash +ddrs import /mnt/ssd1/data/icechunk/daily_lstm_merit_unit_catchments.ic --name daily-lstm +ddrs import /mnt/ssd1/data/icechunk/hourly_lstm_merit_unit_catchments.ic --name hourly-lstm +ddrs sources list +``` + +Expected: both groups listed; `config/sources/daily-lstm.yaml` and `config/sources/hourly-lstm.yaml` exist and differ from `conus.yaml` only in the `streamflow:` line. + +- [ ] **Step 4: Commit the groups** + +```bash +git add config/sources/daily-lstm.yaml config/sources/hourly-lstm.yaml +git commit -m "config: daily-lstm + hourly-lstm data-source groups (via ddrs import)" +``` + +--- + +### Task 9: Smoke trains (the success criterion) + +Short trains proving each read path end-to-end: finite loss, directory-style checkpoints, and the `streamflow resolution:` log line. Keep windows SHORT — the summed-Q' baseline that `ddrs plan` computes reads the full eval window, and on the hourly store that's 24× the daily I/O. + +- [ ] **Step 1: Back up the current ddrs.yaml** + +```bash +cp ddrs.yaml /tmp/ddrs.yaml.pre-nh-smoke +``` + +- [ ] **Step 2: Daily-LSTM smoke train** + +```bash +ddrs sources use daily-lstm +``` + +Edit `ddrs.yaml`: `mode: training`, `workflow: train`, `experiment.epochs: 1`, and a short in-range window, e.g. `start_time: 1995/10/01`, `end_time: 1996/09/30` (the daily-LSTM store starts **1981-01-01** — any window from 1981 on works; do NOT use 1980). Remove any `kan_head.disaggregation:` block and any `experiment.checkpoint:` left from prior experiments. + +```bash +ddrs plan --workflow train +ddrs run --workflow train --max-mini-batches 2 2>&1 | tee /tmp/smoke_daily_lstm.log +grep "streamflow resolution" /tmp/smoke_daily_lstm.log +``` + +Expected: +- `streamflow resolution: Daily` in the log. +- Finite (non-NaN) loss values for both mini-batches. +- A run dir `.ddrs/runs//` whose checkpoints are **directories** (`checkpoints/epoch_*_mb_*/head.mpk`) — flat `.mpk` files would mean a stale binary executed. + +- [ ] **Step 3: Hourly-LSTM smoke train** + +```bash +ddrs sources use hourly-lstm +ddrs plan --workflow train +ddrs run --workflow train --max-mini-batches 2 2>&1 | tee /tmp/smoke_hourly_lstm.log +grep "streamflow resolution" /tmp/smoke_hourly_lstm.log +``` + +Expected: +- `streamflow resolution: Hourly` — the proof the new path actually ran. +- Finite loss, directory-style checkpoints, no disagg-rejection error (the block was removed in Step 2; if it errors, that's the Task 4 guard working — fix the config, not the guard). +- Note wall-clock vs the daily run in the handoff: `collate` reads the hourly store twice per batch (hourly + 24h-mean daily), so meaningfully slower is expected; hours-per-batch is not, and would justify the chunk-aligned-read follow-up flagged in the spec. + +- [ ] **Step 4: Disagg-guard negative test (config level)** + +Temporarily add to `ddrs.yaml` under `kan_head:`: + +```yaml + disaggregation: + use_precip: false +``` + +```bash +ddrs run --workflow train --max-mini-batches 1; echo "exit: $?" +``` + +Expected: non-zero exit with the `hourly-native; remove the disaggregation block` message. Then remove the block again. + +- [ ] **Step 5: Restore the original config and run the full regression suite** + +```bash +cp /tmp/ddrs.yaml.pre-nh-smoke ddrs.yaml +cargo test +cargo run --release --example compare_ddr_sandbox +``` + +Expected: all tests PASS; sandbox reports **ABSOLUTE MATCH** (nothing in `src/routing/`, `src/geometry.rs`, `src/sparse.rs` was touched, but the invariant demands the check). + +- [ ] **Step 6: Commit any smoke-run fallout** + +If Steps 2-5 required code fixes, they were committed as they happened. Nothing else to commit here (`.ddrs/` and `ddrs.yaml` are gitignored). + +--- + +### Task 10: Document in CLAUDE.md and close out + +**Files:** +- Modify: `CLAUDE.md` (the `### ddrs CLI` section, after the data-source-groups paragraph) + +- [ ] **Step 1: Add the import section to CLAUDE.md** + +Insert after the `ddrs sources list/save/use` code block: + +```markdown +**Importing a Q' store** (`src/cli/import.rs`): any store meeting the DDR Q' +contract (`docs/nh-qprime-store-contract.md` — `Qr(divide_id, time)` f32 +m³/s, CF `days since`/`hours since` axis) registers as a source group in one +command: + +```bash +ddrs import --dry-run # validate + coverage report only +ddrs import --name # validate + register config/sources/.yaml +``` + +The icechunk reader sniffs daily vs **hourly-native** resolution from the CF +time units (`StreamflowStore.resolution`); hourly stores are sliced natively +(no repeat-24, no disagg — `kan_head.disaggregation` + hourly source is a +config error). `daily-lstm` / `hourly-lstm` groups (NH CudaLSTM / MTS-LSTM +forwards) ship in-repo; the hourly store starts **1981-01-01**, so experiment +windows must not reach into 1980. Dataset open logs +`streamflow resolution: Daily|Hourly` — check it when validating runs. +``` + +- [ ] **Step 2: Verify the plan's spec coverage one last time** + +Re-read `docs/superpowers/specs/2026-07-01-nh-qprime-import-design.md` §§1-5 and confirm: contract doc (Task 5), sniff + three-method behavior + out-of-range hard error + hourly-alignment validation (Tasks 2-3), disagg guard + resolution log (Task 4), import command with dry-run/coverage/registration (Tasks 6-7), all-four-store validation + LSTM smoke trains + daily-path regression (Tasks 8-9). + +- [ ] **Step 3: Commit** + +```bash +git add CLAUDE.md +git commit -m "docs: ddrs import + hourly-native Q' reading in CLAUDE.md" +``` From b62a4a6c5348ee310fea9fef88a91987412af92d Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Wed, 1 Jul 2026 22:18:21 -0400 Subject: [PATCH 03/41] test: icechunk Qr fixture stores (daily/hourly/bad-units) Adds scripts/make_streamflow_fixtures.py and the three generated stores it produces under tests/fixtures/. Deterministic values let later Rust tests assert exact elements against the resolution-sniff reader. xarray normalises "hours since 1981-01-01 00:00:00" to drop the time component on write; the generator patches the zarr attr back after to_zarr() so the on-disk string is exactly "hours since 1981-01-01 00:00:00". Co-Authored-By: Claude Fable 5 --- scripts/make_streamflow_fixtures.py | 90 ++++++++++++++++++ .../manifests/1K0SX670NNMP5MB1CTF0 | Bin 0 -> 160 bytes .../manifests/3N182AV8DNMXV9K0RQHG | Bin 0 -> 253 bytes .../manifests/DRRYHR9BMCCDDD3XY4PG | Bin 0 -> 157 bytes .../repo.30720721334165.8QQRFD9DK7JXHEQX3DEG | Bin 0 -> 272 bytes tests/fixtures/qr_daily.ic/repo | Bin 0 -> 413 bytes .../snapshots/1CECHNKREP0F1RSTCMT0 | Bin 0 -> 193 bytes .../snapshots/DYKFVQ2NPCXQ0Z3HV1FG | Bin 0 -> 1034 bytes .../transactions/1CECHNKREP0F1RSTCMT0 | Bin 0 -> 118 bytes .../transactions/DYKFVQ2NPCXQ0Z3HV1FG | Bin 0 -> 230 bytes .../qr_hourly.ic/chunks/KGVS91W2STB2KKSTQ0WG | Bin 0 -> 2273 bytes .../manifests/BQR9XRGC4M0SRC5AVP7G | Bin 0 -> 160 bytes .../manifests/JZBF2VD75HAHNKHGW2DG | Bin 0 -> 490 bytes .../manifests/Z0RJM4NPDWHRJ8ZEDQFG | Bin 0 -> 158 bytes .../repo.30720721334158.WGT8Y51TH13CT8FFMHH0 | Bin 0 -> 272 bytes tests/fixtures/qr_hourly.ic/repo | Bin 0 -> 405 bytes .../snapshots/1CECHNKREP0F1RSTCMT0 | Bin 0 -> 193 bytes .../snapshots/Z0XFVR54233XSYW0W880 | Bin 0 -> 1044 bytes .../transactions/1CECHNKREP0F1RSTCMT0 | Bin 0 -> 118 bytes .../transactions/Z0XFVR54233XSYW0W880 | Bin 0 -> 232 bytes .../manifests/7429MVX5PYQ7AGDBR9Z0 | Bin 0 -> 159 bytes .../manifests/AF5ZZYBD9TQDXVMSTYN0 | Bin 0 -> 284 bytes .../manifests/Q49VEZ7AP9MTA33K2VN0 | Bin 0 -> 598 bytes .../repo.30720721334151.XQX3P8H9XJ723JMC3BF0 | Bin 0 -> 272 bytes tests/fixtures/qr_minutes.ic/repo | Bin 0 -> 408 bytes .../snapshots/1CECHNKREP0F1RSTCMT0 | Bin 0 -> 193 bytes .../snapshots/Q2YNXV9BEWYF2TKZ41VG | Bin 0 -> 1040 bytes .../transactions/1CECHNKREP0F1RSTCMT0 | Bin 0 -> 118 bytes .../transactions/Q2YNXV9BEWYF2TKZ41VG | Bin 0 -> 234 bytes 29 files changed, 90 insertions(+) create mode 100644 scripts/make_streamflow_fixtures.py create mode 100644 tests/fixtures/qr_daily.ic/manifests/1K0SX670NNMP5MB1CTF0 create mode 100644 tests/fixtures/qr_daily.ic/manifests/3N182AV8DNMXV9K0RQHG create mode 100644 tests/fixtures/qr_daily.ic/manifests/DRRYHR9BMCCDDD3XY4PG create mode 100644 tests/fixtures/qr_daily.ic/overwritten/repo.30720721334165.8QQRFD9DK7JXHEQX3DEG create mode 100644 tests/fixtures/qr_daily.ic/repo create mode 100644 tests/fixtures/qr_daily.ic/snapshots/1CECHNKREP0F1RSTCMT0 create mode 100644 tests/fixtures/qr_daily.ic/snapshots/DYKFVQ2NPCXQ0Z3HV1FG create mode 100644 tests/fixtures/qr_daily.ic/transactions/1CECHNKREP0F1RSTCMT0 create mode 100644 tests/fixtures/qr_daily.ic/transactions/DYKFVQ2NPCXQ0Z3HV1FG create mode 100644 tests/fixtures/qr_hourly.ic/chunks/KGVS91W2STB2KKSTQ0WG create mode 100644 tests/fixtures/qr_hourly.ic/manifests/BQR9XRGC4M0SRC5AVP7G create mode 100644 tests/fixtures/qr_hourly.ic/manifests/JZBF2VD75HAHNKHGW2DG create mode 100644 tests/fixtures/qr_hourly.ic/manifests/Z0RJM4NPDWHRJ8ZEDQFG create mode 100644 tests/fixtures/qr_hourly.ic/overwritten/repo.30720721334158.WGT8Y51TH13CT8FFMHH0 create mode 100644 tests/fixtures/qr_hourly.ic/repo create mode 100644 tests/fixtures/qr_hourly.ic/snapshots/1CECHNKREP0F1RSTCMT0 create mode 100644 tests/fixtures/qr_hourly.ic/snapshots/Z0XFVR54233XSYW0W880 create mode 100644 tests/fixtures/qr_hourly.ic/transactions/1CECHNKREP0F1RSTCMT0 create mode 100644 tests/fixtures/qr_hourly.ic/transactions/Z0XFVR54233XSYW0W880 create mode 100644 tests/fixtures/qr_minutes.ic/manifests/7429MVX5PYQ7AGDBR9Z0 create mode 100644 tests/fixtures/qr_minutes.ic/manifests/AF5ZZYBD9TQDXVMSTYN0 create mode 100644 tests/fixtures/qr_minutes.ic/manifests/Q49VEZ7AP9MTA33K2VN0 create mode 100644 tests/fixtures/qr_minutes.ic/overwritten/repo.30720721334151.XQX3P8H9XJ723JMC3BF0 create mode 100644 tests/fixtures/qr_minutes.ic/repo create mode 100644 tests/fixtures/qr_minutes.ic/snapshots/1CECHNKREP0F1RSTCMT0 create mode 100644 tests/fixtures/qr_minutes.ic/snapshots/Q2YNXV9BEWYF2TKZ41VG create mode 100644 tests/fixtures/qr_minutes.ic/transactions/1CECHNKREP0F1RSTCMT0 create mode 100644 tests/fixtures/qr_minutes.ic/transactions/Q2YNXV9BEWYF2TKZ41VG diff --git a/scripts/make_streamflow_fixtures.py b/scripts/make_streamflow_fixtures.py new file mode 100644 index 0000000..22c0a21 --- /dev/null +++ b/scripts/make_streamflow_fixtures.py @@ -0,0 +1,90 @@ +"""Write tiny icechunk Qr fixture stores for ddrs integration tests. + +Run under DDR's uv venv (it has icechunk + xarray): + + cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/make_streamflow_fixtures.py + +Layout matches the DDR Q' store contract (docs/nh-qprime-store-contract.md): +Qr(divide_id, time) f32 m^3/s, divide_id int64, CF int64 time axis. + +Deterministic values so tests can assert exact elements: + qr_daily.ic : 4 divides x 10 days, Qr[j, t] = (j+1)*100 + t + qr_hourly.ic : 4 divides x 240 hours, Qr[j, h] = (j+1)*1000 + h + qr_minutes.ic : sniff-rejection fixture (units "minutes since ...") + +Note: xarray normalises "hours since 1981-01-01 00:00:00" to drop the time +component on write. We patch the zarr attr back to the full string after +to_zarr() so the on-disk CF units string is exactly as documented above. +""" +from pathlib import Path +import shutil + +import icechunk +import numpy as np +import xarray as xr +import zarr + +FIXTURES = Path(__file__).resolve().parent.parent / "tests" / "fixtures" +DIVIDES = np.array([101, 102, 103, 104], dtype=np.int64) + + +def write_store( + path: Path, + times: np.ndarray, + qr: np.ndarray, + time_units: str, + *, + time_units_on_disk: str | None = None, +) -> None: + shutil.rmtree(path, ignore_errors=True) + storage = icechunk.local_filesystem_storage(str(path)) + repo = icechunk.Repository.create(storage) + session = repo.writable_session("main") + ds = xr.Dataset( + data_vars={ + "Qr": (["divide_id", "time"], qr.astype(np.float32), {"units": "m^3/s"}), + }, + coords={ + "divide_id": ("divide_id", DIVIDES), + "time": ("time", times), + }, + attrs={"units": "m^3/s", "source": "ddrs test fixture"}, + ) + ds.to_zarr( + session.store, + mode="w", + encoding={"time": {"units": time_units, "dtype": "int64"}}, + ) + # Patch the on-disk units attr if xarray normalised it (e.g. strips + # " 00:00:00" from "hours since 1981-01-01 00:00:00"). + if time_units_on_disk is not None: + z = zarr.open_group(session.store, mode="r+") + z["time"].attrs["units"] = time_units_on_disk + session.commit("fixture") + print(f"wrote {path}") + + +def main() -> None: + n_days = 10 + daily_times = np.datetime64("1981-01-01") + np.arange(n_days).astype("timedelta64[D]") + daily = (np.arange(4)[:, None] + 1) * 100 + np.arange(n_days)[None, :] + write_store(FIXTURES / "qr_daily.ic", daily_times, daily, "days since 1981-01-01") + + n_hours = n_days * 24 + hourly_times = np.datetime64("1981-01-01T00") + np.arange(n_hours).astype("timedelta64[h]") + hourly = (np.arange(4)[:, None] + 1) * 1000 + np.arange(n_hours)[None, :] + write_store( + FIXTURES / "qr_hourly.ic", hourly_times, hourly, + "hours since 1981-01-01 00:00:00", + time_units_on_disk="hours since 1981-01-01 00:00:00", + ) + + # Same data, unsupported units string — exercises the sniff hard-error. + write_store( + FIXTURES / "qr_minutes.ic", hourly_times[:48], hourly[:, :48], + "minutes since 1981-01-01", + ) + + +if __name__ == "__main__": + main() diff --git a/tests/fixtures/qr_daily.ic/manifests/1K0SX670NNMP5MB1CTF0 b/tests/fixtures/qr_daily.ic/manifests/1K0SX670NNMP5MB1CTF0 new file mode 100644 index 0000000000000000000000000000000000000000..5e492e532340b43a8d0ca846cfc08cf79afbef76 GIT binary patch literal 160 zcmeZtcKtAad6%}Y&wZ}Qjni7n;N@8ADC?eT$Y+oIp;2A^NxpIr6Ii*d2;my_308eJDQ?~tggs6LcC ze`!-?D9@q0+eJ$@n`=M1VWT46o|Jko-0G@q&19}c7NTqn{yhG&>l(gEdfl^cHDxda E0MhP4mH+?% literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_daily.ic/manifests/3N182AV8DNMXV9K0RQHG b/tests/fixtures/qr_daily.ic/manifests/3N182AV8DNMXV9K0RQHG new file mode 100644 index 0000000000000000000000000000000000000000..4cbfd9814b18ae5bcb5171c077d8c9361eebdff2 GIT binary patch literal 253 zcmeZtcKtAad6%(r>7k(+sUS;EoBEI=_v2B4$>0}D`!1IWE&$Lf|JvwI2;P=uobC;(EE z1H>SM6&A=cGccVJms_y3_1ORa|LZf9e#q~dolxsMyW1s~tDG_TTY?x#t+o+y@{uzuB-enry%0jo{J_KL3Q6&>XjZQ(P7)LLF97|q~P fYsyThT9y{Y$H2tE%3zhJoUwJG%I{znW&vgZOgU4A literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_daily.ic/manifests/DRRYHR9BMCCDDD3XY4PG b/tests/fixtures/qr_daily.ic/manifests/DRRYHR9BMCCDDD3XY4PG new file mode 100644 index 0000000000000000000000000000000000000000..c8259f4a1eeb1429996a34bdc68cc075571776b4 GIT binary patch literal 157 zcmeZtcKtAad6%5`SRnyDzRe};4ovu?4S)}cEl*C@28lcXJgQwO!&MS_7(fGJCqroOg{gUUF*oQprGv@A;Y+5qKN>`&dXcvR- z(q6U8A2h`n1P%!PoThPLqqFYX-j>cEkB^;@x|p71((<%Dwz9n>;cY->=vL-K&(@tY Q2~04bCKNH-c#ne|0M0FRS^xk5 literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_daily.ic/repo b/tests/fixtures/qr_daily.ic/repo new file mode 100644 index 0000000000000000000000000000000000000000..8ab745faf7ef632daef5296cd959ccfafdeaf19a GIT binary patch literal 413 zcmV;O0b>41Lq+hPr;0;JRZdH3V=Xc+FfKD7ARr(hARr(hARr(hARr(D1_3CwFZ}>m zZ3_U_lSDi~#i}*u{6)R{I#d705#At@Bc`0-4?IaxyuHzOYK7ALOXkgDn2{m=z$V&r zlLdn>LZgE0U$EPMewhL&06PFT0FzGVIMbi|$KUc_ev@DJ&wif#RQXx6T<_e?xz0`B z&g{9iSSyOZ3tz5pw1kzvYSv8an#U|r`n;5>G?A(xS$qy(#K~l8_$)AlMxGZEkw796 z$oq(6Nd5|AP$QN=NPzzW_ue}S9|qJwU@-u)j-Ze2fq!s96CLaeE7Il(t+j)+$_C5PW(ptU2c#n3tkRf-n0Qbnd+Cpg%+@g>CuoD4Joxu7TQwwtD$%@=p~ zMN1I&$~y-F1=kM>ztF5=6FtP(pT(wuO}L54m2esTxDQiq*W?rza&Ohdoh5Mj;(s)c HzhVpWG-$mx literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_daily.ic/snapshots/1CECHNKREP0F1RSTCMT0 b/tests/fixtures/qr_daily.ic/snapshots/1CECHNKREP0F1RSTCMT0 new file mode 100644 index 0000000000000000000000000000000000000000..38958dae63e548c20e26cf2881a437c9c7094e04 GIT binary patch literal 193 zcmeZtcKtAad6%wj~C8-mO)x>8s zrY?E$XT=YPm z)gA!mvZX}u*rqjrBF^mlv;8$z58l)PhS*hxn)mpPAIEap|CS4CyL*d5jmWko>;BrG z+w>kGHj`%FcV|fxS{KEawAV&YO-Krn0GI%t05QhcdG$^Ux@xQQSTaUCInYI|j4MfD zvy&&{N^aGGo7H+QYF!$u-WcUP z8=G54xtT)6q);&_1`Se?vby@k4`*&NuH<)X2Y&Q|HPzlb>Br4hTuB)C*|6w~B{_OS zx$S-?D0gq=WL(K^)~%y5#*bc{nvX>{4;5FEg512(IIgy&jEpO3y*urU(eHBLY|+it z)y-pdRKuY*Ix%KCGyx%C62rtGRSIVA^te_6o)3|PxbOkOtH1&QNzZ`KVt90T7&9wt9o8r)#z4l>ZVe)cu*-FSl$uE#S8qWd4xINe~e`Nh<6GgJJ@K z7i}v207fHG7=}{f4|16lPC4-B{1pDvU)tr8>CyuR7GNYu4jfl2Xi{b-f2aYOF-#gH zGa@omQ&|pR0+AV`Di0Ds;DM;-Ic7)&5=kH;kw`Fcf|Q4H0N@ZQSx$5w(!)Yq{a~Oj zT?33ZNVC+)qnAzK5mI^?B7QdujtV7iRtky)+aH{7E5$Y)=$ppJMW8cgxyc6*ApM4m z;`$p4F*SGM?~dw?pO%@cvYr>KHEeC9lndJ?Yb&uG`l>-)Oa_is>&LMAJd=cvtHRo< zWJ2Y0=gaKdB%s_pNw*W;MlL%1QOD4jZ+2p3!kF~`_sxTK-7?xW%D)n9b@SZ-@4uf> zG;w_lM6eGc{K=TEmTv43?hC2+#BC^~scNLF)2Ln)z4f`OitmHuE(>)|0Tx0k0w+mq3vBnfj%|pRV8+aLY!EMirB1ye`>Zun E4RJN}2><{9 literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_daily.ic/transactions/1CECHNKREP0F1RSTCMT0 b/tests/fixtures/qr_daily.ic/transactions/1CECHNKREP0F1RSTCMT0 new file mode 100644 index 0000000000000000000000000000000000000000..2a1bf888e6d96550cc74c6fe27734d90dcea1679 GIT binary patch literal 118 zcmeZtcKtAad6%yzm|Rf{?xMTp#cX6SBtAn=|tV#tK5}m`7qCGv*c!C Lun0dm)0i0m-Uulg literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_daily.ic/transactions/DYKFVQ2NPCXQ0Z3HV1FG b/tests/fixtures/qr_daily.ic/transactions/DYKFVQ2NPCXQ0Z3HV1FG new file mode 100644 index 0000000000000000000000000000000000000000..88ca3d0bacbaf21e5b0d8bd106805913cf99cf8d GIT binary patch literal 230 zcmeZtcKtAad6%IMZ%*ln_~&wYnpLRO znd3b^jMiIkMXhvCQD#n@lz1WHj;#8ZvT}*@w_V*2vpQ}nd6$>`;myTwUXSJ2D^GaJ zSO|*>vCDAJ6+ZJcyos4LEQy(E%Qn{a%A49-!kGjZ^p!WTH!4a@ciuR)!F-A2)paZ7 Yn9C2O`WJTH{(Eju-m_+D(K(jJ0JjKR2><{9 literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_hourly.ic/chunks/KGVS91W2STB2KKSTQ0WG b/tests/fixtures/qr_hourly.ic/chunks/KGVS91W2STB2KKSTQ0WG new file mode 100644 index 0000000000000000000000000000000000000000..8780bd7bd6e522cf51ad98ced95a3ca8370eac19 GIT binary patch literal 2273 zcmWN|dsLE#0swFjQ#A9DCZ_cHfY6vC7E8&`Of=DHK5DzUBjcd9X>&&d?mh9TGZ9VA z7la~aA=-Aevs>2b>*U`BxWI37 zasIFKC|S%e(=GL>s5Hr@)L zU28@{ew8vV(kHYJOpb_bI2PP!zRZrnzeQK>$V4 zi?dy2=jBRRacA)K+)ihei=?p=7ugu>O; zSykq zR%r8foRPzR4gWw(RooANe}bQ4369e1+@JGxpr0y!45PA&zeDl`8gG+QOL~RHmGRrm z-By^WBuSwVYH};kW|0Y?V0sY86;lX!JF?^;u_1`zl%g4vu{%QS)ys=P@YS z@j2CbdtDFqqY!=R2XG02)awecpTCZsP%Ynqi(Ds^%N2YscB0|G*bVe+bL~Q{a#_7h zo%;F9GX8t6v+AMoj!GzDA-e*&X6r(4v7BvCd;?8Y{R|9s3~vmU@$+5psn3tkJphYb zY4i4)+8&-=v`Nqe)LI z*6kogJyLr(cG7$@vpLp1JDdE+Zbdn-E>^VHlEkSi)$c*W7m|+GrRduYN$PUJ3P;r4 zUA}REGNpq#$N|;)Hh3KKH@|b^{AX|_=HGjsIrUi5jvOFOS&!rv3EAz&BZ(74_GvV1 z;2dk>$Fx}_EOJ6nBo+FbjxdVGihG*YFh50c*~t*B%N(CDT66NjR+lNBb#kW3%N}~Y znAgM^iqLX)6pi}0Wl?q=dbUoSx`}+sb5!z7-|8A-8Od#E`y?E^ zur~#YRg*XvXG5w3gH^uc*eO=f_lQ&g5KV-!?N`{zGFylFH!B>Z+?GPoRHsbSVQ~o{ zfWDON0t+?14oD6LH5_70$r>^E4Aejf}4-Y8CD`$ab zJ4M-F)70iB8o1PG{hLn4QIW!OI)_oI&oiE;lg4;o=#7TcG}2QYQ5@3KoyQOfLw;9W zamT&HvYXL!&92lPq_)XX)nt0~=54Qsk*OF(N!VAOgh!6)tq~8ac|OP<@BYa1i(Jvc z1LCk?OEw2M3lY)Sh3wA*W zA%nWD({CitZpyYt&8IT| z9_u#Drv5^pWL|RP718%Sr;I`GH@9Gf1)0w!t0BuPtwvOW3w)NAV2^Rc;K*%4Io>xv zkiM$zmL�%k+CwjYNBU#O`(6Zl|ZJLuk3e+(}uDHf8Sdh`0J@GyfKk-|ktFwKOue$*X{(G2l`^s_|M0 z6$Rlp@2osr1I!up&5N4{A7?~u6IV5+47`qO*Yq^18*!I{3>iKv6#c~i^X3B`*GXNy zrMMjYdJ(-vfGJoX-u3`K*K(O-i+!9vc=-=-HpHDxUq~lNs_lpH@G&SJ=d6L1TY;`X zoNY!j*K(Qa412ven;!1mNW#}s>WPwX;6tltkOf@5YT65KheKBHVt-xh#s0i@7m9lP z!Z3of(dzM%A0!q@l@0g$>eeTZN07TM)7`1vsTm(5M}{Uk!C7oTKJmVW1HI=O=$$Ck zjBkZGjzknIKy>ZM3A3@3!3|+wLla`TcgXFc0Y^dh3ZWt49T%98B3gxQ{0QnrftNm(*d|0x$V?Ap83l@==KsL!do9p$l=b@{6) z4l~Z`@8-I9`MS!Kl{%9fd-ayMnSJ=iXLRE5!3u76hJ?q3rcv)oueL0d`yI@}EWiu^ DW<)w= literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_hourly.ic/manifests/JZBF2VD75HAHNKHGW2DG b/tests/fixtures/qr_hourly.ic/manifests/JZBF2VD75HAHNKHGW2DG new file mode 100644 index 0000000000000000000000000000000000000000..b5d698ce546b68c6e01af43198e8c1111f7a8521 GIT binary patch literal 490 zcmeZtcKtAad6%wbT_yV_j;-tMy3m*#4l z>)qMTeBNZUkI``vV;_V4Up9H`ZNFmVt+Rg1CNHh!Q;fVc=9g^pRGS`RHgi!A^1cHRxUowIT8wOd&$=Z4+NSXg`QX4=fyu$w6pZLi%(>XZ$;kh-wFyP?-(3in>U7L_?S^jbt>?bWMck+GpyLjrBDUI}7scpt*Vzzhsk1_m|;oBC{@ RN6#9)rM))UyBaVU0sz;Ixmf@J literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_hourly.ic/manifests/Z0RJM4NPDWHRJ8ZEDQFG b/tests/fixtures/qr_hourly.ic/manifests/Z0RJM4NPDWHRJ8ZEDQFG new file mode 100644 index 0000000000000000000000000000000000000000..3e0b92b5009e8c4fe440c773f3049e233e26a75e GIT binary patch literal 158 zcmeZtcKtAad6%x%hKnQnhZHU?)NectB|t9ey#&R3FXkN^O= CwLIDY literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_hourly.ic/overwritten/repo.30720721334158.WGT8Y51TH13CT8FFMHH0 b/tests/fixtures/qr_hourly.ic/overwritten/repo.30720721334158.WGT8Y51TH13CT8FFMHH0 new file mode 100644 index 0000000000000000000000000000000000000000..ce0b4fd60a88eefc1cbe0fb38402f11e33ce1093 GIT binary patch literal 272 zcmeZtcKtAad6%&rO z+YU?;oxmyWF~_3L`Oxl8K9hhY5f!W_s%?(^UG1+haoYv8;Qw>l7dC9u`_h)Y@MOr$ z@|`JSoo$B4m~Ql^mg>#>-l)0VS2Hkm59gWRHs=-_SY3Nwxu)yYLl$ZG9VM~Xmj)QN z^khWDG}QPot@l~{_TZ^CkMoM7Ihr1KN>tb+sbBKkQu~m%f5zOMkxfgcSm|o?2JK?d zUD~Vm_=BbxgTMj7pVKrBY;@LL+uPFlg&(}W^s8}D(D0{~pJcCY{d literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_hourly.ic/repo b/tests/fixtures/qr_hourly.ic/repo new file mode 100644 index 0000000000000000000000000000000000000000..a33fb1df75aae0bf6d45b3f9b85a37d69fc191d7 GIT binary patch literal 405 zcmV;G0c!q9Lq+hPr;0;JRZdH3V=Xc+FfKD7ARr(hARr(hARr(hARr(D1_3CwFZ}>m zEeim)lR`I8ty%-CXaur&HW4c+J&{2U)C7^#9c1q1@6?2&`QHf(#<5#6kY&jC|HfOg zUqcbuigx?EFH? z3x5}v>)b)dUo~rE`Qz?b;d8m{(f+(dF9zLa!C;C1wL;{6KpzjmHK=Cga zXQ8v!(#Gf%B`MbYvIG#5j@oW6Q?5g6wP!pPp zO@wlkte8@`xdU@^x?Zvt3uIZ9YU#1Gq?8gsK8B&FirHVFbMFOt`4ZrTrC8a~zHG-r z3`ZLaVPL~QfbsMOBOowBVvwc>Cpg%+@g>CuoD4Joxu7TQwwtD$%@=p~MN1I&$~y-F z1=kNYywF#%u^z(hPhiv5CY+0jlyEQoU=Q=zZuqwj~C8-UniIWVN zwM`3*wf0T?wIiMTh?x?rl*^P!jgNf!Ck9ps3M(#nE21jgEwb^2!V4zt4(A}Rl? zy-cC+Snc!8X7-m zB_9ALGp9uG*rqi?^pE#xZ}6&kL}(k_#Ak4qMZs?T-gt+L{i!dk9+ zZ|LUPo2hklo7rj=WmYgUD;Svx6l5ZGb@hv%1-+Pznfz|;z>i+=M78%$`fc+SGs%IU zjf=)ya-%qO)9#moZuizr#!PmzY8{m^e)Q&?_*_);Rxy(l+$K(I+v-ck$e2m%-D&3y z{q6?N7u8^0)jU^6TRGH5C&nf?`C)#LtOzrAdfV&8%8N=!DhOE_*`kC8kRJnYa>7wb z@FIctqksdMo-q80q9kGXBVlyu)Fl+n&-#gJwMDgxQ#BTA#)opafB>?7uY za`ndi0B}569XTN4r3Y6g=o9}3&H5SqE@9|H^Xr&uEaj~C z`nqXvi^8V$?w2#IomBI>nV88N|NjovrUym&L8w!YMzHC1{Gf2>SAU6L^auE3M)C&q zER0(4qNuNUKYHV5HO*x~?_1}O@itoIz#PE=V?>eA1a>*VDb)mi8zSZxjbt$KvC08j zaN5zqQ}d%!r(XbxGz!CzO8i5pOCgVff5oriSN*3^pHiPJOz7|cgB-DKbwf+W%mjdd zs3Dm#3>uP=5t%6|4F_NXk)fh04-!D|i6X~AW=I7RNgyJTNFb6XNO>p+0IW^Ez6)@_ zknchzaAF{$?qg;QQeleJ@z_ny5t3LLB66F(Ol6%mE3b-#*q`%9mtvd_ET{2tQ=l9b zZmj|6mBV$&xZcLnofIzsyRBxUOw0IH3CfG-DYibE+7)rpD*G9legSD$?ty953i7PZ zscGF4tMCXb&I6;r^M&jPCg9vWO@9+YD;ND*QgSw!W#40EcANAP5DbHV-8u#~*1wo+ z^~m6OuICoXeu93~MrI}=vXKF`wn}@NfPdsQly}kiUERo8p<6}s6=9BluCDS$y{3z) zb(bAcXh&TF7{LVuAX^yzm|Rf{?xMTp#cX6SBtAn=|tV#tK5}m`7qCGv*c!C Lun0dm)0i0m-Uulg literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_hourly.ic/transactions/Z0XFVR54233XSYW0W880 b/tests/fixtures/qr_hourly.ic/transactions/Z0XFVR54233XSYW0W880 new file mode 100644 index 0000000000000000000000000000000000000000..b2c1d81a1299573a481b61d6c2dae3f462de39df GIT binary patch literal 232 zcmeZtcKtAad6%xB2d zOl=8WO=aE1j}{&H^zvX6^J=D*jM3MuavK+RREP>P=qrCSN{(q3| zg~|Fevhuf=z37;BO!+=bVr$XLgBiO5CkaeqlJ`|Twng;QMZ?PAP(|kU4+r0~elzPV z`q`j&^kGgl^UaIrA6-<@3RhP?D3io(F;z52aP>wDQ8or=9(~^D4y$=pZ_ZbeXpjH^ DFk?Ly literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_minutes.ic/manifests/AF5ZZYBD9TQDXVMSTYN0 b/tests/fixtures/qr_minutes.ic/manifests/AF5ZZYBD9TQDXVMSTYN0 new file mode 100644 index 0000000000000000000000000000000000000000..ec6c7ca3f4b3a255aa3dea13aa5a2c924a87e85c GIT binary patch literal 284 zcmeZtcKtAad6%IDvs_FE0bzha9B^rjmSjZI>`Hs4n8&TBR2&J(+$V{?1<*|W>isyCU>obBWLEhb;jeEO`{ z%r!QjKR!D=liytabj^yN4$mIHtXTHgx7}Da&uZGsfXj_O$2WgGb8I$my4jr^oy`l= zm}mB?Kc3lRE_p6x$H@paXJ0lW$@52z#q;&%>?|nWc&2q0cZ%M;u5_(?Ar`?8mnF^Q zG8H`$QWMG%9=XnFN7t^d2U;gWnZkcAJK}rZw`2A_-}c#aX7iZuoXugbYtEY&WX_rA zl*e`Fi)&~1ylw&Uz1@7`mg16nMdE^bQF>A_OZ0?dT4F?YyqK+$_vsEx;pFnkhkfVF zmdJa=m)m)J-fe;HdvEh?x7;q7SG-*?FFH^9&ayn=J8gHwcDy|;G~IT(qF(iMS-oOC zAS8wvUg?|OyA{QAYK&y dU>!d&8X1@v7+4uD@0|7EQro0A{(?LXJOC~&?w|kw literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_minutes.ic/overwritten/repo.30720721334151.XQX3P8H9XJ723JMC3BF0 b/tests/fixtures/qr_minutes.ic/overwritten/repo.30720721334151.XQX3P8H9XJ723JMC3BF0 new file mode 100644 index 0000000000000000000000000000000000000000..7bd670bbba36ae16c5beaa4e134025a830320f5e GIT binary patch literal 272 zcmeZtcKtAad6%Xb8fML)wSo9Yr0-NWRbVuQ4)K7YJhG< zOGZRYNr4a3dY{E>51#UQl2;tfk@UFHpvNXj{gUUF*oVCRGv@A;Y+5qKN>`&dXcvR- z(q6U4A2h`n1P%!PoThPLqqFYX-j>cEkB^;@x|p71((<%Dwz9n>;cY->=vL-K&(@tY Q2~04bCKNH-c#ne|0G$|fu>b%7 literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_minutes.ic/repo b/tests/fixtures/qr_minutes.ic/repo new file mode 100644 index 0000000000000000000000000000000000000000..a3697632a6616794d4a3a29cf19e3f4ad261cee3 GIT binary patch literal 408 zcmV;J0cZY6Lq+hPr;0;JRZdH3V=Xc+FfKD7ARr(hARr(hARr(hARr(D1_3CwFZ}>m zMGFA7Q$sp1<*GG6p1CG)Y_b5!9Xfm%7__7yrxSkO_P&EwUXT0U#=T|KM~W4|tgGr9 z4bXgS&@ms{a_`~G6ej>W05t&fIp=x)+~4{!f999{v!CUq$;*_NHOzGHZqIb?`S#|| zbfrp3{9T#p8-~hHIdh(L&T9@qG*pKPQyrB|UqS=|Mg1y1kmx`}s#P>9g+`@N1%)k8 z{0r7xlvo-=fPaty_g)gH&{89L#a3pWKsN`9uQ6sj{>``XcR%fTK6gIj2k_6bD3Ct` z^F};g7GMT6jT71Sc5id8ST9uyOxe?E`N629lrlg*hM{%cJa+>lATUB=kfsMGIM}%HCB+7u3^V|_peOCNo2H!27kBqXOAz+T zI|l*<*AF(l&{wgs9>VQUVAIwnoQsK+a4-E}5A)e>_`DI)bS3Q0y1m@mkB`SUtpM^% CV!Ye{ literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_minutes.ic/snapshots/1CECHNKREP0F1RSTCMT0 b/tests/fixtures/qr_minutes.ic/snapshots/1CECHNKREP0F1RSTCMT0 new file mode 100644 index 0000000000000000000000000000000000000000..69fcb5a609787aa556a62cd93ca08a38165e689e GIT binary patch literal 193 zcmeZtcKtAad6%wj~C8-5xWj>zh z&@tpcmUQ2v_ReP3BK@8JDVXx5&m93NM(nJDh{ORx*@k z^)iLNW3|sWo7sO>KHPpoXLG@hmulZV0&bN|6+0-AC$M1QBNGv$x}RT8Tn}$iO~?s| l&=VKSesJ!2)dfz^0yQUhC0FivMrMZN@6XEe32@J01OOz0Ndy1@ literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_minutes.ic/snapshots/Q2YNXV9BEWYF2TKZ41VG b/tests/fixtures/qr_minutes.ic/snapshots/Q2YNXV9BEWYF2TKZ41VG new file mode 100644 index 0000000000000000000000000000000000000000..d885115ba19d6debe9062c7611c4cbdfcf15ba79 GIT binary patch literal 1040 zcmV+r1n>JvLq+hPr;0;JRZdH3V=Xc+FfKD7ARr(hARr(hARr(hARr(D0RbqrFZ}>m z1s?$RvZh4f7^gL$2_kpp7;`vayko$|aDpK%$h58ys}Q89rAAvW3+?WI@~z1>avuW{ zxKpQE+vtsWvI0n!i65s8eRfA?sgV?u0GR-s01!gRb=1oGxoC^)SP{Z1x6d_egsM1S zl8Y;#Dr&U4Ry*9M!-_Q!EJve{T}dmo2w?_;17T)U10jUj2&jsaN9uSH!feHppgc*+ zfBKS$4>m9$2+}cS$>_fqi;2?Dn}&QA2Bjh*=Lf_H`T;L~hyu1SaYX_O22@HE z8Syn9DPNoik_w1=S}d66)GEf!gx5=K>w$WCMiBJ`5XJwT<)3e(r zTT^p!RO?vWT=~o$AP6%coF(FoyooRmXsqnE7R;3|otV`0xGqjONMfPrz-KIs{HX8= z3E9!zz#`mR1WQfsErNi38uiJA!e3LQ^J{c9HQGZp8BfCTLh@W3nIfdYCd-q5zNn4) z0bo9y*r@VE=;lRqm3+82%M_E9dx=OtCi9Q(C4&EtYW@ZP%e_PtfHIsqU$tv_oUC>= zU9xu8^O3d6sdXF;R7ERa0snuF)Y*?x|Hx(3*a^-^A@03VevHIl`UCti7*G~@zOK4v z{cH-_g4UZBdR5a{_3ONW{-CchGy_Qx9xyr>AWh(x1B^;d;IAHH{t`BHz-!q-feif! zDL~}0Q7yj!lL@=`R)Igs<5ie<8Ak z8bV>}xi;$agrw{kq-u)NcaY3M%@sqQ?5@vIDR9>xP=s(l%)bo<;~j5K2E-8rxwS1Z3@&LiRUpkyq_!UJMjq>!XESP=Z%miS5vr57bqZ;8?YO46Ci& z0$dqJTS>xJn|1a|F0txF+@ zML+fv`~+4JU}<*lIP-+}98Wd{c&@eaO)3CSD@hEOCgcRa@ih;*#5Wjyzm|Rf{?xMTp#cX6SBtAn=|tV#tK5}m`7qCGv*c!C Lun0dm)0i0m-Uulg literal 0 HcmV?d00001 diff --git a/tests/fixtures/qr_minutes.ic/transactions/Q2YNXV9BEWYF2TKZ41VG b/tests/fixtures/qr_minutes.ic/transactions/Q2YNXV9BEWYF2TKZ41VG new file mode 100644 index 0000000000000000000000000000000000000000..9f11b6fe21adfed4622c83cede0a2bc5a34e343e GIT binary patch literal 234 zcmeZtcKtAad6%zM!V5E2F6_Cnd;OuAqL)Yw^p2 zC+4Im@^er3i(d8R{4?K&kB!7RXSeN6{<-_IMZSL}#gTC?y_D01S&I=jD_q_Yvwq?<+ cr>o2xguN|upGO Date: Wed, 1 Jul 2026 22:25:20 -0400 Subject: [PATCH 04/41] docs: fixture-script review fixes (overwritten/ note, zarr dep, README entry) Co-Authored-By: Claude Fable 5 --- scripts/make_streamflow_fixtures.py | 3 ++- tests/fixtures/README.md | 25 +++++++++++++++++++++++++ 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/scripts/make_streamflow_fixtures.py b/scripts/make_streamflow_fixtures.py index 22c0a21..4687a80 100644 --- a/scripts/make_streamflow_fixtures.py +++ b/scripts/make_streamflow_fixtures.py @@ -1,6 +1,6 @@ """Write tiny icechunk Qr fixture stores for ddrs integration tests. -Run under DDR's uv venv (it has icechunk + xarray): +Run under DDR's uv venv (it has icechunk + xarray + zarr): cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/make_streamflow_fixtures.py @@ -57,6 +57,7 @@ def write_store( ) # Patch the on-disk units attr if xarray normalised it (e.g. strips # " 00:00:00" from "hours since 1981-01-01 00:00:00"). + # Note: icechunk places the prior repo ref in overwritten/ on ANY write; this is normal. if time_units_on_disk is not None: z = zarr.open_group(session.store, mode="r+") z["time"].attrs["units"] = time_units_on_disk diff --git a/tests/fixtures/README.md b/tests/fixtures/README.md index f56d32c..99cfdde 100644 --- a/tests/fixtures/README.md +++ b/tests/fixtures/README.md @@ -30,3 +30,28 @@ cargo test --features fixtures --test kan_head_init_parity \ |------|----------|----------| | `kan_init_stats_ddr.csv` | `dump_kan_init_stats.py` | `kan_head_init_parity.rs` | | `kan_head_init_seed42.npz` | `dump_kan_fixture.py` | `kan_head_fixture_forward.rs`, `kan_head_fixture_backward.rs` | + +## Streamflow fixture stores + +Tiny icechunk Qr stores used by the hourly-streamflow and import-command +integration tests. Regenerate only if the DDR Q' store contract changes +(see `docs/nh-qprime-store-contract.md`): + +```bash +cd ~/projects/ddr +uv run python ~/projects/ddrs/scripts/make_streamflow_fixtures.py +``` + +Deterministic value formulas (so tests can assert exact elements): + +- `qr_daily.ic` — 4 divides × 10 days from 1981-01-01; `Qr[j, t] = (j+1)*100 + t` +- `qr_hourly.ic` — 4 divides × 240 hours from 1981-01-01T00; `Qr[j, h] = (j+1)*1000 + h` +- `qr_minutes.ic` — same shape as the first 48 hours of the hourly store but with + units `"minutes since 1981-01-01"`; exercises the sniff hard-error path (bad units + → rejection, no data read). + +| File | Producer | Consumer | +|------|----------|----------| +| `qr_daily.ic` | `make_streamflow_fixtures.py` | `hourly_streamflow.rs`, `import_cmd.rs` | +| `qr_hourly.ic` | `make_streamflow_fixtures.py` | `hourly_streamflow.rs`, `import_cmd.rs` | +| `qr_minutes.ic` | `make_streamflow_fixtures.py` | `hourly_streamflow.rs` | From 068acaba0bb1931c0d2870ddfcce8e750915827b Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Wed, 1 Jul 2026 22:27:30 -0400 Subject: [PATCH 05/41] feat(data): parse_cf_units sniffs daily vs hourly CF time axes Co-Authored-By: Claude Fable 5 --- src/data/store/icechunk.rs | 112 +++++++++++++++++++++++++++++++------ 1 file changed, 96 insertions(+), 16 deletions(-) diff --git a/src/data/store/icechunk.rs b/src/data/store/icechunk.rs index 3b5ab28..11f4ed6 100644 --- a/src/data/store/icechunk.rs +++ b/src/data/store/icechunk.rs @@ -231,12 +231,19 @@ impl StreamflowStore { } } -/// Parse `units` CF attribute of the form `"days since YYYY-MM-DD"` and -/// return the epoch as a `NaiveDate`. -pub(crate) fn parse_cf_epoch( +/// Parse the CF `units` attribute of a time coordinate and return the epoch +/// plus the native axis resolution. Supported forms (see +/// docs/nh-qprime-store-contract.md): +/// "days since YYYY-MM-DD[ HH:MM:SS]" → Daily +/// "hours since YYYY-MM-DD[ HH:MM:SS]" → Hourly +/// Anything else is a hard error naming the store and the units string — a +/// mis-scaled time axis must never be silently accepted. +pub(crate) fn parse_cf_units( attrs: &serde_json::Map, path: &Path, -) -> Result { +) -> Result<(NaiveDate, crate::data::dates::Frequency)> { + use crate::data::dates::Frequency; + let units = attrs .get("units") .and_then(|v| v.as_str()) @@ -244,21 +251,42 @@ pub(crate) fn parse_cf_epoch( path: path.to_path_buf(), message: "time array missing 'units' attribute".into(), })?; - // Expected: "days since YYYY-MM-DD" (CF convention). - let date_str = units - .strip_prefix("days since ") - .ok_or_else(|| DataError::Malformed { + let (date_str, resolution) = if let Some(rest) = units.strip_prefix("days since ") { + (rest, Frequency::Daily) + } else if let Some(rest) = units.strip_prefix("hours since ") { + (rest, Frequency::Hourly) + } else { + return Err(DataError::Malformed { path: path.to_path_buf(), - message: format!("unexpected time units format: {units:?}"), - })?; + message: format!( + "unsupported time units {units:?}: expected \"days since …\" \ + or \"hours since …\"" + ), + }); + }; // The date portion may be followed by a time-of-day component, e.g. - // "1980-01-01 00:00:00" (USGS store) vs "1980-01-01" (streamflow store). - // Take only the first token. + // "1981-01-01 00:00:00" — take only the first token. let date_part = date_str.split_whitespace().next().unwrap_or(""); - NaiveDate::parse_from_str(date_part, "%Y-%m-%d").map_err(|e| DataError::Malformed { - path: path.to_path_buf(), - message: format!("cannot parse epoch from units {units:?}: {e}"), - }) + let epoch = + NaiveDate::parse_from_str(date_part, "%Y-%m-%d").map_err(|e| DataError::Malformed { + path: path.to_path_buf(), + message: format!("cannot parse epoch from units {units:?}: {e}"), + })?; + Ok((epoch, resolution)) +} + +/// Daily-only wrapper for stores whose axis MUST be daily (USGS observations). +pub(crate) fn parse_cf_epoch( + attrs: &serde_json::Map, + path: &Path, +) -> Result { + match parse_cf_units(attrs, path)? { + (epoch, crate::data::dates::Frequency::Daily) => Ok(epoch), + (_, crate::data::dates::Frequency::Hourly) => Err(DataError::Malformed { + path: path.to_path_buf(), + message: "expected a daily time axis (\"days since …\"), got hourly".into(), + }), + } } /// Repeat a `(rho_days, N)` daily slab to `(n_hourly, N)` by replicating @@ -750,4 +778,56 @@ mod tests { first ); } + + fn attrs_with_units(u: &str) -> serde_json::Map { + let mut m = serde_json::Map::new(); + m.insert("units".into(), serde_json::Value::String(u.into())); + m + } + + #[test] + fn parse_cf_units_daily() { + let (epoch, res) = + parse_cf_units(&attrs_with_units("days since 1980-01-01"), Path::new("/t")).unwrap(); + assert_eq!(epoch, chrono::NaiveDate::from_ymd_opt(1980, 1, 1).unwrap()); + assert_eq!(res, crate::data::dates::Frequency::Daily); + } + + #[test] + fn parse_cf_units_daily_with_time_of_day() { + // daily_lstm store encodes "days since 1981-01-01 00:00:00". + let (epoch, res) = + parse_cf_units(&attrs_with_units("days since 1981-01-01 00:00:00"), Path::new("/t")) + .unwrap(); + assert_eq!(epoch, chrono::NaiveDate::from_ymd_opt(1981, 1, 1).unwrap()); + assert_eq!(res, crate::data::dates::Frequency::Daily); + } + + #[test] + fn parse_cf_units_hourly() { + // hourly_lstm store encodes "hours since 1981-01-01 00:00:00". + let (epoch, res) = + parse_cf_units(&attrs_with_units("hours since 1981-01-01 00:00:00"), Path::new("/t")) + .unwrap(); + assert_eq!(epoch, chrono::NaiveDate::from_ymd_opt(1981, 1, 1).unwrap()); + assert_eq!(res, crate::data::dates::Frequency::Hourly); + } + + #[test] + fn parse_cf_units_rejects_other_resolutions() { + let err = parse_cf_units(&attrs_with_units("minutes since 1981-01-01"), Path::new("/t")) + .unwrap_err(); + let msg = err.to_string(); + assert!(msg.contains("minutes since"), "error must name the units: {msg}"); + assert!(msg.contains("days since"), "error must name what IS supported: {msg}"); + } + + #[test] + fn parse_cf_epoch_rejects_hourly_axis() { + // The daily-only wrapper (used by the USGS observations store) must + // refuse an hourly axis rather than silently mis-scaling. + let err = parse_cf_epoch(&attrs_with_units("hours since 1980-01-01"), Path::new("/t")) + .unwrap_err(); + assert!(err.to_string().contains("daily"), "got: {err}"); + } } From a1d1e2d73f2d0267b5c8a947d4beb400e8a62b1d Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Wed, 1 Jul 2026 22:33:49 -0400 Subject: [PATCH 06/41] =?UTF-8?q?refactor(data):=20parse=5Fcf=20review=20m?= =?UTF-8?q?inors=20=E2=80=94=20Frequency=20import,=20missing-units=20test,?= =?UTF-8?q?=20doc=20fix?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- src/data/store/icechunk.rs | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/src/data/store/icechunk.rs b/src/data/store/icechunk.rs index 11f4ed6..5587604 100644 --- a/src/data/store/icechunk.rs +++ b/src/data/store/icechunk.rs @@ -31,7 +31,7 @@ use zarrs::storage::{ use ndarray::Array2; -use crate::data::dates::RhoWindow; +use crate::data::dates::{Frequency, RhoWindow}; use crate::data::error::{DataError, Result}; use crate::data::ids::{Comid, IdIndex, Staid}; @@ -241,9 +241,7 @@ impl StreamflowStore { pub(crate) fn parse_cf_units( attrs: &serde_json::Map, path: &Path, -) -> Result<(NaiveDate, crate::data::dates::Frequency)> { - use crate::data::dates::Frequency; - +) -> Result<(NaiveDate, Frequency)> { let units = attrs .get("units") .and_then(|v| v.as_str()) @@ -275,14 +273,14 @@ pub(crate) fn parse_cf_units( Ok((epoch, resolution)) } -/// Daily-only wrapper for stores whose axis MUST be daily (USGS observations). +/// Daily-only wrapper for stores whose axis MUST be daily (USGS observations, global Q' zarr). pub(crate) fn parse_cf_epoch( attrs: &serde_json::Map, path: &Path, ) -> Result { match parse_cf_units(attrs, path)? { - (epoch, crate::data::dates::Frequency::Daily) => Ok(epoch), - (_, crate::data::dates::Frequency::Hourly) => Err(DataError::Malformed { + (epoch, Frequency::Daily) => Ok(epoch), + (_, Frequency::Hourly) => Err(DataError::Malformed { path: path.to_path_buf(), message: "expected a daily time axis (\"days since …\"), got hourly".into(), }), @@ -790,7 +788,7 @@ mod tests { let (epoch, res) = parse_cf_units(&attrs_with_units("days since 1980-01-01"), Path::new("/t")).unwrap(); assert_eq!(epoch, chrono::NaiveDate::from_ymd_opt(1980, 1, 1).unwrap()); - assert_eq!(res, crate::data::dates::Frequency::Daily); + assert_eq!(res, Frequency::Daily); } #[test] @@ -800,7 +798,7 @@ mod tests { parse_cf_units(&attrs_with_units("days since 1981-01-01 00:00:00"), Path::new("/t")) .unwrap(); assert_eq!(epoch, chrono::NaiveDate::from_ymd_opt(1981, 1, 1).unwrap()); - assert_eq!(res, crate::data::dates::Frequency::Daily); + assert_eq!(res, Frequency::Daily); } #[test] @@ -810,7 +808,14 @@ mod tests { parse_cf_units(&attrs_with_units("hours since 1981-01-01 00:00:00"), Path::new("/t")) .unwrap(); assert_eq!(epoch, chrono::NaiveDate::from_ymd_opt(1981, 1, 1).unwrap()); - assert_eq!(res, crate::data::dates::Frequency::Hourly); + assert_eq!(res, Frequency::Hourly); + } + + #[test] + fn parse_cf_units_rejects_missing_units() { + let empty = serde_json::Map::new(); + let err = parse_cf_units(&empty, Path::new("/t")).unwrap_err(); + assert!(err.to_string().contains("missing"), "got: {err}"); } #[test] From fa03a3eb247c9047c92c0f47810acafb62be5b47 Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Wed, 1 Jul 2026 22:42:45 -0400 Subject: [PATCH 07/41] feat(data): hourly-native Q' reading in StreamflowStore MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolution sniffed from CF time units at open. Daily path unchanged (read_slab is the old read_window_daily body, renames only). Hourly stores slice the native axis in read_window/read_test_window and 24h-mean in read_window_daily. Also adds Debug impl to StreamflowSource (manual, to avoid propagating the bound to ZarrArray internals) — required by the fixture test. --- src/data/store/icechunk.rs | 217 +++++++++++++++++++++++++++---------- src/data/store/mod.rs | 9 ++ tests/hourly_streamflow.rs | 154 ++++++++++++++++++++++++++ 3 files changed, 320 insertions(+), 60 deletions(-) create mode 100644 tests/hourly_streamflow.rs diff --git a/src/data/store/icechunk.rs b/src/data/store/icechunk.rs index 5587604..bc8587d 100644 --- a/src/data/store/icechunk.rs +++ b/src/data/store/icechunk.rs @@ -173,12 +173,18 @@ impl ReadableStorageTraits for IcZarrStorage { /// `Qr` reader over `merit_dhbv2_UH_retrospective.ic`-style icechunk repos. /// -/// Opened once at dataset construction. Task 3 will add `read_window`. +/// Opened once at dataset construction. pub struct StreamflowStore { pub path: PathBuf, pub index: IdIndex, + /// First calendar day covered by the store (for hourly stores, the day + /// containing the first hour — open() enforces hour-0 alignment). pub time_start: NaiveDate, + /// Length of the NATIVE time axis: days for daily stores, hours for + /// hourly stores. pub n_time: usize, + /// Native axis resolution, sniffed from the CF `units` attribute. + pub resolution: Frequency, // SP-3 may consolidate to a shared runtime; keep the Arc alive so the // icechunk Store is not dropped while `qr` is in use. #[allow(dead_code)] @@ -194,12 +200,12 @@ impl StreamflowStore { // zarrs Array::open takes Arc — cast via type alias let readable: ReadableStorage = storage.clone(); - // 1. Read `time` coord: shape (n_time,), dtype int64. - // The encoding is CF-convention "days since YYYY-MM-DD" (units attr). + // 1. Read `time` coord: shape (n_time,), dtype int64. CF units are + // "days since …" (daily) or "hours since …" (hourly) — the sniff + // that decides this store's native resolution. let time_arr = ZarrArray::open(readable.clone(), "/time") .map_err(|e| ic_err(&path, e))?; - // Parse the epoch from the `units` attribute ("days since 1980-01-01"). - let time_epoch = parse_cf_epoch(time_arr.attributes(), &path)?; + let (time_epoch, resolution) = parse_cf_units(time_arr.attributes(), &path)?; let time_subset = time_arr.subset_all(); let time_i64: Vec = time_arr .retrieve_array_subset(&time_subset) @@ -211,8 +217,35 @@ impl StreamflowStore { message: "time axis is empty".into(), }); } - let time_start = time_epoch - + chrono::Duration::days(time_i64[0]); + let time_start = match resolution { + Frequency::Daily => { + time_epoch + chrono::Duration::days(time_i64[0]) + } + Frequency::Hourly => { + // Contract: hourly axes start at hour 0 of a day and are + // contiguous (docs/nh-qprime-store-contract.md). The full + // scan is cheap (~2.8 MB of i64 for 40 years of hours). + if time_i64[0] % 24 != 0 { + return Err(DataError::Malformed { + path: path.clone(), + message: format!( + "hourly store must start at hour 0 of a day; \ + first time value is {}", + time_i64[0] + ), + }); + } + if let Some(i) = + (1..time_i64.len()).find(|&i| time_i64[i] - time_i64[i - 1] != 1) + { + return Err(DataError::Malformed { + path: path.clone(), + message: format!("hourly time axis has a gap at index {i}"), + }); + } + time_epoch + chrono::Duration::days(time_i64[0] / 24) + } + }; // 2. Read `divide_id` coord; build IdIndex. let div_arr = ZarrArray::open(readable.clone(), "/divide_id") @@ -227,7 +260,7 @@ impl StreamflowStore { let qr = ZarrArray::open(readable.clone(), "/Qr") .map_err(|e| ic_err(&path, e))?; - Ok(Self { path, index, time_start, n_time, storage, qr }) + Ok(Self { path, index, time_start, n_time, resolution, storage, qr }) } } @@ -309,25 +342,32 @@ pub(crate) fn daily_to_hourly_trim(daily: &Array2, n_hourly: usize) -> Arra hourly } +/// Collapse a `(n_days * 24, N)` hourly slab to `(n_days, N)` by averaging +/// each 24-hour block. Q' is a rate (m³/s): the daily value is the day's +/// mean flow, so total daily volume is preserved. +pub(crate) fn hourly_to_daily_mean(hourly: &Array2) -> Array2 { + let (n_hours, n_div) = hourly.dim(); + debug_assert_eq!(n_hours % 24, 0, "hourly slab length {n_hours} not a multiple of 24"); + let n_days = n_hours / 24; + let mut daily = Array2::::zeros((n_days, n_div)); + for d in 0..n_days { + for j in 0..n_div { + let mut acc = 0.0f32; + for h in 0..24 { + acc += hourly[(d * 24 + h, j)]; + } + daily[(d, j)] = acc / 24.0; + } + } + daily +} + impl StreamflowStore { - /// Read `Qr` daily for `[window_start, window_start + n_days)` and - /// `comids`. Returns `(n_days, N)` f32 matrix; missing COMIDs are - /// filled with `0.001` (discharge minimum, mirrors DDR's - /// `torch.full(..., fill_value=0.001)` in `readers.py:464-468`). - /// - /// Used directly by the summed Q' baseline (which needs daily output - /// over a 15-yr window where the hourly form would be ~8.5 GB). - /// `read_window` and `read_test_window` wrap this and add the - /// daily → hourly repeat. - pub fn read_window_daily( - &self, - window_start: NaiveDate, - n_days: usize, - comids: &[Comid], - ) -> Result> { - // 1. Resolve time window to store-local day indices. - let store_start_day_i64 = (window_start - self.time_start).num_days(); - if store_start_day_i64 < 0 { + /// Store-local index of `window_start` on the NATIVE time axis + /// (day index for daily stores, hour index for hourly stores). + fn native_start_index(&self, window_start: NaiveDate) -> Result { + let days = (window_start - self.time_start).num_days(); + if days < 0 { return Err(DataError::Malformed { path: self.path.clone(), message: format!( @@ -336,19 +376,35 @@ impl StreamflowStore { ), }); } - let store_start_day = store_start_day_i64 as usize; - let end_day = store_start_day + n_days; - if end_day > self.n_time { + Ok(match self.resolution { + Frequency::Daily => days as usize, + Frequency::Hourly => days as usize * 24, + }) + } + + /// Read `(n_steps, N)` from native time-axis positions + /// `[start_step, start_step + n_steps)` for `comids`. Missing COMIDs are + /// filled with `0.001` (discharge minimum, mirrors DDR's + /// `torch.full(..., fill_value=0.001)` in `readers.py:464-468`). + fn read_slab( + &self, + start_step: usize, + n_steps: usize, + comids: &[Comid], + ) -> Result> { + let end_step = start_step + n_steps; + if end_step > self.n_time { return Err(DataError::Malformed { path: self.path.clone(), message: format!( - "window extends to store day {end_day} but n_time={}", - self.n_time + "window extends to store step {end_step} but n_time={} \ + ({:?} axis)", + self.n_time, self.resolution ), }); } - // 2. Resolve COMIDs → divide-axis positions. + // Resolve COMIDs → divide-axis positions. // `positions_of` returns positions in the order of non-missing inputs, // plus a list of indices (into `comids`) that were missing. let (positions, missing_indices) = self.index.positions_of(comids); @@ -357,17 +413,13 @@ impl StreamflowStore { let n_out = comids.len(); // Pre-fill with the discharge minimum; missing COMIDs keep this value. - let mut daily = Array2::::from_elem((n_days, n_out), 0.001); + let mut out = Array2::::from_elem((n_steps, n_out), 0.001); if positions.is_empty() { - // All COMIDs missing — return filled daily result. - return Ok(daily); + return Ok(out); } - // 3. Contiguous divide-axis read covering [min_pos, max_pos]. - // Transient memory: (max_pos - min_pos + 1) * n_days * 4 bytes. - // For 50 COMIDs spanning ~100K positions × 90 days = ~36 MB — acceptable - // for SP-2. SP-3 may revisit with gather-style reads. + // Contiguous divide-axis read covering [min_pos, max_pos]. let min_pos = *positions.iter().min().unwrap(); let max_pos = *positions.iter().max().unwrap(); let div_range_end = max_pos + 1; @@ -376,30 +428,28 @@ impl StreamflowStore { // Qr is stored as (divide_id, time). Subset: axis 0 = divide, axis 1 = time. let subset = zarrs::array::ArraySubset::new_with_ranges(&[ (min_pos as u64)..(div_range_end as u64), - (store_start_day as u64)..(end_day as u64), + (start_step as u64)..(end_step as u64), ]); let raw_f32: Vec = self .qr .retrieve_array_subset(&subset) .map_err(|e| ic_err(&self.path, e))?; - // raw_f32 is row-major: shape (div_count, n_days). - // Element at (i, t) is at index i * n_days + t. - debug_assert_eq!(raw_f32.len(), div_count * n_days); + // raw_f32 is row-major: shape (div_count, n_steps). + debug_assert_eq!(raw_f32.len(), div_count * n_steps); - // 4. Scatter into the output. Walk `comids` in order; for each + // Scatter into the output. Walk `comids` in order; for each // non-missing entry consume the next element of `positions`. let mut next_present = 0usize; for (out_col, _) in comids.iter().enumerate() { if missing_set.contains(&out_col) { - // Already pre-filled with 0.001. continue; } let div_pos = positions[next_present]; next_present += 1; let local_div = div_pos - min_pos; - for d in 0..n_days { - let raw_idx = local_div * n_days + d; - daily[(d, out_col)] = raw_f32[raw_idx]; + for t in 0..n_steps { + let raw_idx = local_div * n_steps + t; + out[(t, out_col)] = raw_f32[raw_idx]; } } @@ -409,28 +459,75 @@ impl StreamflowStore { "scatter walked past `positions` — IdIndex::positions_of invariant broken" ); - Ok(daily) + Ok(out) + } + + /// Read `Qr` daily for `[window_start, window_start + n_days)` and + /// `comids`. Returns `(n_days, N)` f32 matrix. On hourly-native stores + /// each day is the mean of its 24 hours (Q' is a rate in m³/s, so the + /// daily value is the day's average flow — keeps the summed-Q' baseline + /// meaningful on hourly stores). + pub fn read_window_daily( + &self, + window_start: NaiveDate, + n_days: usize, + comids: &[Comid], + ) -> Result> { + let start = self.native_start_index(window_start)?; + match self.resolution { + Frequency::Daily => self.read_slab(start, n_days, comids), + Frequency::Hourly => { + let hourly = self.read_slab(start, n_days * 24, comids)?; + Ok(hourly_to_daily_mean(&hourly)) + } + } } - /// Read `Qr` for `window` and `comids`. Returns `(n_hourly, N)` f32 - /// matrix; missing COMIDs (not in the store) are filled with `0.001` - /// (discharge minimum, mirrors DDR's `torch.full(..., fill_value=0.001)` - /// in `readers.py:464-468`). + /// Read `Qr` for `window` and `comids`. Returns `(n_hourly, N)` f32. + /// Daily stores upsample via repeat-24 + trailing-day trim (unchanged); + /// hourly stores slice the native axis directly — no upsampling. pub fn read_window(&self, window: &RhoWindow, comids: &[Comid]) -> Result> { - let daily = self.read_window_daily(window.window_start, window.rho_days, comids)?; - Ok(daily_to_hourly_trim(&daily, window.n_hourly())) + match self.resolution { + Frequency::Daily => { + let daily = + self.read_window_daily(window.window_start, window.rho_days, comids)?; + Ok(daily_to_hourly_trim(&daily, window.n_hourly())) + } + Frequency::Hourly => { + let start = self.native_start_index(window.window_start)?; + self.read_slab(start, window.n_hourly(), comids) + } + } } - /// Same as `read_window` but for `TestWindow` — returns `n_days * 24` - /// hours (no trailing-day trim) so chunks tile cleanly. Used by SP-5 - /// `evaluate()`. + /// Same as `read_window` but for `TestWindow` — `n_days * 24` hours + /// (no trailing-day trim) so chunks tile cleanly. pub fn read_test_window( &self, window: &crate::data::TestWindow, comids: &[Comid], ) -> Result> { - let daily = self.read_window_daily(window.window_start, window.n_days, comids)?; - Ok(daily_to_hourly_trim(&daily, window.n_hourly())) + match self.resolution { + Frequency::Daily => { + let daily = + self.read_window_daily(window.window_start, window.n_days, comids)?; + Ok(daily_to_hourly_trim(&daily, window.n_hourly())) + } + Frequency::Hourly => { + let start = self.native_start_index(window.window_start)?; + self.read_slab(start, window.n_hourly(), comids) + } + } + } + + /// `units` attribute of the `/Qr` variable, if present. Used by + /// `ddrs import` to check the m³/s contract. + pub fn qr_units(&self) -> Option { + self.qr + .attributes() + .get("units") + .and_then(|v| v.as_str()) + .map(str::to_string) } } diff --git a/src/data/store/mod.rs b/src/data/store/mod.rs index 44ac020..61fb12b 100644 --- a/src/data/store/mod.rs +++ b/src/data/store/mod.rs @@ -142,3 +142,12 @@ impl StreamflowSource { } } } + +impl std::fmt::Debug for StreamflowSource { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Icechunk(_) => write!(f, "StreamflowSource::Icechunk(..)"), + Self::GlobalZarr(_) => write!(f, "StreamflowSource::GlobalZarr(..)"), + } + } +} diff --git a/tests/hourly_streamflow.rs b/tests/hourly_streamflow.rs new file mode 100644 index 0000000..3d5727b --- /dev/null +++ b/tests/hourly_streamflow.rs @@ -0,0 +1,154 @@ +//! Fixture-backed tests for resolution-aware Q' reading. +//! +//! Fixtures are generated by `scripts/make_streamflow_fixtures.py` (run under +//! DDR's uv venv) and checked into tests/fixtures/. Deterministic values: +//! qr_daily.ic : 4 divides [101..104] x 10 days, Qr[j, t] = (j+1)*100 + t +//! qr_hourly.ic : 4 divides [101..104] x 240 hours, Qr[j, h] = (j+1)*1000 + h +//! Both axes start 1981-01-01. + +use chrono::NaiveDate; + +use ddrs::data::dates::{Frequency, RhoWindow, TimeAxis}; +use ddrs::data::ids::Comid; +use ddrs::data::store::{StreamflowSource, StreamflowStore}; +use ddrs::data::TestWindow; + +fn fixture(name: &str) -> std::path::PathBuf { + std::path::Path::new(env!("CARGO_MANIFEST_DIR")) + .join("tests/fixtures") + .join(name) +} + +fn d(y: i32, m: u32, day: u32) -> NaiveDate { + NaiveDate::from_ymd_opt(y, m, day).unwrap() +} + +const COMIDS: [Comid; 4] = [Comid(101), Comid(102), Comid(103), Comid(104)]; + +#[test] +fn hourly_store_opens_with_hourly_resolution() { + let s = StreamflowStore::open(fixture("qr_hourly.ic")).expect("open"); + assert_eq!(s.resolution, Frequency::Hourly); + assert_eq!(s.time_start, d(1981, 1, 1)); + assert_eq!(s.n_time, 240); + assert_eq!(s.index.len(), 4); +} + +#[test] +fn daily_store_opens_with_daily_resolution() { + let s = StreamflowStore::open(fixture("qr_daily.ic")).expect("open"); + assert_eq!(s.resolution, Frequency::Daily); + assert_eq!(s.time_start, d(1981, 1, 1)); + assert_eq!(s.n_time, 10); +} + +#[test] +fn minutes_axis_is_rejected_at_open() { + let err = StreamflowSource::open(fixture("qr_minutes.ic")).unwrap_err(); + assert!( + err.to_string().contains("unsupported time units"), + "got: {err}" + ); +} + +#[test] +fn hourly_read_window_slices_natively() { + let s = StreamflowStore::open(fixture("qr_hourly.ic")).expect("open"); + // Window: days [2, 6) of the axis → hours [48, 120); n_hourly = 3*24 = 72. + let w = RhoWindow { + start_day_idx: 2, + rho_days: 4, + window_start: d(1981, 1, 3), + }; + let q = s.read_window(&w, &COMIDS).expect("read_window"); + assert_eq!(q.shape(), &[72, 4]); + for h in 0..72 { + for j in 0..4 { + let expect = (j as f32 + 1.0) * 1000.0 + (48 + h) as f32; + assert_eq!(q[(h, j)], expect, "mismatch at hour {h}, divide {j}"); + } + } +} + +#[test] +fn hourly_read_window_daily_is_24h_mean() { + let s = StreamflowStore::open(fixture("qr_hourly.ic")).expect("open"); + let q = s + .read_window_daily(d(1981, 1, 3), 4, &COMIDS) + .expect("read_window_daily"); + assert_eq!(q.shape(), &[4, 4]); + // Day d of the window covers hours 48+24d .. 48+24d+24; the mean of a + // 24-term arithmetic ramp k..k+23 is k + 11.5. + for day in 0..4 { + for j in 0..4 { + let expect = (j as f32 + 1.0) * 1000.0 + (48 + 24 * day) as f32 + 11.5; + assert_eq!(q[(day, j)], expect, "mismatch at day {day}, divide {j}"); + } + } +} + +#[test] +fn hourly_read_test_window_is_contiguous() { + let s = StreamflowStore::open(fixture("qr_hourly.ic")).expect("open"); + let axis = TimeAxis::new(d(1981, 1, 1), d(1981, 1, 10)); + let w = TestWindow::new(&axis, 2, 4); // hours [48, 144), no trailing trim + let q = s.read_test_window(&w, &COMIDS).expect("read_test_window"); + assert_eq!(q.shape(), &[96, 4]); + assert_eq!(q[(0, 0)], 1000.0 + 48.0); + assert_eq!(q[(95, 3)], 4000.0 + 143.0); +} + +#[test] +fn hourly_missing_comid_gets_fill() { + let s = StreamflowStore::open(fixture("qr_hourly.ic")).expect("open"); + let w = RhoWindow { + start_day_idx: 0, + rho_days: 2, + window_start: d(1981, 1, 1), + }; + let q = s + .read_window(&w, &[Comid(101), Comid(999)]) + .expect("read_window"); + assert_eq!(q.shape(), &[24, 2]); + assert_eq!(q[(5, 0)], 1000.0 + 5.0); + assert_eq!(q[(5, 1)], 0.001, "missing COMID must fill with 0.001"); +} + +#[test] +fn hourly_out_of_range_windows_hard_error() { + let s = StreamflowStore::open(fixture("qr_hourly.ic")).expect("open"); + // Before store start. + let before = RhoWindow { + start_day_idx: 0, + rho_days: 2, + window_start: d(1980, 12, 1), + }; + let err = s.read_window(&before, &COMIDS).unwrap_err(); + assert!(err.to_string().contains("before store start"), "got: {err}"); + // Past store end (store holds 10 days). + let past = RhoWindow { + start_day_idx: 8, + rho_days: 5, + window_start: d(1981, 1, 9), + }; + assert!(s.read_window(&past, &COMIDS).is_err()); +} + +#[test] +fn daily_fixture_read_window_keeps_repeat24_semantics() { + // Pins the daily path: values repeat 24x per day with the trailing-day trim. + let s = StreamflowStore::open(fixture("qr_daily.ic")).expect("open"); + let w = RhoWindow { + start_day_idx: 2, + rho_days: 4, + window_start: d(1981, 1, 3), + }; + let q = s.read_window(&w, &COMIDS).expect("read_window"); + assert_eq!(q.shape(), &[72, 4]); + for h in 0..72 { + for j in 0..4 { + let expect = (j as f32 + 1.0) * 100.0 + (2 + h / 24) as f32; + assert_eq!(q[(h, j)], expect, "mismatch at hour {h}, divide {j}"); + } + } +} From 704c0f8f6230973b40fdd98f34f988b3391cd867 Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Wed, 1 Jul 2026 22:54:22 -0400 Subject: [PATCH 08/41] fix(data): chunk hourly daily-reads, extract+test hourly axis validation Review fixes on b1efd1e: bound the baseline-path transient to 32-day blocks, unit-test the hour-0/contiguity checks, pin qr_units, make hourly_to_daily_mean row-major friendly. Co-Authored-By: Claude Fable 5 --- src/data/store/icechunk.rs | 90 ++++++++++++++++++++++++++------------ tests/hourly_streamflow.rs | 1 + 2 files changed, 63 insertions(+), 28 deletions(-) diff --git a/src/data/store/icechunk.rs b/src/data/store/icechunk.rs index bc8587d..1477112 100644 --- a/src/data/store/icechunk.rs +++ b/src/data/store/icechunk.rs @@ -222,27 +222,7 @@ impl StreamflowStore { time_epoch + chrono::Duration::days(time_i64[0]) } Frequency::Hourly => { - // Contract: hourly axes start at hour 0 of a day and are - // contiguous (docs/nh-qprime-store-contract.md). The full - // scan is cheap (~2.8 MB of i64 for 40 years of hours). - if time_i64[0] % 24 != 0 { - return Err(DataError::Malformed { - path: path.clone(), - message: format!( - "hourly store must start at hour 0 of a day; \ - first time value is {}", - time_i64[0] - ), - }); - } - if let Some(i) = - (1..time_i64.len()).find(|&i| time_i64[i] - time_i64[i - 1] != 1) - { - return Err(DataError::Malformed { - path: path.clone(), - message: format!("hourly time axis has a gap at index {i}"), - }); - } + validate_hourly_axis(&time_i64, &path)?; time_epoch + chrono::Duration::days(time_i64[0] / 24) } }; @@ -264,6 +244,29 @@ impl StreamflowStore { } } +/// Contract checks for an hourly time axis (docs/nh-qprime-store-contract.md): +/// must start at hour 0 of a calendar day and step by exactly 1 hour. +/// The full scan is cheap (~2.8 MB of i64 for 40 years of hours). +fn validate_hourly_axis(time_i64: &[i64], path: &Path) -> Result<()> { + if time_i64[0] % 24 != 0 { + return Err(DataError::Malformed { + path: path.to_path_buf(), + message: format!( + "hourly store must start at hour 0 of a day; \ + first time value is {}", + time_i64[0] + ), + }); + } + if let Some(i) = (1..time_i64.len()).find(|&i| time_i64[i] - time_i64[i - 1] != 1) { + return Err(DataError::Malformed { + path: path.to_path_buf(), + message: format!("hourly time axis has a gap at index {i}"), + }); + } + Ok(()) +} + /// Parse the CF `units` attribute of a time coordinate and return the epoch /// plus the native axis resolution. Supported forms (see /// docs/nh-qprime-store-contract.md): @@ -351,12 +354,14 @@ pub(crate) fn hourly_to_daily_mean(hourly: &Array2) -> Array2 { let n_days = n_hours / 24; let mut daily = Array2::::zeros((n_days, n_div)); for d in 0..n_days { - for j in 0..n_div { - let mut acc = 0.0f32; - for h in 0..24 { - acc += hourly[(d * 24 + h, j)]; + for h in 0..24 { + let row = hourly.row(d * 24 + h); + for j in 0..n_div { + daily[(d, j)] += row[j]; } - daily[(d, j)] = acc / 24.0; + } + for j in 0..n_div { + daily[(d, j)] /= 24.0; } } daily @@ -477,8 +482,26 @@ impl StreamflowStore { match self.resolution { Frequency::Daily => self.read_slab(start, n_days, comids), Frequency::Hourly => { - let hourly = self.read_slab(start, n_days * 24, comids)?; - Ok(hourly_to_daily_mean(&hourly)) + // Chunk the hourly read in day blocks to bound the transient + // allocation: the summed-Q' baseline asks for ~15-year windows, + // and a single (n_days*24, N) read would be multi-GB at CONUS + // scale (plus the raw divide-span buffer inside read_slab). + const CHUNK_DAYS: usize = 32; + let n_out = comids.len(); + let mut daily = Array2::::zeros((n_days, n_out)); + let mut day = 0usize; + while day < n_days { + let chunk = CHUNK_DAYS.min(n_days - day); + let hourly = self.read_slab(start + day * 24, chunk * 24, comids)?; + let mean = hourly_to_daily_mean(&hourly); + for local_d in 0..chunk { + for j in 0..n_out { + daily[(day + local_d, j)] = mean[(local_d, j)]; + } + } + day += chunk; + } + Ok(daily) } } } @@ -767,6 +790,17 @@ mod tests { use super::*; use std::path::Path; + #[test] + fn hourly_axis_validation_rejects_misalignment_and_gaps() { + let p = Path::new("/t"); + assert!(validate_hourly_axis(&[0, 1, 2, 3], p).is_ok()); + assert!(validate_hourly_axis(&[24, 25, 26], p).is_ok()); + let err = validate_hourly_axis(&[5, 6, 7], p).unwrap_err(); + assert!(err.to_string().contains("hour 0"), "got: {err}"); + let err = validate_hourly_axis(&[0, 1, 3], p).unwrap_err(); + assert!(err.to_string().contains("gap at index 2"), "got: {err}"); + } + #[test] fn daily_to_hourly_trim_repeats_and_truncates() { use ndarray::Array2; diff --git a/tests/hourly_streamflow.rs b/tests/hourly_streamflow.rs index 3d5727b..03e54db 100644 --- a/tests/hourly_streamflow.rs +++ b/tests/hourly_streamflow.rs @@ -32,6 +32,7 @@ fn hourly_store_opens_with_hourly_resolution() { assert_eq!(s.time_start, d(1981, 1, 1)); assert_eq!(s.n_time, 240); assert_eq!(s.index.len(), 4); + assert_eq!(s.qr_units().as_deref(), Some("m^3/s")); } #[test] From 970e0a55e9a030ede7abf57ef9b795c424dc5ddf Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Wed, 1 Jul 2026 22:57:53 -0400 Subject: [PATCH 09/41] feat(data): expose Q' resolution; reject disagg head on hourly-native source Co-Authored-By: Claude Fable 5 --- src/data/dataset.rs | 41 +++++++++++++++++++++++++++++++++++++- src/data/store/mod.rs | 9 +++++++++ tests/hourly_streamflow.rs | 8 ++++++++ 3 files changed, 57 insertions(+), 1 deletion(-) diff --git a/src/data/dataset.rs b/src/data/dataset.rs index c0b927d..9592496 100644 --- a/src/data/dataset.rs +++ b/src/data/dataset.rs @@ -11,7 +11,7 @@ use ndarray::{Array1, Array2}; use crate::config::Config; use crate::data::collate::{build_flow_scale, compress, union_subgraphs}; -use crate::data::dates::{RhoWindow, TimeAxis}; +use crate::data::dates::{Frequency, RhoWindow, TimeAxis}; use crate::data::error::{DataError, Result}; use crate::data::ids::{Comid, Staid}; use crate::data::statistics::{fill_nans, AttrStats}; @@ -287,6 +287,27 @@ pub struct MeritGagesDataset { leakance_impervious_threshold: Option, } +/// Reject the disaggregation head when the streamflow store is hourly-native: +/// disaggregating an already-hourly signal is a config contradiction, and +/// after the 2026-07-01 stale-binary incident nothing in the forcing path is +/// allowed to silently degrade. +fn validate_disagg_vs_resolution( + resolution: Frequency, + has_disagg: bool, + streamflow_path: &std::path::Path, +) -> Result<()> { + if resolution == Frequency::Hourly && has_disagg { + return Err(DataError::Malformed { + path: streamflow_path.to_path_buf(), + message: "kan_head.disaggregation is set but the streamflow store is \ + hourly-native; remove the disaggregation block (an hourly \ + store needs no daily→hourly head)" + .into(), + }); + } + Ok(()) +} + impl MeritGagesDataset { /// Open all five stores and apply the training-mode filter pipeline. /// Mirrors `Merit.__init__` + `_init_training` in `geodatazoo/merit.py`. @@ -387,6 +408,13 @@ impl MeritGagesDataset { // ---------- 3. Icechunk stores ---------- let streamflow = Arc::new(StreamflowSource::open(&ds.streamflow)?); + // The smoke-train self-check line: proves which read path executed. + eprintln!("streamflow resolution: {:?}", streamflow.resolution()); + validate_disagg_vs_resolution( + streamflow.resolution(), + head_cfg.disaggregation.is_some(), + &ds.streamflow, + )?; let observations = Arc::new(ObservationsStore::open(&ds.observations)?); // Optional hourly precip store for the precip-driven disaggregation @@ -1344,4 +1372,15 @@ mod tests { assert_eq!(b1.observations.nrows(), 15, "observations sliced per window"); assert_eq!(b2.observations.nrows(), 15); } + + #[test] + fn disagg_rejected_on_hourly_native_source() { + use crate::data::dates::Frequency; + let p = std::path::Path::new("/mnt/fake/qr_hourly.ic"); + let err = validate_disagg_vs_resolution(Frequency::Hourly, true, p).unwrap_err(); + assert!(err.to_string().contains("hourly-native"), "got: {err}"); + assert!(validate_disagg_vs_resolution(Frequency::Hourly, false, p).is_ok()); + assert!(validate_disagg_vs_resolution(Frequency::Daily, true, p).is_ok()); + assert!(validate_disagg_vs_resolution(Frequency::Daily, false, p).is_ok()); + } } diff --git a/src/data/store/mod.rs b/src/data/store/mod.rs index 61fb12b..19e0d85 100644 --- a/src/data/store/mod.rs +++ b/src/data/store/mod.rs @@ -112,6 +112,15 @@ impl StreamflowSource { } } + /// Native time-axis resolution of the underlying store. The global + /// zarr v2 layout is daily by construction. + pub fn resolution(&self) -> crate::data::dates::Frequency { + match self { + Self::Icechunk(s) => s.resolution, + Self::GlobalZarr(_) => crate::data::dates::Frequency::Daily, + } + } + pub fn read_window_daily( &self, window_start: chrono::NaiveDate, diff --git a/tests/hourly_streamflow.rs b/tests/hourly_streamflow.rs index 03e54db..b4f47fd 100644 --- a/tests/hourly_streamflow.rs +++ b/tests/hourly_streamflow.rs @@ -135,6 +135,14 @@ fn hourly_out_of_range_windows_hard_error() { assert!(s.read_window(&past, &COMIDS).is_err()); } +#[test] +fn streamflow_source_reports_resolution() { + let daily = StreamflowSource::open(fixture("qr_daily.ic")).expect("open daily"); + assert_eq!(daily.resolution(), Frequency::Daily); + let hourly = StreamflowSource::open(fixture("qr_hourly.ic")).expect("open hourly"); + assert_eq!(hourly.resolution(), Frequency::Hourly); +} + #[test] fn daily_fixture_read_window_keeps_repeat24_semantics() { // Pins the daily path: values repeat 24x per day with the trailing-day trim. From d3688e583a1d1b502da37f11f9854870854bad58 Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Wed, 1 Jul 2026 23:02:37 -0400 Subject: [PATCH 10/41] refactor(data): consistent Frequency import in store/mod.rs Co-Authored-By: Claude Fable 5 --- src/data/store/mod.rs | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/data/store/mod.rs b/src/data/store/mod.rs index 19e0d85..160235d 100644 --- a/src/data/store/mod.rs +++ b/src/data/store/mod.rs @@ -29,7 +29,7 @@ pub use zarr_qprime::GlobalStreamflowStore; use ndarray::Array2; -use crate::data::dates::RhoWindow; +use crate::data::dates::{Frequency, RhoWindow}; use crate::data::error::Result; use crate::data::ids::{Comid, Staid}; @@ -114,10 +114,11 @@ impl StreamflowSource { /// Native time-axis resolution of the underlying store. The global /// zarr v2 layout is daily by construction. - pub fn resolution(&self) -> crate::data::dates::Frequency { + /// (GlobalZarr arm has no zarr-v2 fixture; covered by real-store smoke runs.) + pub fn resolution(&self) -> Frequency { match self { Self::Icechunk(s) => s.resolution, - Self::GlobalZarr(_) => crate::data::dates::Frequency::Daily, + Self::GlobalZarr(_) => Frequency::Daily, } } From d288ef79deaea26faf5ab257719a5e4fad51e47d Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Wed, 1 Jul 2026 23:03:38 -0400 Subject: [PATCH 11/41] docs: DDR Q' store contract (producer/consumer interface) --- docs/nh-qprime-store-contract.md | 66 ++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 docs/nh-qprime-store-contract.md diff --git a/docs/nh-qprime-store-contract.md b/docs/nh-qprime-store-contract.md new file mode 100644 index 0000000..0db4e12 --- /dev/null +++ b/docs/nh-qprime-store-contract.md @@ -0,0 +1,66 @@ +# DDR Q' store contract + +The interface between runoff producers (neural-hydrology LSTMs, dHBV2, …) +and ddrs routing. Any store meeting this contract can be validated and +registered with `ddrs import --name ` and then routed. + +The reference producer is +`~/projects/neuralhydrology/examples/merit_hydro/forward_merit.py` +(`--mode daily|hourly`), which runs a trained NH model over the MERIT unit +catchments and writes a conforming store. Producers that RUN neural +hydrology live in the NH repo; everything downstream of the written store +lives here. + +## Contract + +- An **icechunk repository** (`main` branch, local filesystem), root group. +- One data variable **`Qr(divide_id, time)`**, dtype **float32**, attr + `units: m^3/s`. +- `Qr` values are the **local lateral inflow per MERIT unit catchment** — + no upstream accumulation (routing does that). +- `divide_id`: int64 MERIT COMIDs. +- `time`: int64, CF-encoded as either + - `days since YYYY-MM-DD[ HH:MM:SS]` — a **daily** store, or + - `hours since YYYY-MM-DD[ HH:MM:SS]` — an **hourly** store. + The axis must be contiguous (no gaps); an hourly axis must start at + hour 0 of a calendar day. Any other units string is rejected at open. +- Values strictly positive: producers floor NaN/negative predictions to + `1e-6` (as `forward_merit.py::mm_day_to_m3s` does). +- COMIDs **absent** from the store are ddrs's concern, not the producer's: + reads fill them with `0.001` m³/s, never error. + +## How ddrs reads each resolution + +| ddrs read | daily store | hourly store | +|---|---|---| +| `read_window` (training) | repeat-24 + trailing-day trim (or disagg head) | native hourly slice | +| `read_test_window` (eval) | repeat-24, `n_days*24` | native hourly slice | +| `read_window_daily` (baseline, disagg input) | direct | mean of each 24-h block | + +`kan_head.disaggregation` is **rejected** when the streamflow source is +hourly-native — disaggregating an already-hourly signal is a config +contradiction (`src/data/dataset.rs::validate_disagg_vs_resolution`). + +## Conforming stores (2026-07-01) + +| Store (`/mnt/ssd1/data/icechunk/`) | resolution | range | divides | +|---|---|---|---| +| `daily_lstm_merit_unit_catchments.ic` | daily | 1981-01-01 → 2020-12-30 | 288,421 | +| `hourly_lstm_merit_unit_catchments.ic` | hourly | 1981-01-01 → 2020-12-31T23 | 197,088 | +| `daily_dhbv2_merit_unit_catchments.ic` | daily | 1980-01-01 → 2020-12-30 | 288,421 | +| `merit_dhbv2_UH_retrospective.ic` | daily | 1980-01-01 → 2020-12-31 | 197,088 | + +Note the hourly store starts **1981-01-01** (1980 was LSTM warmup): an +experiment window reaching into 1980 hard-errors rather than clamping. + +## Onboarding a new NH dataset + +1. In `~/projects/neuralhydrology`, write/adapt a forward script that emits + a conforming store (start from `forward_merit.py`). +2. `ddrs import --dry-run` — validates the contract + prints a + COMID-coverage report. +3. `ddrs import --name ` — registers it under + `config/sources/.yaml`. +4. `ddrs sources use && ddrs plan && ddrs run --workflow train`. + +Design history: `docs/superpowers/specs/2026-07-01-nh-qprime-import-design.md`. From 5167f5118cc9876cfe8e96bfcc4024f56fe0336a Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Wed, 1 Jul 2026 23:10:14 -0400 Subject: [PATCH 12/41] =?UTF-8?q?feat(cli):=20ddrs=20import=20=E2=80=94=20?= =?UTF-8?q?validate=20Q'=20store=20contract=20+=20register=20source=20grou?= =?UTF-8?q?p?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- src/cli/import.rs | 246 ++++++++++++++++++++++++++++++++++++++++++++ src/cli/mod.rs | 1 + src/cli/plan.rs | 2 +- src/cli/sources.rs | 21 +++- tests/import_cmd.rs | 125 ++++++++++++++++++++++ 5 files changed, 389 insertions(+), 6 deletions(-) create mode 100644 src/cli/import.rs create mode 100644 tests/import_cmd.rs diff --git a/src/cli/import.rs b/src/cli/import.rs new file mode 100644 index 0000000..a977f8e --- /dev/null +++ b/src/cli/import.rs @@ -0,0 +1,246 @@ +//! `ddrs import` — validate a Q' store against the DDR store contract and +//! register it as a named data-source group. +//! +//! One command turns a conforming store (see docs/nh-qprime-store-contract.md) +//! into a routable dataset: +//! +//! ```text +//! ddrs import /mnt/ssd1/data/icechunk/hourly_lstm_merit_unit_catchments.ic \ +//! --name hourly-lstm +//! ddrs sources use hourly-lstm && ddrs plan && ddrs run --workflow train +//! ``` +//! +//! Validation opens the store through the same `StreamflowSource::open` the +//! training loop uses, so "import succeeded" means "training will read it". +//! The coverage report is best-effort: it needs a resolvable adjacency +//! (explicit paths or a warm `.ddrs/adjacency` cache) and degrades to a +//! warning without one. + +use std::fs; +use std::path::{Path, PathBuf}; + +use crate::cli::sources; +use crate::cli::workspace::Workspace; +use crate::config::{Config, ConfigMode}; +use crate::data::dates::Frequency; +use crate::data::store::{ConusAdjacencyStore, StreamflowSource}; +use crate::error::CliError; + +pub struct ImportInput { + pub store_path: PathBuf, + /// Group name to register under `config/sources/`. `None` is only valid + /// with `dry_run`. + pub name: Option, + /// Validate + report only; write nothing. + pub dry_run: bool, + /// Overwrite an existing group of the same name. + pub force: bool, +} + +pub fn run_import( + cfg_path: Option<&Path>, + ws: &Workspace, + input: ImportInput, +) -> Result<(), CliError> { + if input.name.is_none() && !input.dry_run { + return Err(CliError::Runtime( + "pass --name to register the store, or --dry-run to \ + validate only" + .into(), + )); + } + if let Some(name) = &input.name { + // Fail on a bad name BEFORE the (possibly slow) store open. + sources::validate_name(name)?; + } + if !input.store_path.exists() { + return Err(CliError::DataSourceMissing { + path: input.store_path.clone(), + }); + } + + // ---- 1. Open & detect (same code path the training loop uses) ---- + let source = StreamflowSource::open(&input.store_path) + .map_err(|e| CliError::Runtime(format!("store failed to open: {e}")))?; + + println!("store {}", input.store_path.display()); + match &source { + StreamflowSource::Icechunk(s) => { + let (res_str, n_days) = match s.resolution { + Frequency::Daily => ("daily", s.n_time), + Frequency::Hourly => ("hourly", s.n_time / 24), + }; + let time_end = s.time_start + chrono::Duration::days(n_days as i64 - 1); + println!("format icechunk"); + println!("resolution {res_str}"); + println!( + "time {} .. {} ({} native steps)", + s.time_start, time_end, s.n_time + ); + println!("divides {}", s.index.len()); + + // ---- 2. Contract checks ---- + match s.qr_units() { + Some(u) if u == "m^3/s" => println!("Qr units m^3/s"), + Some(u) => println!( + "Qr units WARNING: {u:?} (contract expects \"m^3/s\"; \ + the solver will treat values as m³/s regardless)" + ), + None => println!( + "Qr units WARNING: no units attribute (contract expects \ + \"m^3/s\")" + ), + } + sample_read(s)?; + + // ---- 3. Coverage report (best-effort) ---- + coverage_report(cfg_path, ws, s); + } + StreamflowSource::GlobalZarr(_) => { + println!("format global zarr v2 (daily)"); + println!( + "note detailed contract validation and coverage are \ + icechunk-only; open succeeded, which exercises the same \ + reader the training loop uses" + ); + } + } + + // ---- 4. Register ---- + if input.dry_run { + println!("dry-run no group written"); + return Ok(()); + } + let name = input.name.expect("checked at entry"); + let cfg = cfg_path.ok_or_else(|| CliError::ConfigInvalid { + path: ".".into(), + source: "no ddrs.yaml found — registration copies its data_sources \ + block. Run inside a ddrs workspace or pass --config." + .into(), + })?; + let cfg_text = fs::read_to_string(cfg)?; + let block = sources::extract_block(&cfg_text, cfg)?; + let swapped = swap_streamflow_line(&block, &input.store_path)?; + let dest = sources::save_block(cfg, &name, &swapped, input.force)?; + println!("registered {}", dest.display()); + println!("activate ddrs sources use {name}"); + Ok(()) +} + +/// Read a tiny sample (first 5 divides × up to 3 days) and require finite, +/// positive values — catches unit disasters and all-NaN stores. +fn sample_read(s: &crate::data::store::StreamflowStore) -> Result<(), CliError> { + let comids: Vec<_> = s.index.ids().iter().take(5).copied().collect(); + let n_days_native = match s.resolution { + Frequency::Daily => s.n_time, + Frequency::Hourly => s.n_time / 24, + }; + let n_days = n_days_native.min(3); + let q = s + .read_window_daily(s.time_start, n_days, &comids) + .map_err(|e| CliError::Runtime(format!("sample read failed: {e}")))?; + for &v in q.iter() { + if !v.is_finite() || v <= 0.0 { + return Err(CliError::Runtime(format!( + "sample read violates the contract: value {v} (must be \ + finite and > 0; producers floor to 1e-6)" + ))); + } + } + println!( + "sample {} COMIDs × {} days: finite, positive ✓", + comids.len(), + n_days + ); + Ok(()) +} + +/// Intersect the store's divide_ids with the resolved CONUS adjacency and +/// report coverage. Best-effort: any failure (no config, unreadable +/// adjacency) prints a warning instead of failing the import. NOTE: with a +/// fabric-only config and a cold cache this triggers the managed adjacency +/// build (~10 s CONUS), same as `ddrs plan`. +fn coverage_report( + cfg_path: Option<&Path>, + ws: &Workspace, + s: &crate::data::store::StreamflowStore, +) { + let Some(cfg_path) = cfg_path else { + println!("coverage skipped (no ddrs.yaml — run inside a workspace for a report)"); + return; + }; + let resolved = Config::from_yaml_file_with_mode(cfg_path, ConfigMode::Training) + .map_err(|e| e.to_string()) + .and_then(|config| { + crate::cli::plan::resolve_adjacency(&config, cfg_path, ws) + .map_err(|e| e.to_string()) + }) + .and_then(|resolved| { + ConusAdjacencyStore::open(&resolved.conus).map_err(|e| e.to_string()) + }); + match resolved { + Ok(conus) => { + let total = conus.order.len(); + let covered = conus.order.iter().filter(|c| s.index.contains(c)).count(); + let pct = 100.0 * covered as f64 / total.max(1) as f64; + println!( + "coverage {covered}/{total} fabric COMIDs ({pct:.1}%); \ + the rest read as 0.001 m³/s fill" + ); + } + Err(e) => println!("coverage skipped ({e})"), + } +} + +/// Replace the value of the `streamflow:` key inside a `data_sources:` block, +/// preserving indentation and every other line (comments included). +fn swap_streamflow_line(block: &str, store_path: &Path) -> Result { + let mut out = String::new(); + let mut swapped = false; + for line in block.lines() { + let trimmed = line.trim_start(); + if !swapped && trimmed.starts_with("streamflow:") { + let indent = &line[..line.len() - trimmed.len()]; + out.push_str(&format!("{indent}streamflow: {}\n", store_path.display())); + swapped = true; + } else { + out.push_str(line); + out.push('\n'); + } + } + if !swapped { + return Err(CliError::Runtime( + "config's data_sources block has no `streamflow:` key".into(), + )); + } + Ok(out) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn swap_streamflow_preserves_everything_else() { + let block = "\ +data_sources: + attributes: /a.nc + # comment stays + streamflow: /old.ic + observations: /obs +"; + let out = swap_streamflow_line(block, Path::new("/new/store.ic")).unwrap(); + assert!(out.contains("streamflow: /new/store.ic")); + assert!(!out.contains("/old.ic")); + assert!(out.contains("# comment stays")); + assert!(out.contains("attributes: /a.nc")); + assert!(out.contains("observations: /obs")); + } + + #[test] + fn swap_errors_without_streamflow_key() { + let err = swap_streamflow_line("data_sources:\n gages: /g.csv\n", Path::new("/x")) + .unwrap_err(); + assert!(err.to_string().contains("streamflow")); + } +} diff --git a/src/cli/mod.rs b/src/cli/mod.rs index 59bd049..c0f3da8 100644 --- a/src/cli/mod.rs +++ b/src/cli/mod.rs @@ -3,6 +3,7 @@ pub mod error; pub mod fingerprint; pub mod gc; +pub mod import; pub mod lockfile; pub mod manifest; pub mod plan; diff --git a/src/cli/plan.rs b/src/cli/plan.rs index 44f73d9..0df20c0 100644 --- a/src/cli/plan.rs +++ b/src/cli/plan.rs @@ -293,7 +293,7 @@ pub fn plan(input: PlanInput, workspace: &Workspace) -> Result Result<(), CliError> { +pub(crate) fn validate_name(name: &str) -> Result<(), CliError> { let ok = !name.is_empty() && name .chars() @@ -93,7 +93,7 @@ fn block_range(lines: &[&str]) -> Option<(usize, usize)> { } /// Extract the `data_sources:` block from the config, verbatim. -fn extract_block(cfg_text: &str, cfg_path: &Path) -> Result { +pub(crate) fn extract_block(cfg_text: &str, cfg_path: &Path) -> Result { let lines: Vec<&str> = cfg_text.lines().collect(); let (start, end) = block_range(&lines).ok_or_else(|| CliError::ConfigInvalid { path: cfg_path.to_path_buf(), @@ -106,11 +106,22 @@ fn extract_block(cfg_text: &str, cfg_path: &Path) -> Result { /// Save the current config's `data_sources:` block as group `name`. pub fn run_save(cfg_path: &Path, name: &str, force: bool) -> Result { - validate_name(name)?; let cfg_text = fs::read_to_string(cfg_path)?; let block = extract_block(&cfg_text, cfg_path)?; - // Validate before persisting. - serde_yaml::from_str::(&block).map_err(|e| CliError::ConfigInvalid { + save_block(cfg_path, name, &block, force) +} + +/// Persist `block` (a full `data_sources:` block) as group `name`, after +/// validating it deserializes to `DataSources`. Shared by `ddrs sources save` +/// (verbatim block) and `ddrs import` (block with `streamflow:` swapped). +pub(crate) fn save_block( + cfg_path: &Path, + name: &str, + block: &str, + force: bool, +) -> Result { + validate_name(name)?; + serde_yaml::from_str::(block).map_err(|e| CliError::ConfigInvalid { path: cfg_path.to_path_buf(), source: Box::new(e), })?; diff --git a/tests/import_cmd.rs b/tests/import_cmd.rs new file mode 100644 index 0000000..7e4b92e --- /dev/null +++ b/tests/import_cmd.rs @@ -0,0 +1,125 @@ +//! `ddrs import` behavior against the checked-in fixture stores. + +use std::fs; +use std::path::{Path, PathBuf}; + +use ddrs::cli::import::{run_import, ImportInput}; +use ddrs::cli::workspace::Workspace; + +fn fixture(name: &str) -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")) + .join("tests/fixtures") + .join(name) +} + +/// Minimal parseable ddrs.yaml (mirrors src/cli/sources.rs test CFG). +const CFG: &str = "\ +mode: training +geodataset: merit +seed: 1 +np_seed: 1 +data_sources: + attributes: /dev/null/attrs.nc + conus_adjacency: /dev/null/conus.zarr + gages_adjacency: /dev/null/gages.zarr + streamflow: /dev/null/sf.ic + observations: /dev/null/obs.ic + gages: /dev/null/gages.csv +"; + +fn setup() -> (tempfile::TempDir, PathBuf, Workspace) { + let tmp = tempfile::tempdir().unwrap(); + let cfg = tmp.path().join("ddrs.yaml"); + fs::write(&cfg, CFG).unwrap(); + let ws = Workspace::with_root(tmp.path().join(".ddrs")); + (tmp, cfg, ws) +} + +#[test] +fn dry_run_validates_without_writing_a_group() { + let (_tmp, cfg, ws) = setup(); + run_import( + Some(&cfg), + &ws, + ImportInput { + store_path: fixture("qr_hourly.ic"), + name: None, + dry_run: true, + force: false, + }, + ) + .expect("dry-run import of hourly fixture"); + assert!( + !cfg.parent().unwrap().join("config/sources").exists(), + "dry-run must not create a group" + ); +} + +#[test] +fn import_registers_group_with_swapped_streamflow() { + let (_tmp, cfg, ws) = setup(); + run_import( + Some(&cfg), + &ws, + ImportInput { + store_path: fixture("qr_daily.ic"), + name: Some("test-daily".into()), + dry_run: false, + force: false, + }, + ) + .expect("import daily fixture"); + + let group = cfg.parent().unwrap().join("config/sources/test-daily.yaml"); + let text = fs::read_to_string(&group).expect("group file written"); + assert!(text.contains("qr_daily.ic"), "streamflow swapped: {text}"); + assert!( + text.contains("observations: /dev/null/obs.ic"), + "other keys carried over from ddrs.yaml: {text}" + ); + // Registering again without --force refuses; with force succeeds. + let again = ImportInput { + store_path: fixture("qr_daily.ic"), + name: Some("test-daily".into()), + dry_run: false, + force: false, + }; + assert!(run_import(Some(&cfg), &ws, again).is_err()); +} + +#[test] +fn import_rejects_nonconforming_store() { + let (_tmp, cfg, ws) = setup(); + let err = run_import( + Some(&cfg), + &ws, + ImportInput { + store_path: fixture("qr_minutes.ic"), + name: None, + dry_run: true, + force: false, + }, + ) + .unwrap_err(); + assert!( + err.to_string().contains("unsupported time units"), + "got: {err}" + ); +} + +#[test] +fn register_without_name_or_dry_run_is_an_error() { + let (_tmp, cfg, ws) = setup(); + let err = run_import( + Some(&cfg), + &ws, + ImportInput { + store_path: fixture("qr_daily.ic"), + name: None, + dry_run: false, + force: false, + }, + ) + .unwrap_err(); + assert!(err.to_string().contains("--name"), "got: {err}"); +} From 1d5a4ed41fd7ed15bc8391f358d91c293a688419 Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Thu, 2 Jul 2026 07:16:52 -0400 Subject: [PATCH 13/41] =?UTF-8?q?fix(cli):=20import=20review=20polish=20?= =?UTF-8?q?=E2=80=94=20force-path=20test,=20cheap-fail=20name=20check,=20r?= =?UTF-8?q?eport=20labels?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- src/cli/import.rs | 10 +++++----- src/cli/sources.rs | 1 + tests/import_cmd.rs | 7 +++++++ 3 files changed, 13 insertions(+), 5 deletions(-) diff --git a/src/cli/import.rs b/src/cli/import.rs index a977f8e..ade0021 100644 --- a/src/cli/import.rs +++ b/src/cli/import.rs @@ -81,13 +81,13 @@ pub fn run_import( // ---- 2. Contract checks ---- match s.qr_units() { - Some(u) if u == "m^3/s" => println!("Qr units m^3/s"), + Some(u) if u == "m^3/s" => println!("units m^3/s"), Some(u) => println!( - "Qr units WARNING: {u:?} (contract expects \"m^3/s\"; \ + "units WARNING: {u:?} (contract expects \"m^3/s\"; \ the solver will treat values as m³/s regardless)" ), None => println!( - "Qr units WARNING: no units attribute (contract expects \ + "units WARNING: no units attribute (contract expects \ \"m^3/s\")" ), } @@ -113,7 +113,7 @@ pub fn run_import( } let name = input.name.expect("checked at entry"); let cfg = cfg_path.ok_or_else(|| CliError::ConfigInvalid { - path: ".".into(), + path: std::path::PathBuf::from("ddrs.yaml"), source: "no ddrs.yaml found — registration copies its data_sources \ block. Run inside a ddrs workspace or pass --config." .into(), @@ -128,7 +128,7 @@ pub fn run_import( } /// Read a tiny sample (first 5 divides × up to 3 days) and require finite, -/// positive values — catches unit disasters and all-NaN stores. +/// positive values — catches unit disasters and all-NaN or all-zero stores. fn sample_read(s: &crate::data::store::StreamflowStore) -> Result<(), CliError> { let comids: Vec<_> = s.index.ids().iter().take(5).copied().collect(); let n_days_native = match s.resolution { diff --git a/src/cli/sources.rs b/src/cli/sources.rs index c3b4e3e..48a4ae9 100644 --- a/src/cli/sources.rs +++ b/src/cli/sources.rs @@ -106,6 +106,7 @@ pub(crate) fn extract_block(cfg_text: &str, cfg_path: &Path) -> Result Result { + validate_name(name)?; let cfg_text = fs::read_to_string(cfg_path)?; let block = extract_block(&cfg_text, cfg_path)?; save_block(cfg_path, name, &block, force) diff --git a/tests/import_cmd.rs b/tests/import_cmd.rs index 7e4b92e..6147bf4 100644 --- a/tests/import_cmd.rs +++ b/tests/import_cmd.rs @@ -85,6 +85,13 @@ fn import_registers_group_with_swapped_streamflow() { force: false, }; assert!(run_import(Some(&cfg), &ws, again).is_err()); + let force = ImportInput { + store_path: fixture("qr_daily.ic"), + name: Some("test-daily".into()), + dry_run: false, + force: true, + }; + run_import(Some(&cfg), &ws, force).expect("--force overwrites the existing group"); } #[test] From 54a0a482ed3b4f9fbb59f4a9bac0d7764ad62963 Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Thu, 2 Jul 2026 07:19:02 -0400 Subject: [PATCH 14/41] feat(cli): wire ddrs import subcommand --- src/bin/ddrs.rs | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/src/bin/ddrs.rs b/src/bin/ddrs.rs index 1dfebd6..8fb156d 100644 --- a/src/bin/ddrs.rs +++ b/src/bin/ddrs.rs @@ -95,6 +95,19 @@ enum Cmd { /// Print the manifest as JSON. #[arg(long)] json: bool, }, + /// Validate a Q' store against the DDR store contract + /// (docs/nh-qprime-store-contract.md) and register it as a data-source + /// group under config/sources/. + Import { + /// Path to the Q' store (icechunk repo or global zarr). + store: PathBuf, + /// Group name to register (omit together with --dry-run to validate only). + #[arg(long)] name: Option, + /// Validate and report only; don't write a source group. + #[arg(long)] dry_run: bool, + /// Overwrite an existing group with the same name. + #[arg(long)] force: bool, + }, /// Named data-source groups ("save files") under config/sources/. Sources { #[command(subcommand)] @@ -213,6 +226,18 @@ fn dispatch(cli: Cli) -> Result<(), CliError> { Ok(()) } Cmd::Show { run_id, json } => ddrs::cli::show::run_show(&ws, &run_id, json), + Cmd::Import { store, name, dry_run, force } => { + ddrs::cli::import::run_import( + cfg_path.as_deref(), + &ws, + ddrs::cli::import::ImportInput { + store_path: store, + name, + dry_run, + force, + }, + ) + } Cmd::Sources { cmd } => { let cfg = cfg_path.ok_or_else(|| CliError::ConfigInvalid { path: ".".into(), From d93c7baf0dc4072f3861b05b7ae9c751df5aaf17 Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Thu, 2 Jul 2026 07:25:16 -0400 Subject: [PATCH 15/41] config: daily-lstm + hourly-lstm data-source groups (via ddrs import) --- config/sources/daily-lstm.yaml | 10 ++++++++++ config/sources/hourly-lstm.yaml | 10 ++++++++++ 2 files changed, 20 insertions(+) create mode 100644 config/sources/daily-lstm.yaml create mode 100644 config/sources/hourly-lstm.yaml diff --git a/config/sources/daily-lstm.yaml b/config/sources/daily-lstm.yaml new file mode 100644 index 0000000..875fb03 --- /dev/null +++ b/config/sources/daily-lstm.yaml @@ -0,0 +1,10 @@ +data_sources: + attributes: /home/tbindas/projects/ddr/data/merit_global_attributes_v2.nc + conus_adjacency: /home/tbindas/projects/ddr/data/merit_conus_adjacency.zarr + gages_adjacency: /home/tbindas/projects/ddr/data/merit_gages_conus_adjacency.zarr + streamflow: /mnt/ssd1/data/icechunk/daily_lstm_merit_unit_catchments.ic + observations: /mnt/ssd1/data/icechunk/usgs_daily_observations + gages: /home/tbindas/projects/ddr/references/gage_info/gages_3000.csv + # Hourly AORC precip (zarr v3, CONUS) — drives the precip-conditioned + # mass-preserving disaggregation head below. + aorc_precip: /mnt/ssd1/data/aorc/merit_unit_catchments.zarr diff --git a/config/sources/hourly-lstm.yaml b/config/sources/hourly-lstm.yaml new file mode 100644 index 0000000..5e6a44b --- /dev/null +++ b/config/sources/hourly-lstm.yaml @@ -0,0 +1,10 @@ +data_sources: + attributes: /home/tbindas/projects/ddr/data/merit_global_attributes_v2.nc + conus_adjacency: /home/tbindas/projects/ddr/data/merit_conus_adjacency.zarr + gages_adjacency: /home/tbindas/projects/ddr/data/merit_gages_conus_adjacency.zarr + streamflow: /mnt/ssd1/data/icechunk/hourly_lstm_merit_unit_catchments.ic + observations: /mnt/ssd1/data/icechunk/usgs_daily_observations + gages: /home/tbindas/projects/ddr/references/gage_info/gages_3000.csv + # Hourly AORC precip (zarr v3, CONUS) — drives the precip-conditioned + # mass-preserving disaggregation head below. + aorc_precip: /mnt/ssd1/data/aorc/merit_unit_catchments.zarr From fe56e7785bcfb652fc2f50a1d7490b6e9cc208e6 Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Thu, 2 Jul 2026 07:43:40 -0400 Subject: [PATCH 16/41] fix(cli): propagate workflow failure as non-zero exit from ddrs run Found by the Task 9 disagg-guard negative test: run() wrote status=failed + exit_reason to the manifest but returned Ok, so every workflow failure exited 0. Manifest is still written before the error returns. Co-Authored-By: Claude Fable 5 --- src/cli/run.rs | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/cli/run.rs b/src/cli/run.rs index b649e55..019b284 100644 --- a/src/cli/run.rs +++ b/src/cli/run.rs @@ -124,6 +124,8 @@ pub fn run(input: RunInput) -> Result { } // 6. Finalize manifest.json. + let failed = matches!(status, RunStatus::Failed); + let failure_reason = if failed { exit_reason.clone() } else { None }; let manifest = Manifest { run_id: run_id.clone(), ddrs_version: env!("CARGO_PKG_VERSION").into(), @@ -155,6 +157,14 @@ pub fn run(input: RunInput) -> Result { max_mini_batches: input.max_mini_batches, }; manifest.write_atomic(&run_dir.join("manifest.json"))?; + // Propagate workflow failures as a non-zero exit. The manifest is always + // written first so the run directory and exit_reason are preserved. The + // `catch_unwind` in `dispatch` ensures panics also land here as Failed. + if failed { + return Err(CliError::Runtime( + failure_reason.unwrap_or_else(|| "workflow failed (no exit reason captured)".into()), + )); + } Ok(run_dir) } From 46d4e84bc8a2e46a0be9f89ba28a4f5d5043df41 Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Thu, 2 Jul 2026 07:45:42 -0400 Subject: [PATCH 17/41] docs: ddrs import + hourly-native Q' reading in CLAUDE.md Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index 56e4454..91184eb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -124,7 +124,7 @@ ddrs gc --keep 5 --keep-successful # prune .ddrs/runs/ **Data-source groups** (`src/cli/sources.rs`): named "save files" for the `data_sources:` block, stored as `config/sources/.yaml` (tracked; -`conus`, `conus-hourly`, and `global` ship in-repo). Switching datasets never +`conus`, `conus-hourly`, `global`, `daily-lstm`, and `hourly-lstm` ship in-repo). Switching datasets never requires hand-editing `ddrs.yaml`: `conus-hourly` = `conus` + `aorc_precip: @@ -151,6 +151,24 @@ exists) so `ddrs plan` sees no drift. Starting a global train from a CONUS workspace is therefore: `ddrs sources use global && ddrs plan --workflow train && ddrs run --workflow train`. +**Importing a Q' store** (`src/cli/import.rs`): any store meeting the DDR Q' +contract (`docs/nh-qprime-store-contract.md` — `Qr(divide_id, time)` f32 +m³/s, CF `days since`/`hours since` axis) registers as a source group in one +command: + +```bash +ddrs import --dry-run # validate + coverage report only +ddrs import --name # validate + register config/sources/.yaml +``` + +The icechunk reader sniffs daily vs **hourly-native** resolution from the CF +time units (`StreamflowStore.resolution`); hourly stores are sliced natively +(no repeat-24, no disagg — `kan_head.disaggregation` + hourly source is a +config error). `daily-lstm` / `hourly-lstm` groups (NH CudaLSTM / MTS-LSTM +forwards) ship in-repo; the hourly store starts **1981-01-01**, so experiment +windows must not reach into 1980. Dataset open logs +`streamflow resolution: Daily|Hourly` — check it when validating runs. + **Bootstrap source prompt** (`src/cli/plan_bootstrap.rs`): when `ddrs plan` materializes a missing `ddrs.yaml` and a previous successful run exists, it asks whether to start from that run's `config.yaml` snapshot or the bundled From 4351fc7f8dc0ffa1192120e110c9d7c207fa7819 Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Thu, 2 Jul 2026 07:52:32 -0400 Subject: [PATCH 18/41] docs(config): correct aorc_precip comment in hourly-lstm group Co-Authored-By: Claude Fable 5 --- config/sources/hourly-lstm.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/config/sources/hourly-lstm.yaml b/config/sources/hourly-lstm.yaml index 5e6a44b..7cf6b23 100644 --- a/config/sources/hourly-lstm.yaml +++ b/config/sources/hourly-lstm.yaml @@ -5,6 +5,7 @@ data_sources: streamflow: /mnt/ssd1/data/icechunk/hourly_lstm_merit_unit_catchments.ic observations: /mnt/ssd1/data/icechunk/usgs_daily_observations gages: /home/tbindas/projects/ddr/references/gage_info/gages_3000.csv - # Hourly AORC precip (zarr v3, CONUS) — drives the precip-conditioned - # mass-preserving disaggregation head below. + # Hourly AORC precip (zarr v3, CONUS). NOTE: unused with this group — the + # streamflow store is hourly-native, and kan_head.disaggregation (the only + # consumer of precip) is a config error with an hourly-native source. aorc_precip: /mnt/ssd1/data/aorc/merit_unit_catchments.zarr From cade9e4390647577ea368ea9c5b3375d12d685a9 Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Mon, 6 Jul 2026 15:29:17 -0400 Subject: [PATCH 19/41] feat(skills): add 16-skill reference library for ddrs project handoff Covers: change-control, debugging-playbook, failure-archaeology, architecture-contract, hydrology-reference, config-and-flags, build-and-env, run-and-operate, diagnostics-and-tooling, validation-and-qa, docs-and-writing, external-positioning, identifiability-campaign, proof-and-analysis-toolkit, research-frontier, research-methodology. Co-Authored-By: Claude Sonnet 4.6 --- .../ddrs-architecture-contract/SKILL.md | 428 +++++++++++++ .claude/skills/ddrs-build-and-env/SKILL.md | 439 +++++++++++++ .claude/skills/ddrs-change-control/SKILL.md | 413 +++++++++++++ .claude/skills/ddrs-config-and-flags/SKILL.md | 400 ++++++++++++ .../skills/ddrs-debugging-playbook/SKILL.md | 543 ++++++++++++++++ .../ddrs-diagnostics-and-tooling/SKILL.md | 582 ++++++++++++++++++ .claude/skills/ddrs-docs-and-writing/SKILL.md | 366 +++++++++++ .../skills/ddrs-external-positioning/SKILL.md | 386 ++++++++++++ .../skills/ddrs-failure-archaeology/SKILL.md | 475 ++++++++++++++ .../skills/ddrs-hydrology-reference/SKILL.md | 398 ++++++++++++ .../ddrs-identifiability-campaign/SKILL.md | 523 ++++++++++++++++ .../ddrs-proof-and-analysis-toolkit/SKILL.md | 569 +++++++++++++++++ .../skills/ddrs-research-frontier/SKILL.md | 388 ++++++++++++ .../skills/ddrs-research-methodology/SKILL.md | 551 +++++++++++++++++ .claude/skills/ddrs-run-and-operate/SKILL.md | 377 ++++++++++++ .../skills/ddrs-validation-and-qa/SKILL.md | 396 ++++++++++++ 16 files changed, 7234 insertions(+) create mode 100644 .claude/skills/ddrs-architecture-contract/SKILL.md create mode 100644 .claude/skills/ddrs-build-and-env/SKILL.md create mode 100644 .claude/skills/ddrs-change-control/SKILL.md create mode 100644 .claude/skills/ddrs-config-and-flags/SKILL.md create mode 100644 .claude/skills/ddrs-debugging-playbook/SKILL.md create mode 100644 .claude/skills/ddrs-diagnostics-and-tooling/SKILL.md create mode 100644 .claude/skills/ddrs-docs-and-writing/SKILL.md create mode 100644 .claude/skills/ddrs-external-positioning/SKILL.md create mode 100644 .claude/skills/ddrs-failure-archaeology/SKILL.md create mode 100644 .claude/skills/ddrs-hydrology-reference/SKILL.md create mode 100644 .claude/skills/ddrs-identifiability-campaign/SKILL.md create mode 100644 .claude/skills/ddrs-proof-and-analysis-toolkit/SKILL.md create mode 100644 .claude/skills/ddrs-research-frontier/SKILL.md create mode 100644 .claude/skills/ddrs-research-methodology/SKILL.md create mode 100644 .claude/skills/ddrs-run-and-operate/SKILL.md create mode 100644 .claude/skills/ddrs-validation-and-qa/SKILL.md diff --git a/.claude/skills/ddrs-architecture-contract/SKILL.md b/.claude/skills/ddrs-architecture-contract/SKILL.md new file mode 100644 index 0000000..2b2493a --- /dev/null +++ b/.claude/skills/ddrs-architecture-contract/SKILL.md @@ -0,0 +1,428 @@ +--- +name: ddrs-architecture-contract +description: "Use when you are about to touch src/routing/, src/sparse.rs, src/geometry.rs, src/nn/kan_head.rs, src/cuda_graph/, Cargo.toml's rskan pin, or any training/eval path; when you need to know which invariants are load-bearing and why; when a test fails and you need to triage against known weak points; when you are running an experiment and need to know the current performance baseline and which claims are proven vs open; or when setting up the binary install / CLI workflow." +--- + +# ddrs Architecture Contract + +## When NOT to use this skill + +- For CLI lifecycle / workflow orchestration details → read `docs/superpowers/specs/2026-05-30-ddrs-cli-lifecycle-design.md` +- For the BURN 0.21 autograd API recipe → `.claude/references/ddrs-burn-autograd.md` +- For data-source path details (zarr layout, icechunk sniffing) → `.claude/references/ddrs-reading-inputs.md` +- For eval output format / zeta netcdf schema → `.claude/references/ddrs-reading-outputs.md` + +--- + +## 1. What ddrs is (two sentences) + +`ddrs` is a **BURN-0.21 Rust port** of DDR, a differentiable Muskingum-Cunge routing solver originally in Python/PyTorch. The port must produce **gradient-exact** outputs against DDR's reference on the 5-reach RAPID sandbox; that guarantee is the V1 invariant and must hold after every commit. + +**BURN** = Rust deep-learning framework (analogous to PyTorch). **Muskingum-Cunge** (MC) = a linear reservoir routing scheme where each reach has coefficients c1–c4 derived from channel geometry and Manning's equation. **Gradient-exact** = `max(|ddrs_output - ddr_output|) < 1e-3 m³/s` on that sandbox. + +--- + +## 2. The seven invariants — break any of these and the port is meaningless + +| # | Invariant | File(s) | Test / guard | +|---|---|---|---| +| 1 | `examples/compare_ddr_sandbox` reports **ABSOLUTE MATCH** (max abs diff < 1e-3 m³/s) | `src/routing/`, `src/geometry.rs`, `src/sparse.rs` | `cargo run --release --example compare_ddr_sandbox` | +| 2 | **f32 throughout the routing core** — no f64 or bf16 casts inside the timestep chain | `src/routing/mmc_op.rs`, `src/sparse/mod.rs`, `src/geometry.rs` | V1 test; any precision drift breaks DDR parity at the f32 floor (~1e-7 rel diff per reach) | +| 3 | **Adjacency is topologically ordered, lower-triangular** (`rows[k] >= cols[k]`) | `src/sparse/`, `src/adjacency/build.rs` | `cargo test data_zarr_store::conus_adjacency_loads_real_merit_zarr` | +| 4 | **Do NOT replace the hand-written sparse backward** in `src/sparse/mod.rs` (`CsrSolveOp impl Backward`) | `src/sparse/mod.rs` | `cargo test --test sparse_gradcheck` | +| 5 | **Routing head is `rskan::KanLayer`** via `src/nn/kan_head.rs` — `Linear(F,H) → KanLayer(H,H)×N → Linear(H,P) → Sigmoid`, no inter-block ReLU | `src/nn/kan_head.rs` | `cargo test --test kan_head` | +| 6 | **rskan pinned to a tag** in `Cargo.toml` — bump tag, re-run KAN parity tests, validate before merging | `Cargo.toml` | `cargo test --features fixtures --test kan_head_init_repro --test kan_head_init_parity --test kan_head_fixture_forward --test kan_head_fixture_backward` | +| 7 | **leakance + `use_cuda_graphs: true` is a config error** — config load rejects this combination | `src/routing/leakance.rs`, `src/config.rs` | Config validation at `ddrs plan` / `ddrs run` | + +### Why invariant 4 matters (O(nnz) vs O(n²)) + +BURN's default autograd records one node per tensor operation. If you replace `CsrSolveOp`'s hand-written backward with plain tensor unrolling, the tape grows O(n²) per timestep (n = 346,321 CONUS reaches). The custom backward keeps it O(nnz) = O(338,814 edges). Same logic applies to `TimestepOp` in `src/routing/mmc_op.rs` — one node per timestep, not ~33. + +### Why invariant 3 matters + +Forward substitution on `A = I − c1·N` requires N to be strictly lower-triangular (every reach appears after all its upstream neighbors in the sorted order). If any `rows[k] < cols[k]` entry exists, the solver silently produces wrong answers with no error. + +--- + +## 3. Source tree in one screen + +``` +src/ +├── routing/ +│ ├── mmc.rs MuskingumCunge: setup_inputs, forward, route_timestep +│ ├── mmc_op.rs TimestepOp — single Backward per timestep; saves 23 intermediates +│ ├── leakance.rs GW–SW loss term; TimestepLeakanceOp: Backward (experimental) +│ └── utils.rs denormalize, hotstart, dense helpers +├── sparse/ +│ ├── mod.rs CsrPattern (Arc-shared), CsrSolveOp + hand-written Backward +│ ├── cusparse.rs cuSPARSE SpMV + SpSV FFI wrappers (SP-9) +│ └── dispatch.rs CPU forward-sub vs cuSPARSE SpSV selector +├── cuda_graph/ SP-10 CUDA Graph capture/replay (forward-only; backward not yet captured) +│ ├── capture.rs +│ ├── geometry_kernel.rs fused #[cube] kernels K1/K2/K3 +│ └── scratch.rs +├── geometry.rs Trapezoidal channel geometry (Leopold & Maddock) +├── config.rs YAML config, parameter ranges, log-space flags, SparseSolver enum +├── nn/kan_head.rs KAN head via rskan — matches DDR's kan.py exactly +├── adjacency/build.rs Managed adjacency builder (topological_sort matches petgraph DFS) +├── data/ Live zarr/netcdf/icechunk readers — no export step +│ ├── ids.rs Comid(i64), Staid(String) newtypes; IdIndex +│ ├── dates.rs TimeAxis + rho-window sampler +│ └── store/ zarr.rs, zarr_obs.rs, zarr_qprime.rs, obs_writer.rs +├── training/ +│ ├── loss.rs L1 (default) or nnse-kge; config-selectable +│ ├── forward.rs LeakanceOverride seam (eval path only) +│ ├── bootstrap.rs Checkpoint resume: weights + optim + RNG state +│ └── probe.rs lift_leaf, probe_forward, GradAccum (gradient probe instruments) +└── bin/ + ├── ddrs.rs Primary CLI: plan / run / show / status / gc / sources / import + ├── probe_zeta_gradient.rs gradient probe + synthetic teacher (--mode grad|perturb|teacher|floor|state-cache) + ├── train.rs Legacy (deprecated, removed in 0.4) + └── eval.rs Legacy; still used for --zeta-output on existing checkpoints +``` + +--- + +## 4. Per-timestep dataflow (the MC routing step) + +Everything below runs inside `forward_chain_inner` in `src/routing/mmc_op.rs` at the **inner-backend primitive level** — no autograd nodes are created inside this function. One `TimestepOp` node wraps the entire chain. + +``` +inputs: (n, q_spatial, p_spatial, q_t, q_prime_t) +fixed: (length, slope, x_storage, dt=3600 s) + +K1 — geometry + Muskingum coefficients (one fused #[cube] kernel on CUDA): + depth = ((Q·n·(q+1)) / (p·√slope))^(3/(3q+5)) + top_width = p · depth^q + side_slope = clamp(top_width·q / (2·depth), 0.5, 50) + bottom_width = clamp(top_width − 2·side_slope·depth, bw_lb) + hyd_radius = ((top_width+bottom_width)·depth/2) / (bottom_width + 2·depth·√(ss²+1)) + velocity = clamp((1/n)·R^(2/3)·√slope, v_lb, 15) + celerity = velocity · 5/3 + k_musk = length / celerity + denom = 2·k·(1−x) + dt + c1..c4 = Muskingum coefficients + +SpMV: i_t = N · q_t (cuSPARSE SpMV on GPU; scatter on CPU) + +K2 — RHS assembly: + b_rhs = c2·i_t + c3·q_t + c4·q_prime_t + +[optional leakance, when params.use_leakance: true] + area_z = (p · depth)^q_eps · length + zeta = leakance_factor · area_z · K_D · (depth − d_gw) + b_rhs = b_rhs − zeta + +A-values: a_values = assemble_primitive(c1) [CSR values of A = I − c1·N] + +SpSV: x_sol = triangular_csr_solve(a_values, b_rhs) [lower-triangular] + +K3 — clamp: + q_next = clamp_min(x_sol, discharge_lb) +``` + +On the CUDA path (SP-10), K1+K2+K3 are fused `#[cube]` kernels, and the captured per-step sequence is **K1 → SpMV → K2 → assemble → SpSV → K3** — six kernel launches replayed as one `cuGraphLaunch`. + +**Cold start (t = 0):** solves `(I − N)·Q_0 = q'_0`. On a linear chain this reduces to `Q_0[i] = Σ_{j ≤ i} q'_0[j]` (cumulative sum). + +--- + +## 5. KAN head architecture + +``` +Linear(F, H) + → KanLayer(H, H) × num_hidden_layers [ALL layers receive the SAME init seed — DDR kan.py :24-34 quirk] + → Linear(H, P) + → Sigmoid +→ output in [0, 1] (denormalized to physical units in setup_inputs via config.rs bounds) +``` + +- **F** = number of catchment attributes, **H** = hidden size, **P** = number of learnable routing parameters +- **No inter-block ReLU** — DDR's `kan.py` has none; adding one breaks parity +- `rskan` version as of 2026-07-05: **v0.1.3** (verify with `grep rskan Cargo.toml`) +- All `num_hidden_layers` KanLayers use the **same seed** (a DDR quirk preserved intentionally for parity — see `src/nn/kan_head.rs`) + +--- + +## 6. Operational traps and known weak points + +### STALE-BINARY TRAP (high severity — has caused silent wrong results) + +`cargo build` and `cargo run` do NOT update `~/.cargo/bin/ddrs`. If you type `ddrs run` after editing `src/`, you silently execute the old binary. The manifest's `git.sha` is stamped from `.git` at runtime, not from the binary, so the run log looks current. + +**After any `src/` change, do ONE of:** +```bash +cargo install --path . # canonical refresh +# or, faster if target/release is warm: +cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs +# or bypass the installed copy entirely: +cargo run --release --bin ddrs -- run --workflow … +``` + +**Self-check:** current checkpoints are **directories** (`.ddrs/runs//checkpoints/epoch_E_mb_M/head.mpk`). A stale pre-checkpoint-resume binary writes flat files (`epoch_E_mb_M.mpk`). Flat files = stale binary. + +This trap caused the 2026-07-01 leakance×hourly 2×2 first run: the hourly cell silently ran flat-repeat-24 because the installed binary predated the disaggregation feature. + +### CUDA Graphs mask NaN (high severity) + +`use_cuda_graphs: true` can return stale finite loss when the forward produces NaN. Always validate new forward-path changes with `use_cuda_graphs: false` before benchmarking with graphs on. + +`leakance + use_cuda_graphs: true` is rejected at config load time — this combination is a hard error, not a silent wrong result. + +### Checkpoint resume drifts slightly + +Checkpoints store weights/moments in **f16** (`CompactRecorder = HalfPrecisionSettings`). A resumed trajectory diverges slowly from an uninterrupted one. Exact state (epoch, mini-batch cursor, RNG permutation) is preserved, but weight precision is not. See `docs/2026-06-07-checkpoint-resume-handoff.md` follow-up #1. + +### Fixture regeneration caveat (as of 2026-07-05) + +The DDR reference state used to validate V1 lives only in the desktop's `~/projects/ddr` working tree (unpushed `geometry/trapezoidal.py` work). A fixture regenerated from a clean DDR clone diverges ~1% at every ddrs commit — that is a wrong reference, not a port bug. See `.claude/references/ddrs-comparing-to-ddr.md` §Regenerating fixtures. + +### Worktree binary path + +Fresh worktrees lack the gitignored `output/` and fixture directories. Relative `target/release` resolves to the main tree's stale binary in some shells. Always use absolute paths or `cargo run` in worktrees. See `.claude/memory/ddrs-worktree-gotchas.md`. + +--- + +## 7. Performance numbers (as of 2026-07-05) + +### CONUS network + +| Metric | Value | +|---|---| +| Reaches (CONUS MERIT) | 346,321 | +| Edges | 338,814 | +| Gauge training set | ~2,365 (CONUS) | + +### Summed-Q' baseline (no routing, no learned params) + +This is the sanity floor: per-gauge sum of upstream divide Q' over the eval window. + +| Metric | Value | +|---|---| +| Median NSE | 0.689 | +| Median KGE | 0.723 | + +**If a trained run does not beat NSE 0.689, routing is not earning its keep. Check training loss curves and KAN gradient stats first.** + +### Best trained result (as of 2026-07-05) + +Precip-driven disaggregation + L1 loss, 2,365 CONUS gauges, eval 2026-06-23: + +| Metric | Value | +|---|---| +| Median NSE | 0.715 (+0.037 vs baseline) | +| Median KGE | 0.711 (−0.012 vs baseline) | + +**KGE does NOT beat the summed-Q' baseline in any config as of 2026-07-05.** NSE does (+0.037 with precip disagg). The NSE gain is real; the KGE regression traces to over-attenuation of flood peaks (the L1 / NSE gradient rewards the MC solver for attenuating, reducing `α = σ_sim/σ_obs` below 1). + +The `nnse-kge` loss mode (`experiment.loss.kind: nnse-kge`) exists to restore the KGE gradient, but no validated CONUS result with this mode is available as of 2026-07-05. + +--- + +## 8. Leakance — experimental GW–SW water-loss term + +### What it is + +A losing-stream correction subtracted from the routing RHS `b` at each timestep: + +``` +zeta = leakance_factor · area_z · K_D · (depth − d_gw) +area_z = (p · depth)^q_eps · length (plan-view wetted area, m²) +b ← b − zeta positive zeta = losing reach +``` + +Implementation: `src/routing/leakance.rs`. Gradient is analytical via `TimestepLeakanceOp: Backward`. + +### How to enable (three required config changes) + +```yaml +params: + use_leakance: true # activates term; forces use_cuda_graphs: false + parameter_ranges: + K_D: [1.0e-8, 1.0e-5] # log-space; hydraulic exchange rate, 1/s + d_gw: [-2.0, 2.0] # groundwater depth offset, m + leakance_factor: [0.0, 1.0] # dimensionless scale + +kan_head: + learnable_parameters: [n, q_spatial, x_storage, K_D, d_gw, leakance_factor] +``` + +Note: original K_D range was `[1e-8, 1e-6]`. The recoverability experiment (2026-07-04) widened to `[1e-8, 1e-5]` to achieve 58/96 expressible sites (vs 23/96 at the original ceiling). Use `[1e-8, 1e-5]` for any future leakance work. + +### Gradient-exactness guard (run after any change to leakance.rs) + +```bash +cargo test --test leakance_gradcheck # analytical ≈ finite-difference (8/8) +cargo test --test leakance_off_parity # byte-identical to no-leakance when off (3/3) +cargo test --test zeta_accum # eval zeta == what was subtracted from b (6/6) +cargo run --release --example compare_ddr_sandbox # must still report ABSOLUTE MATCH +``` + +### Leakance status summary (as of 2026-07-05) + +**2×2 verdict (leakance × forcing, 2026-07-01): GO-marginal.** +- Leakance + hourly: ΔNSE +0.0005, ΔKGE +0.0018 on the losing-stream subset (55.5% of gauges improve) +- Leakance + daily: ΔNSE −0.0017, ΔKGE −0.0009 (35.6% improve — hurts) +- Zeta gate: |zeta| > 0.01 m³/s on 10.4% of 64,892 eval reaches (bar: ≥10%) + +**Low-zeta diagnosis (2026-07-02):** + +| Hypothesis | Verdict | Key evidence | +|---|---|---| +| H1 — K_D box clips flux | REFUTED | 71.5% of reaches CAN exceed 0.01 m³/s inside the current box; median utilization 3.4% | +| H2 — driving-head starvation | SUPPORTED | median head `(depth − d_gw)` = 0.02 m; 47% of reaches gaining at eval-window mean | +| H3 — KAN variance collapse | REFUTED | d_gw–meanP Spearman +0.71; K_D–aridity +0.61 — strong learned structure | +| H4 — gauge bias / gradient starvation | SUPPORTED | zeta–uparea ρ +0.76; gauged median |zeta| 6.7e-3 vs ungauged 5.9e-4; dry/wet ratio 0.40 (inverted from physics) | +| H5 — equifinality (n absorbs loss) | SUPPORTED (daily only) | daily Δn = +0.012 (0.59 IQR, ~20%); hourly Δn nil (0.05 IQR) | +| H6 — wrong yardstick | REFUTED | fractional loss agrees: 8.4% lose >1% of local flow | +| H7 — d_gw model-form error | REFUTED | 0.0% of d_gw at bounds, incl. dry tercile | + +**Implication of diagnosis: K_D widening alone is NOT recommended.** The binding constraint is the training signal (H2 + H4), not the parameter box. The pre-registered Phase-3 gate FAILED; the widened-K_D retrain was not run. This supersedes the "widen K_D — top follow-up" recommendation from the 2026-07-01 findings. + +**Gradient probe (2026-07-03, worktree `origin/worktree-zeta-sensitivity`):** + +| Hypothesis | Bar | Measured | Verdict | +|---|---|---|---| +| P1 — starvation (gradient dead off-gauge) | gauged/ungauged |g| ≥ 10× | 1.5× (trained), 2.9× (cold) | REFUTED | +| P2 — rejection (gradient pushes zeta down) | >67% dry-tercile push-down | 52.5% (≈ neutral) | REFUTED | +| P3 — detectability (real-magnitude loss visible at gauge) | ≥10% of Ref δ=0.01 probes detectable | 4.2% (4/96); delta is 53× smaller than median 5% discharge-uncertainty band | NO-GO | + +**Synthetic recoverability positive control (2026-07-04, worktree):** + +The positive control FAILED: median recovery ratio = 0.009 (bar: ≥0.5). Root cause: the windowed training objective has a ~130× hotstart-transient noise floor. Continuous residual with teacher weights on teacher obs = 0.0076 mean L1; step-0 windowed training loss = 1.017. The planted signal (0.8% of training loss) is invisible. After 5 epochs, Adam actively degrades the model (continuous residual grew from 0.0076 to 0.4431 — 58× worse than not training). + +**Leakance identifiability is NOT proven. The positive control must pass (Phase B objective: windowed training loss ≤ 0.25 mean L1, i.e. ≤10% of a converged run's loss) before any identifiability claim can be made.** + +--- + +## 9. Phase B objective and current state (as of 2026-07-05) + +**Phase B goal:** state-cache hotstart — inject continuous-run discharge state at each training window boundary to eliminate the hotstart-transient noise floor. + +**Target:** windowed training loss ≤ 0.25 mean L1 (≤10% of a converged run's 1.017 step-0 loss). + +**Status:** NOT YET MET as of 2026-07-05. The state-cache infrastructure (`experiment.state_cache`, `src/data/store/obs_writer.rs`, `src/training/forward.rs` injection seam, `--mode state-cache` in probe binary) is implemented in `origin/worktree-zeta-sensitivity` but the floor validation target has not been hit. + +**Until Phase B passes, do not claim leakance is learnable from gauge-only supervision.** + +--- + +## 10. CLI quick-reference + +### First-time setup + +```bash +cargo install --path . # installs ddrs to ~/.cargo/bin/ +ddrs plan # GPU probe + smoke test + writes ddrs.yaml (opens $EDITOR) +ddrs run --workflow train-and-test # train + eval + write manifest +``` + +### After any src/ change + +```bash +cargo install --path . # or: cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs +``` + +### Key commands + +```bash +ddrs sources use conus-hourly # switch to hourly AORC precip source group +ddrs sources list # show active group (* = match) +ddrs show # inspect run manifest +ddrs status # workspace summary + disk usage +ddrs gc --keep 5 --keep-successful # prune old runs + +# Hourly forcing requires BOTH: +# 1. ddrs sources use conus-hourly (adds aorc_precip path) +# 2. kan_head.disaggregation.use_precip: true in ddrs.yaml +# Without aorc_precip, config with use_precip: true is a hard error. + +# Resume from checkpoint: +# Set experiment.checkpoint: .ddrs/runs//checkpoints/epoch_E_mb_M in ddrs.yaml +# Then raise experiment.epochs above E. +``` + +### Regression tests (run after touching core routing) + +```bash +cargo run --release --example compare_ddr_sandbox # V1: must print ABSOLUTE MATCH +DDRS_FORCE_GRAPHS=1 cargo run --release --example compare_ddr_sandbox # V9: CUDA graphs bit-match +cargo test --test mmc # hotstart, coefficients, forward, autodiff +cargo test --test sparse_gradcheck # CsrSolveOp backward +cargo test --test sp8_gradcheck # fused TimestepOp backward +cargo test --test leakance_gradcheck # leakance backward (8/8) +cargo test --test leakance_off_parity # byte-identical to no-leakance (3/3) +cargo test --test zeta_accum # zeta diagnostic identity (6/6) +``` + +--- + +## 11. Workspace layout + +| Path | Purpose | +|---|---| +| `ddrs.yaml` | Workflow + experiment config (gitignored) | +| `.ddrs/system.json` | GPU/driver/smoke-test record | +| `.ddrs/sources.lock` | Fingerprints of data_sources paths | +| `.ddrs/adjacency//` | Cached CONUS + gauges adjacency zarr stores (content-addressed) | +| `.ddrs/baselines//` | Cached summed-Q' baseline (blake3 of data sources + time window) | +| `.ddrs/runs//manifest.json` | Per-run manifest (config + sources + git SHA + outputs) | +| `.ddrs/runs//config.yaml` | Snapshot of the config that produced this run | +| `.ddrs/runs//run.log` | Timestamped stdout+stderr (fd-level tee) | +| `.ddrs/runs//checkpoints/epoch_E_mb_M/` | Checkpoint directory: `head.mpk`, `optim.mpk`, `state.json` | +| `.ddrs/runs//kan_parameters.nc` | Eval-window per-reach zeta/zeta_net/depth_mean/area_z_mean/q_mean | + +--- + +## 12. Data sources summary + +| Source | Type | Path (as of 2026-07-05) | +|---|---|---| +| MERIT adjacency | managed zarr (built from fabric) | `.ddrs/adjacency//` | +| Streamflow Q' (CONUS) | icechunk | `/mnt/ssd1/data/icechunk/merit_dhbv2_UH_retrospective.ic` | +| USGS observations (CONUS) | icechunk | `/mnt/ssd1/data/icechunk/usgs_daily_observations` | +| AORC precip (hourly) | zarr-v3, catchment-major, mm/hr | `/mnt/ssd1/data/aorc/merit_unit_catchments.zarr` | +| Global streamflow Q' | zarr-v2 multi-zone (60 zones) | `/gpfs/hjj5218/data/dmc_forcing/streamflow/zarr/8km/merit_global_v2.7` | +| Global observations | zarr-v2, one array per `Provider__GageId` | `/gpfs/hjj5218/data/dmc_forcing/observation/dMC_global_v3.1` | + +Global Q' units: m³/s (confirmed empirically — no units attribute on the zarr). Time axis: CF `days since 1980-01-01`. ~42k fabric reaches lack predictions → 0.001 fill at read. + +Hourly AORC precip: zarr-v3, catchment-major (COMID-first), mm/hr, starts 1980-01-01 UTC. Experiment windows using hourly forcing must not reach into 1980 (hourly-lstm store starts 1981-01-01). + +--- + +## 13. Branches + +| Branch | Description | +|---|---| +| `master` | Main integration branch | +| `unit_catchments` | Current working branch (as of 2026-07-05) | +| `origin/worktree-zeta-sensitivity` | Most advanced — Phase B state-cache hotstart, gradient probe, recoverability control, unit-catchment attribute wiring | + +--- + +## Provenance and maintenance + +Re-verify commands (copy-pasteable, all from project root): + +```bash +# V1 invariant +cargo run --release --example compare_ddr_sandbox + +# Leakance gradient-exactness suite +cargo test --test leakance_gradcheck && cargo test --test leakance_off_parity && cargo test --test zeta_accum + +# KAN head parity +cargo test --features fixtures --test kan_head_init_repro --test kan_head_init_parity --test kan_head_fixture_forward --test kan_head_fixture_backward + +# Sparse backward +cargo test --test sparse_gradcheck && cargo test --test sp8_gradcheck + +# Check rskan version +grep rskan Cargo.toml + +# Check current binary is fresh (flat files = stale) +ls ~/.cargo/bin/ddrs -la && ls .ddrs/runs/ 2>/dev/null | tail -3 +``` + +Ground-truth sources read to produce this skill: `CLAUDE.md`, `.claude/ARCHITECTURE.md`, `.claude/references/ddrs-burn-autograd.md`, `.claude/references/ddrs-architecture.md`, `docs/2026-07-02-leakance-diagnosis-findings.md`, `origin/worktree-zeta-sensitivity:docs/2026-07-03-zeta-gradient-probe-findings.md`, `origin/worktree-zeta-sensitivity:docs/2026-07-04-synthetic-recoverability-findings.md`. Volatile facts dated 2026-07-05. Re-read those files when key numbers or experiment verdicts change. diff --git a/.claude/skills/ddrs-build-and-env/SKILL.md b/.claude/skills/ddrs-build-and-env/SKILL.md new file mode 100644 index 0000000..2fad55a --- /dev/null +++ b/.claude/skills/ddrs-build-and-env/SKILL.md @@ -0,0 +1,439 @@ +--- +name: ddrs-build-and-env +description: Use when setting up a fresh ddrs checkout, diagnosing build failures, hitting the forked-dependency trap, missing fixture errors, static netcdf/HDF5 cmake issues, stale-binary symptoms, or CUDA graphs masking NaN. Also use when cargo build succeeds but runtime behaves as an old version, or when fixture regeneration is needed after DDR solver changes. +--- + +# ddrs Build and Environment Runbook + +## Glossary (read once; terms used throughout) + +| Term | Meaning | +|---|---| +| **BURN** | Rust deep-learning framework (like PyTorch but for Rust). Version 0.21 in ddrs. | +| **DDR** | The Python/PyTorch reference implementation at `~/projects/ddr`. ddrs is its Rust port. | +| **V1 gate** | The regression test that must always pass: `compare_ddr_sandbox` reports ABSOLUTE MATCH (max abs diff < 1e-3 m³/s). | +| **`[patch.crates-io]`** | Cargo mechanism to globally replace a dependency's source. ddrs uses this to swap crates.io `cubecl`/`burn` for forked GitHub branches. | +| **KAN head** | Kolmogorov-Arnold Network routing head (`rskan::KanLayer`). Replaces MLP. Must stay at tag v0.1.3. | +| **f32 invariant** | All routing-core tensors stay float32. No f64, bf16, or mixed precision. | +| **uv** | Python package manager (like pip + venv). DDR's venv is managed by uv; needed for fixture regeneration only. | +| **icechunk** | Transactional Zarr-over-filesystem store for streamflow + observations data. | +| **cuSPARSE** | NVIDIA sparse linear algebra library. Used for the GPU triangular solve in `src/sparse.rs`. | + +--- + +## When NOT to use this skill + +| If you need... | Use instead | +|---|---| +| MC routing algorithm math | `.claude/references/ddrs-algorithm.md` | +| Autograd / sparse backward internals | `.claude/references/ddrs-burn-autograd.md` | +| DDR parity / V1 failure debugging | `.claude/references/ddrs-comparing-to-ddr.md` | +| Training a run from scratch | `CLAUDE.md` §"ddrs CLI" or README §"Getting started" | +| Leakance experiment status | `docs/2026-07-01-leakance-hourly-findings.md` | +| Architecture diagram | `.claude/ARCHITECTURE.md` | + +--- + +## 1. Prerequisites Checklist + +Before `cargo build` can succeed, verify every item below. + +### 1a. Rust toolchain + +```bash +rustc --version # must be >= 1.80; tested on 1.94.0 as of 2026-07-05 +cargo --version +``` + +Install or update via rustup: + +```bash +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +rustup update stable +``` + +### 1b. cmake (required for static netcdf/HDF5) + +`Cargo.toml` declares `netcdf = { version = "0.12", features = ["static"] }`. The `static` feature compiles bundled HDF5 and netcdf-c from source during `cargo build`. This avoids depending on system-installed dev packages (HPC hosts often have only Intel-MPI-flavored shared libs that would break `ddrs` outside `module load`). + +**Consequence:** cmake must be available on your PATH before building. + +```bash +cmake --version # must exist; any reasonably recent version works +``` + +If missing: +```bash +# Debian/Ubuntu +sudo apt-get install cmake +# Arch +sudo pacman -S cmake +# HPC (check module system) +module load cmake +``` + +### 1c. CUDA Toolkit + +Required only for the GPU path (`sparse_solver: cuda`, `use_cuda_graphs: true`). The CPU/NdArray backend path builds and runs without CUDA. + +```bash +nvcc --version # want CUDA 12+ (13.2 validated as of 2026-07-05) +nvidia-smi # driver must support the CUDA version +``` + +Validated configurations: +- RTX 4080 SUPER, driver 610.43.02, CUDA 13.2 (desktop, as of 2026-07-05) +- 8× A100, driver 575.57.08, CUDA 12, sm_80 (HPC) + +For CPU-only machines: skip CUDA setup. Override the default GPU config at runtime (see §5 "CPU-only override"). + +### 1d. git (for fork resolution) + +The forked cubecl and burn are fetched via HTTPS git by Cargo automatically. No local clones of cubecl or burn are needed — `git clone ddrs && cargo build` is sufficient. + +```bash +git --version # any recent version +``` + +--- + +## 2. Build Steps + +```bash +# Clone ddrs +git clone git@github.com:taddyb/ddrs ~/projects/ddrs +cd ~/projects/ddrs + +# Release build (LTO=thin; matches what ddrs CLI uses) +cargo build --release + +# Sanity check: V1 gate +mkdir -p output +cargo run --release --example compare_ddr_sandbox +# Expected last line: "verdict: ABSOLUTE MATCH (max abs < 1e-3 m³/s)" +``` + +The first build fetches: +- `github.com/taddyb/cubecl` branch `ddrs-release` (all cubecl-* crates) +- `github.com/taddyb/burn` branch `ddrs-sp7-primitive-ctor` (all burn-* crates) +- `github.com/taddyb/rskan` tag `v0.1.3` + +This takes several minutes on first run; subsequent builds use the Cargo registry cache. + +--- + +## 3. The Forked-Dependency Trap + +### What is patched and why + +`Cargo.toml` contains a `[patch.crates-io]` block that replaces the published crates.io versions of cubecl and burn with fork branches on github.com/taddyb: + +``` +[patch.crates-io] +cubecl = { git = "https://github.com/taddyb/cubecl.git", branch = "ddrs-release" } +cubecl-cuda = { git = "..." } # ... and 8 more cubecl-* crates +burn-cubecl = { git = "https://github.com/taddyb/burn.git", branch = "ddrs-sp7-primitive-ctor" } +burn-autodiff = { git = "..." } # ... and 12 more burn-* crates +``` + +The patches add exactly two `pub` accessors needed by ddrs's cuSPARSE GPU solve (SP-7): + +| Crate | Added accessor | +|---|---| +| `cubecl-cuda` 0.10 | `pub fn CudaServer::stream() -> CUstream` | +| `burn-cubecl` 0.21 | `pub fn CubeTensor::from_handle(...) -> Self` | + +These were `pub(crate)` in the upstream releases. The plan is to upstream them as SP-8 and remove `[patch.crates-io]` once merged. + +### Rules when working with forks + +1. **All burn-* crates must come from the same fork branch.** If Cargo resolves any burn-* crate from crates.io while another resolves from the fork, you get duplicate `Device` trait objects and cryptic link errors. The `[patch.crates-io]` block covers all 13 burn sub-crates; do not add a direct dependency on a crates.io burn sub-crate that would escape the patch. + +2. **Same rule for cubecl-***: all 10 crates in the monorepo must come from `ddrs-release`. + +3. **To iterate on the fork locally**: push your changes to the fork branch, then pull into ddrs with `cargo update -p cubecl` (or the changed crate). Do NOT commit `path = "..."` overrides — they break the public build. + +4. **rskan is pinned to a tag**, not a branch: `rskan = { git = "https://github.com/taddyb/rskan.git", tag = "v0.1.3" }`. Bumping the tag requires re-running the KAN parity sweep (CLAUDE.md invariant 6-7). + +### Diagnosing fork resolution failures + +``` +error[E0277]: the trait `Device` is not implemented for ... +``` +→ burn crate split across crates.io and fork. Run `cargo tree -p burn-std` to find which sub-crate is from crates.io. Add it to `[patch.crates-io]`. + +``` +error: failed to resolve patches for ... +``` +→ The fork branch was renamed or deleted. Check `github.com/taddyb/cubecl` or `github.com/taddyb/burn` for current branch name. + +--- + +## 4. Static netcdf/HDF5 Build Details + +`netcdf = { version = "0.12", features = ["static"] }` causes Cargo's build script to: +1. Download HDF5 and netcdf-c sources. +2. Compile them via cmake during `cargo build`. +3. Link them statically into the final binary. + +This is intentional: the HPC hosts lack usable `libnetcdf-dev`/`libhdf5-dev` packages (the module-provided ones are Intel-MPI-flavored and break `ddrs` outside `module load`). + +**Build time impact**: first build takes several extra minutes for the cmake compile. Subsequent builds are cached by Cargo. + +**If cmake is not found**: +``` +error: failed to execute process `cmake`: No such file or directory +``` +Install cmake (see §1b) and retry `cargo build`. + +**If cmake finds system HDF5 but produces link errors**: the `static` feature should bypass system HDF5. Ensure you have not set `HDF5_DIR` in your environment pointing at an incompatible installation: +```bash +unset HDF5_DIR +unset NETCDF_DIR +cargo build --release +``` + +--- + +## 5. CPU-only Override + +On a machine without CUDA, the default config (`config/merit_training.yaml`) fails because it requests `sparse_solver: cuda`. Override it via a minimal config file: + +```yaml +# cpu_override.yaml (do not commit) +sparse_solver: cpu +use_cuda_graphs: false +``` + +```bash +ddrs --config cpu_override.yaml plan +ddrs --config cpu_override.yaml run --workflow train +# or for just the V1 gate (V1 always defaults to NdArray/CPU): +cargo run --release --example compare_ddr_sandbox # no override needed +``` + +The CPU NdArray backend is the default for `compare_ddr_sandbox`; V1 always passes on CPU. Only training-scale runs need the GPU config override. + +--- + +## 6. Gitignored Fixtures and Outputs + +Three categories of files are gitignored and must be recreated after a fresh clone: + +### 6a. Sandbox fixtures (`/fixtures/`, `examples/fixtures/`) + +These are the V1 gate inputs generated by DDR's Python solver. They are gitignored because they are derived artifacts. `tests/fixtures/` IS tracked; only root-level and examples-level fixtures are excluded. + +```bash +# Regenerate after DDR's solver changes, or on a fresh clone +cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/export_ddr_sandbox.py +``` + +CRITICAL CAVEAT (as of 2026-07-05): The valid V1 fixture lives ONLY in the desktop's `~/projects/ddr` working tree. That tree contains unpushed work — `src/ddr/geometry/trapezoidal.py` — that does not exist in any DDR public commit. Regenerating from a clean DDR clone produces a ~1%-divergent reference (max abs ≈ 0.55 m³/s) that would make V1 fail at every ddrs commit. This is a wrong-reference artifact, not a port bug. Until DDR's geometry work is pushed, only the desktop DDR tree produces a valid V1 fixture. + +### 6b. Output directory (`output/`) + +The `compare_ddr_sandbox` example writes to `output/ddrs_vs_ddr.{csv,png}` using `File::create`, which does NOT `mkdir -p`. A fresh clone will panic on the file create. + +```bash +mkdir -p output +cargo run --release --example compare_ddr_sandbox +``` + +### 6c. Workspace artifacts (`.ddrs/`) + +The entire `.ddrs/` directory is gitignored. This includes adjacency caches, baseline caches, run manifests, checkpoints, and run logs. They are rebuilt by `ddrs plan` on first run. + +--- + +## 7. Stale-Binary Trap + +This is one of the most common sources of silent wrong behavior. + +**The problem**: `ddrs` on your PATH is `~/.cargo/bin/ddrs`. `cargo build` and `cargo run` compile into `target/release/ddrs` but do NOT copy it to `~/.cargo/bin/`. If you edit `src/` and then type `ddrs run`, you silently execute the old binary. + +The manifest's `git.sha` field is stamped from `.git` at runtime, NOT from the binary. A run can look like current code in the manifest while a weeks-old binary actually executed. This caused the 2026-07-01 leakance×hourly 2×2 to produce byte-identical hourly cells — the installed binary predated the disaggregation feature. + +**Self-check**: current checkpoints are DIRECTORIES (`epoch_E_mb_M/head.mpk` etc.). If you see flat `.mpk` files at `epoch_E_mb_M.mpk`, you ran a pre-checkpoint-resume binary. + +**Fix after every `src/` change**: + +```bash +# Canonical (always correct): +cargo install --path . + +# Faster when target/release is already built: +cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs + +# Bypass the installed binary entirely (safest during development): +cargo run --release --bin ddrs -- run --workflow train +``` + +--- + +## 8. CUDA Graphs Masking NaN + +**The trap**: `use_cuda_graphs: true` records a graph on the first forward pass and replays it on subsequent steps. If the first forward passes with a finite loss but a later one would produce NaN (e.g., from a bad parameter initialization or data batch), the graph replay returns the stale finite loss from the captured pass. You see a training run that appears to converge smoothly but is actually computing nothing. + +**Affected config key**: `use_cuda_graphs: true` in `ddrs.yaml` or `config/merit_training.yaml`. + +**Config rejection rule**: `use_leakance: true` + `use_cuda_graphs: true` is REJECTED at config load time (no exception path). These cannot be used together; the leakance kernel has no separate CUDA graph capture path. + +**Diagnosis**: + +```bash +# 1. Disable graphs and rerun the suspicious config +# Add to ddrs.yaml: +# use_cuda_graphs: false + +# 2. Watch for NaN in the loss log — if it now appears, graphs were masking it + +# 3. Find the NaN source: typically a parameter initialized to zero or +# a data batch with all-NaN streamflow +``` + +**Rule**: always validate new configs and new parameter initializations with `use_cuda_graphs: false` first. Only re-enable graphs after confirming the forward is NaN-free. + +--- + +## 9. Critical Invariants (Do Not Break) + +Breaking any of these makes the ddrs port meaningless or incorrect. + +| # | Invariant | Test | +|---|---|---| +| 1 | `compare_ddr_sandbox` must report ABSOLUTE MATCH (max abs < 1e-3 m³/s) | `cargo run --release --example compare_ddr_sandbox` | +| 2 | f32 throughout routing core; no f64/bf16 casts in `src/routing/`, `src/geometry.rs`, `src/sparse.rs` | `grep -rn 'f64\|bf16\|cast\|to_dtype' src/routing/ src/geometry.rs src/sparse.rs` | +| 3 | Adjacency is topologically ordered, lower-triangular (`rows[k] >= cols[k]`) | `cargo test data_zarr_store::conus_adjacency_loads_real_merit_zarr` | +| 4 | Hand-written sparse backward in `src/sparse.rs` must NOT be replaced with tape unrolling | `cargo test --test sparse_gradcheck` | +| 5 | KAN head = `rskan::KanLayer` via `src/nn/kan_head.rs`; no MLP placeholder; no inter-block ReLU | `cargo test --test kan_head` | +| 6 | rskan pinned to tag `v0.1.3` in Cargo.toml | `grep 'rskan.*tag' Cargo.toml` | +| 7 | KAN parity vs DDR must pass on every PR touching `src/nn/`, rskan pin, or DDR's `nn/kan.py` | See §11 KAN parity command | + +--- + +## 10. Full Verification Command Set + +Run these in order after a fresh build or after touching `src/`: + +```bash +# V1 gate — must report ABSOLUTE MATCH +mkdir -p output +cargo run --release --example compare_ddr_sandbox + +# V1 gate on CUDA + graph-capture path (if GPU available) +DDRS_FORCE_GRAPHS=1 cargo run --release --example compare_ddr_sandbox + +# Sparse gradient correctness +cargo test --test sparse_gradcheck + +# Routing correctness (linear chain) +cargo test --test mmc mc_routes_linear_chain + +# Leakance gradient-exactness (if leakance was touched) +cargo test --test leakance_gradcheck +cargo test --test leakance_off_parity +cargo test --test zeta_accum +``` + +--- + +## 11. KAN Head Parity (after touching `src/nn/`, rskan pin, or DDR's `nn/kan.py`) + +```bash +cargo test --features fixtures \ + --test kan_head_init_repro \ + --test kan_head_init_parity \ + --test kan_head_fixture_forward \ + --test kan_head_fixture_backward +``` + +If DDR's `nn/kan.py` changed: regenerate fixtures first, then re-validate: +```bash +cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/dump_kan_fixture.py +cd ~/projects/ddrs && cargo test --features fixtures --test kan_head_fixture_forward --test kan_head_fixture_backward +``` + +--- + +## 12. DDR Reference Clone (for fixture regeneration) + +```bash +git clone git@github.com:mhpi/ddr ~/projects/ddr +cd ~/projects/ddr && uv sync --all-packages +``` + +uv creates a `.venv` automatically. Fixtures are generated by running scripts under this venv: + +```bash +cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/export_ddr_sandbox.py +``` + +The uv venv must remain at `~/projects/ddr/.venv` — ddrs scripts import from DDR's Python packages. + +--- + +## 13. Data File Paths + +`config/merit_training.yaml`'s `data_sources:` block references these paths. Edit the YAML to match your machine if they live elsewhere. + +| Source | Default path | +|---|---| +| Geospatial fabric | `riv_pfaf_7_MERIT_Hydro_v07_Basins_v01_bugfix1.shp` (+ sibling `.dbf`), or a `.gpkg` | +| MERIT adjacency | Managed — built by `ddrs plan` into `.ddrs/adjacency//` | +| Per-gauge subgraphs | Managed — same directory | +| Catchment attributes | `~/projects/ddr/data/merit_global_attributes_v2.nc` | +| Streamflow forcing | `/mnt/ssd1/data/icechunk/merit_dhbv2_UH_retrospective.ic` | +| USGS observations | `/mnt/ssd1/data/icechunk/usgs_daily_observations` | +| Gauges list | `~/projects/ddr/references/gage_info/gages_3000.csv` | + +To skip the managed adjacency build (e.g., you have pre-built zarr stores): +```yaml +# in ddrs.yaml, replace geospatial_fabric with: +conus_adjacency: /path/to/merit_conus_adjacency.zarr +gages_adjacency: /path/to/merit_gages_conus_adjacency.zarr +``` + +--- + +## 14. Common Failure Modes at a Glance + +| Symptom | Root cause | Fix | +|---|---|---| +| `cmake: No such file or directory` during `cargo build` | cmake not on PATH; needed for static netcdf | Install cmake (§1b) | +| `the trait Device is not implemented` link error | burn crate split across crates.io and fork | Check `cargo tree -p burn-std`; add missing crate to `[patch.crates-io]` | +| `failed to resolve patches` | Fork branch renamed or deleted | Check `github.com/taddyb/{cubecl,burn}` for current branch | +| `thread 'main' panicked at 'No such file or directory' (output/...)` | `output/` missing on fresh clone | `mkdir -p output` | +| V1 fails with max abs ≈ 0.55 m³/s | Fixtures regenerated from wrong DDR clone (no trapezoidal.py) | Use desktop's `~/projects/ddr` working tree | +| Training appears to converge but loss never moves | CUDA graphs masking NaN | Set `use_cuda_graphs: false` and check for NaN (§8) | +| `ddrs run` uses wrong/old feature after `cargo build` | Stale installed binary at `~/.cargo/bin/ddrs` | `cargo install --path .` (§7) | +| Checkpoint is a flat `.mpk` file, not a directory | Stale pre-checkpoint-resume binary executed | `cargo install --path .` and re-run | +| `use_leakance + use_cuda_graphs rejected at config load` | These two are mutually exclusive by design | Remove `use_cuda_graphs: true` from leakance configs | +| `fixtures/sandbox/` missing, V1 panics on CSV read | gitignored artifact not regenerated | `cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/export_ddr_sandbox.py` | + +--- + +## Provenance and maintenance + +Skill written 2026-07-05 from `Cargo.toml`, `CLAUDE.md`, `README.md`, `.claude/references/ddrs-setup.md`, `.claude/references/ddrs-comparing-to-ddr.md`, and `vendor/README.md`. + +Re-verification commands: +```bash +# Confirm fork branches still exist +git ls-remote https://github.com/taddyb/cubecl.git ddrs-release +git ls-remote https://github.com/taddyb/burn.git ddrs-sp7-primitive-ctor +git ls-remote https://github.com/taddyb/rskan.git refs/tags/v0.1.3 + +# Confirm static netcdf feature still declared +grep 'netcdf.*static' /home/tbindas/projects/ddrs/Cargo.toml + +# Confirm gitignore entries +grep -E 'fixtures|output|\.ddrs' /home/tbindas/projects/ddrs/.gitignore + +# Confirm Rust version meets minimum +rustc --version # must be >= 1.80 + +# V1 gate +mkdir -p /home/tbindas/projects/ddrs/output +cargo run --release --example compare_ddr_sandbox 2>&1 | grep -E 'verdict|ABSOLUTE|FAIL' +``` diff --git a/.claude/skills/ddrs-change-control/SKILL.md b/.claude/skills/ddrs-change-control/SKILL.md new file mode 100644 index 0000000..0c3464b --- /dev/null +++ b/.claude/skills/ddrs-change-control/SKILL.md @@ -0,0 +1,413 @@ +--- +name: ddrs-change-control +description: "Use when reviewing, gating, or merging a change to ddrs source code, config files, or Cargo dependencies; when assessing whether a modification to src/routing/, src/sparse.rs, src/geometry.rs, src/nn/kan_head.rs, or Cargo.toml is safe; when a run produced unexpected results and binary staleness or an invariant violation may be the cause; or when designing an experiment that touches leakance, CUDA graphs, or the routing core." +--- + +# ddrs change-control runbook + +## Overview + +`ddrs` is a BURN-0.21 Rust port of the DDR differentiable Muskingum-Cunge routing model (Python/PyTorch reference at `~/projects/ddr/`). The port must remain **gradient-exact** against DDR. Breaking any of the seven invariants below makes the port meaningless; every PR that touches the affected files must clear its gate before merge. + +**Glossary for PyTorch engineers:** +- `BURN` — Rust deep-learning framework, analogous to PyTorch. BURN 0.21 is pinned. +- `Backward` — BURN's trait for a custom autograd function (analogous to `torch.autograd.Function`). `I` = backend (CPU/CUDA), `N` = number of saved tensors. +- `CsrPattern` — the sparsity structure of the river network adjacency matrix, stored as a Rust struct (row/col index arrays). Analogous to `torch.sparse_csr_tensor`. +- `KanLayer` — Kolmogorov-Arnold Network layer from the `rskan` crate (Rust equivalent of DDR's `kan.py`). +- `CompactRecorder` / `HalfPrecisionSettings` — BURN serializer for checkpoints. Saves weights as f16. +- `COMID` — unique 64-bit integer ID for each river reach in the MERIT-Hydro fabric. +- `Q'` (Qr) — lateral inflow forcing (divide-level runoff, m³/s) from the pre-trained DHBV2 model. Not observed discharge. +- `zeta` — leakance flux (m³/s), the GW–SW water-loss term. Subtracted from the routing RHS at every timestep. +- `rho-window` — a training mini-batch: a contiguous time slice of length `rho` (default 90 days) sampled from the training period. + +--- + +## When NOT to use this skill + +Do not use this skill for: +- **Plotting or analysis scripts only** (no `src/` change) — use `ddrs-eval-plots` instead. +- **Config tuning within documented safe ranges** (changing `experiment.epochs`, `learning_rate`, `batch_size`, loss weights) — no gate applies; these do not affect the routing core or port invariants. +- **Data source path changes only** — consult `CLAUDE.md §Data sources` directly. +- **CLI / workspace questions** — consult `CLAUDE.md §ddrs CLI`. + +--- + +## Change classification matrix + +Every change falls into one of four tiers. Look up the modified file(s) in the left column; the tier determines which gate checklist you must run. + +| Modified file(s) | Tier | Rationale | +|---|---|---| +| `src/routing/mmc.rs`, `src/routing/mmc_op.rs`, `src/routing/utils.rs` | **A — routing core** | Directly implements the Muskingum-Cunge timestep; must remain gradient-exact vs DDR | +| `src/routing/leakance.rs` | **A — routing core + leakance** | Custom `Backward` for the GW–SW term; both the forward kernel and analytical gradients must stay exact | +| `src/geometry.rs` | **A — routing core** | Trapezoidal geometry; changes cascade into every geometry-dependent variable | +| `src/sparse.rs` | **A — routing core** | Hand-written CSR triangular solve + custom `CsrSolveOp: Backward`; O(nnz) autograd tape invariant | +| `src/nn/kan_head.rs` | **B — KAN head** | Must match DDR `kan.py` exactly; rskan version pin governs this | +| `Cargo.toml` (rskan tag) | **B — KAN head** | rskan pin is the single authoritative version for KAN parity | +| `src/config.rs` | **C — config/ranges** | Parameter ranges and log-space flags affect denormalization; wrong range silently mis-scales gradients | +| `src/training/loss.rs` | **C — objective** | Autograd is unchanged (invariant 4 intact) but loss changes affect all metrics comparisons | +| `src/training/forward.rs` | **C — training path** | Disaggregation, leakance threading; changes can silently no-op features (see STALE-BINARY TRAP) | +| `config/experiments/*.yaml`, `config/sources/*.yaml` | **D — config only** | No Rust changes; validate with `ddrs plan` before running | +| Any other `src/` file | **C — default** | Run full test suite and DDR regression | + +--- + +## Tier A gate — routing core + +Run ALL of the following. A single failure is a merge blocker. + +```bash +# 1. DDR parity — THE non-negotiable regression +cargo run --release --example compare_ddr_sandbox +# Must print: "ABSOLUTE MATCH" with max abs diff < 1e-3 m³/s + +# 2. Core unit + integration tests +cargo test --lib +cargo test --test mmc +cargo test --test sparse_gradcheck + +# 3. Leakance gates (required even if you did not touch leakance.rs, +# because any routing change can disturb the leakance OFF parity) +cargo test --test leakance_gradcheck # 8/8 analytical ≈ finite-difference +cargo test --test leakance_off_parity # 3/3 byte-identical to no-leakance when off +cargo test --test zeta_accum # 6/6 accumulated zeta == headwater q difference +``` + +If you touched `src/routing/leakance.rs` specifically: +```bash +# Confirm gradient-exactness for all 8 leakance backward inputs +cargo test --test leakance_gradcheck -- --nocapture +``` + +**After passing all Tier A gates, run the binary self-check:** +```bash +# Stale-binary check: directory checkpoints = current binary +ls .ddrs/runs//checkpoints/ +# Must show directories like epoch_5_mb_9/, NOT flat files like epoch_5_mb_9.mpk +# Flat files = stale binary ran. Refresh: cargo install --path . +``` + +--- + +## Tier B gate — KAN head + +```bash +# Full KAN parity sweep (required on every PR touching src/nn/ or Cargo.toml rskan pin) +cargo test --features fixtures \ + --test kan_head_init_repro \ + --test kan_head_init_parity \ + --test kan_head_fixture_forward \ + --test kan_head_fixture_backward + +# Then run Tier A DDR parity to confirm the head change did not break routing +cargo run --release --example compare_ddr_sandbox +``` + +If a DDR-side change to `kan.py` broke the fixture: +```bash +# Regenerate under DDR's venv, then re-validate +cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/dump_kan_head.py +# Re-run the fixture tests above +``` + +--- + +## Tier C gate — config, training path, other src/ + +```bash +cargo test --lib +cargo test # full suite +cargo run --release --example compare_ddr_sandbox # DDR parity +``` + +For `src/training/forward.rs` changes that affect disaggregation or leakance threading, also run: +```bash +cargo test --test leakance_off_parity # leakance OFF must stay byte-identical +``` + +--- + +## Tier D gate — config files only + +```bash +# Validate the config parses and data sources resolve +ddrs plan --config config/experiments/.yaml \ + --workspace /home/tbindas/projects/ddrs/.ddrs +# Must exit 0 with no "drift" warnings +``` + +--- + +## The 7 non-negotiables with rationale and incidents + +### Invariant 1 — DDR sandbox ABSOLUTE MATCH + +**Rule.** `cargo run --release --example compare_ddr_sandbox` must print "ABSOLUTE MATCH" (max abs diff < 1e-3 m³/s on the 5-reach RAPID sandbox). Re-run after every change to `src/routing/`, `src/geometry.rs`, or `src/sparse.rs`. + +**Rationale.** The port exists to be gradient-exact against DDR. Any drift makes subsequent metric comparisons meaningless — you cannot tell whether a difference is a port bug or a genuine model improvement. + +**Caveat (as of 2026-06-06).** The reference DDR state lives only in the desktop's `~/projects/ddr` working tree (contains unpushed `geometry/trapezoidal.py` changes). A fixture regenerated from a clean DDR clone diverges ~1% per commit — that is a wrong reference, not a port bug. See `.claude/references/ddrs-comparing-to-ddr.md §Regenerating fixtures` before regenerating. + +--- + +### Invariant 2 — f32 throughout routing core + +**Rule.** No casts to f64 or bf16 inside `src/routing/`, `src/geometry.rs`, or `src/sparse.rs`. The DDR comparison sits at the f32 precision floor (~1e-7 relative difference per reach); any precision change breaks reproducibility. + +**Rationale.** Mixed precision introduces per-reach rounding that accumulates across the 346,321-reach CONUS network; the 1e-3 m³/s sandbox tolerance is calibrated for f32-only arithmetic. + +--- + +### Invariant 3 — lower-triangular adjacency + +**Rule.** The adjacency matrix must be topologically sorted and lower-triangular: `rows[k] >= cols[k]` for every non-zero entry. The forward-substitution solver (`triangular_solve_lower`) assumes no upstream values are uncomputed when it processes a reach. + +**Rationale.** Forward substitution over a topological order is the entire basis for the O(n) per-timestep solve. A non-lower-triangular entry means a downstream reach tries to read an upstream value before that upstream reach is solved — silent wrong output, no error. + +**Test.** `cargo test data_zarr_store::conus_adjacency_loads_real_merit_zarr` verifies the invariant on the real CONUS zarr store. + +--- + +### Invariant 4 — hand-written sparse backward + +**Rule.** Do NOT replace the hand-written `CsrSolveOp impl Backward` in `src/sparse.rs` with autograd-tape unrolling. + +**Rationale.** The entire point of the custom backward is O(nnz) tape entries per timestep. Tape unrolling would be O(n²) for a triangular solve — quadratic memory and time, infeasible at CONUS scale (346,321 reaches). The analytical backward is `∇A = -gradb[rows]·x[cols]`, exactly as in DDR's `torch.autograd.Function`. + +**Reference.** `.claude/references/ddrs-burn-autograd.md` has the full BURN-0.21 recipe. + +--- + +### Invariant 5 — KAN head architecture matches DDR + +**Rule.** The routing head is `rskan::KanLayer` via `src/nn/kan_head.rs`. The architecture is `Linear(F, H) → KanLayer(H, H) × num_hidden_layers → Linear(H, P) → Sigmoid`. No inter-block ReLU. All `num_hidden_layers` inner KanLayers receive the SAME seed (DDR's `kan.py` lines 24–34 quirk — preserved for parity). + +**Rationale.** DDR parity requires identical weight initialization. A ReLU between KAN blocks or different per-layer seeds changes the initialization and breaks the fixture tests. + +**What NOT to do.** Do not reintroduce the prior MLP placeholder. + +--- + +### Invariant 6 — rskan pinned to a tag + +**Rule.** `rskan` in `Cargo.toml` must be a git dependency pinned to a tag, currently `v0.1.3`. When updating, bump the tag, then re-run all Tier B tests and the Tier A DDR regression before merging. + +**Rationale.** An unpinned git dependency (`branch = "main"`) can change silently on `cargo update`, breaking KAN parity without any local code change. + +**Current pin (as of 2026-07-05):** +```toml +rskan = { git = "https://github.com/taddyb/rskan.git", tag = "v0.1.3" } +``` + +--- + +### Invariant 7 — KAN head parity on every relevant PR + +**Rule.** Any PR touching `src/nn/`, `Cargo.toml`'s rskan pin, or DDR's `nn/kan.py` must pass the full KAN parity suite (see Tier B gate above). + +**Rationale.** Invariant 5 is not self-enforcing at the compiler level. The fixture tests are the only automated check that the architecture, seed, and weight initialization actually match DDR. + +--- + +## The STALE-BINARY TRAP (historical incident: 2026-07-01) + +**Rule.** After touching ANY file under `src/`, refresh the installed binary before running experiments. + +```bash +# Canonical (always correct): +cargo install --path . + +# Faster if target/release is already built: +cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs + +# Bypass entirely (safest for one-off experiments): +cargo run --release --bin ddrs -- run --workflow train-and-test ... +``` + +**Why this matters.** `ddrs` on your PATH is `~/.cargo/bin/ddrs`. `cargo build` and `cargo run` do NOT update it. The manifest's `git.sha` is stamped from `.git` at runtime, so a run appears to have the correct SHA while silently executing a weeks-old binary. + +**What happened (2026-07-01).** The installed `ddrs` was dated 2026-06-03, before disaggregation (landed 2026-06-19) and leakance (landed 2026-06-29). The hourly-forcing cell silently ran flat repeat-24 with no leakance. Both hourly-ON and daily-ON cells produced byte-identical eval predictions (`52ec721`). The manifest showed `git.sha = 2cdd341` (correct HEAD), masking the stale binary completely. + +**Self-check.** Current checkpoints are DIRECTORIES: `.ddrs/runs//checkpoints/epoch_E_mb_M/head.mpk`. A stale pre-checkpoint-resume binary writes FLAT files: `epoch_E_mb_M.mpk`. Flat files = stale binary. + +--- + +## CUDA graphs + NaN masking (known gotcha) + +**Rule.** Validate model forwards with `use_cuda_graphs: false` when debugging NaN loss or unexpected constant loss. + +**Why.** `use_cuda_graphs: true` captures a kernel graph on the first forward pass and replays it on subsequent passes. If the first forward produces a NaN (e.g., during early training on bad data), the captured graph replays stale finite values rather than recomputing and propagating the NaN. The result is a constant finite loss that does not go to NaN even when the actual computation is invalid. See memory file `cuda-graphs-mask-nan.md` for the full diagnosis. + +**Hard constraint.** `params.use_leakance: true` combined with `use_cuda_graphs: true` is REJECTED at config load time. The leakance kernel is not captured in the current CUDA graph implementation; the rejection prevents silent wrong results. + +--- + +## Leakance-specific gates + +Leakance (`params.use_leakance: true`) is experimental and off by default. Any change that enables, modifies, or interacts with leakance must satisfy these gates in addition to the appropriate tier gates. + +### Enabling leakance requires three config changes together + +Missing any one causes either a config-load error or silent wrong behavior: + +| Config key | Required value | Why | +|---|---|---| +| `params.use_leakance` | `true` | Activates the leakance kernel in `route_timestep` | +| `kan_head.learnable_parameters` | Include `K_D`, `d_gw`, `leakance_factor` | Without these, the KAN head does not emit leakance params → all-zero zeta | +| `params.parameter_ranges.K_D` | `[1e-8, 1e-6]` (log-space) | Range gate; current recommendation is `[1e-8, 1e-5]` for recoverability experiments (see §Research status) | +| `params.parameter_ranges.d_gw` | `[-2, 2]` | Groundwater depth offset (m) | +| `params.parameter_ranges.leakance_factor` | `[0, 1]` | Dimensionless scale | +| `use_cuda_graphs` | `false` | Enforced by config load; leakance + graphs = rejected | + +### Leakance gradient-exactness gate + +```bash +cargo test --test leakance_gradcheck # 8/8 — all analytical grads match finite-diff +cargo test --test leakance_off_parity # 3/3 — OFF is byte-identical to no-leakance +cargo test --test zeta_accum # 6/6 — accumulated zeta == headwater q difference +cargo run --release --example compare_ddr_sandbox # must still say ABSOLUTE MATCH +``` + +### Leakance eval-time zeta diagnostic + +`dump_parameters` exports learned `K_D`/`d_gw`/`leakance_factor` per COMID but NOT the actual zeta flux (which depends on routed depth, only available during eval). The zeta diagnostic runs during `ddrs run --workflow train-and-test` Phase 2 automatically. For an existing checkpoint: + +```bash +cargo build --release --bin eval +target/release/eval \ + --config config/experiments/leakance_hourly_on.yaml \ + --checkpoint .ddrs/runs//checkpoints/epoch_5_mb_9 \ + --output /tmp/eval.zarr \ + --zeta-output .ddrs/runs//kan_parameters.nc +``` + +Output variables in `kan_parameters.nc` (dimension `COMID_eval`, 64,892 reaches on the CONUS eval network): +- `zeta` — mean |zeta| (m³/s) over eval window +- `zeta_net` — signed mean zeta (positive = losing reach) + +**GO/NO-GO bar.** |zeta| > 0.01 m³/s on at least 10% of eval reaches = zeta is physically active. + +--- + +## Research-status facts (as of 2026-07-05) + +These facts govern what claims can be made about leakance. Cite dates; do not generalize beyond what is measured. + +### Leakance 2×2 (as of 2026-07-01) — DONE + +Four valid arms, seed 42, eval window 1995/10/01–2010/09/30, 2,365 gauges: + +| arm | run id | NSE med | KGE med | +|---|---|---|---| +| hourly-OFF | `2026-06-23T02-49-12Z-conus-hourly-train-and-test` | 0.7153 | 0.7104 | +| hourly-ON | `2026-07-01T13-43-32Z-train-and-test` | 0.7145 | 0.7150 | +| daily-OFF | `2026-06-05T01-41-16Z-train-and-test` | 0.7004 | 0.7244 | +| daily-ON | `2026-07-01T21-20-27Z-train-and-test` | 0.6963 | 0.7250 | + +Losing-stream subset (1,883/2,365 gauges): hourly leakance ΔNSE +0.0005, ΔKGE +0.0018, 55.5% of gauges improve. Daily leakance ΔNSE −0.0017, ΔKGE −0.0009. Verdict: **GO — marginal** (3/3 gates met; zeta |>0.01| on 10.4% of 64,892 eval reaches). + +**Summed-Q' baseline (CONUS):** median NSE 0.689, KGE 0.723. Best trained result (precip-driven disagg + L1): median NSE 0.715, KGE 0.711 (2,365 gauges). NSE beats the baseline by +0.026; KGE does NOT beat the summed-Q' baseline in any config as of 2026-07-05. + +### Low-zeta diagnosis (as of 2026-07-02) + +| Hypothesis | Verdict | +|---|---| +| H1 — K_D ceiling clips zeta | REFUTED (71.5% of reaches CAN exceed 0.01 m³/s in-box; utilization median 3.4%) | +| H2 — driving-head starvation | SUPPORTED (median head 0.021 m; 47% of reaches gaining at eval-window mean) | +| H3 — KAN variance collapse | REFUTED (K_D–aridity ρ +0.61, d_gw–meanP ρ +0.71 — strong learned structure) | +| H4 — gauge bias / gradient starvation | SUPPORTED (gauged median |zeta| 11× ungauged; dry/wet zeta ratio 0.40, inverse of physics) | +| H5 — equifinality (daily only) | SUPPORTED (daily Δn +0.012, 0.59 IQR; hourly Δn nil) | +| H6 — wrong yardstick | REFUTED (fractional loss agrees: 8.4% of reaches lose >1% of local flow) | +| H7 — model-form error | REFUTED (0.0% of d_gw at bounds) | + +**Implication.** The K_D-widening follow-up (`[1e-8, 1e-5]`) recommended in the 2×2 findings is NOT recommended by the diagnosis. The diagnosis shows the K_D box is not the binding constraint. Widening K_D alone is expected to re-pin at the new ceiling with negligible zeta or skill change. + +### Gradient probe (as of 2026-07-03, worktree: zeta-sensitivity) + +| Probe | Verdict | Key number | +|---|---|---| +| P1 — gradient starvation | REFUTED | gauged/ungauged \|g\| ratio 1.5× trained, 2.9× cold (bar: ≥10×) | +| P2 — rejection at trained point | REFUTED | 52.5% of dry-tercile grads push zeta down (bar: >67%) | +| P3 — detectability | NO-GO | 4.2% of Ref probes detectable at δ=0.01 m³/s (bar: ≥10%); median 5%-band 0.531 m³/s vs planted signal 0.01 m³/s = **53× dilution** | + +P3 NO-GO means: gauge-only discharge supervision cannot distinguish real-world leakance magnitudes from measurement uncertainty. Transmission is fine (~95% fidelity); the problem is signal-to-noise at the sensor. + +### Synthetic recoverability positive control (as of 2026-07-04, worktree: zeta-sensitivity) + +| Metric | Measured | Verdict | +|---|---|---| +| R1 — recovery ratio median (n=58) | 0.009 | FAILED (bar: ≥0.5) | +| R2 — non-planted spatial precision | 1.11× baseline | PRECISE (trivial: model didn't move) | +| R3 — loss gap A vs B | A=1.339 vs B=2.317, +42% | A3) | + +**Root cause.** Windowed training objective (rho=90, warmup=5) has a ~130× hotstart-transient noise floor vs the planted signal. The continuous residual with teacher weights + teacher obs is 0.0076 mean L1; step-0 windowed training loss is 1.017. The optimizer chases irreducible initial-condition noise; after 5 epochs the continuous residual degrades from 0.0076 to 0.4431. + +**Implication.** Leakance identifiability is NOT proven. Phase B (state-cache hotstart, target: windowed loss ≤ 0.25 mean L1, ≤10% of a converged run) is required before any identifiability claim. Phase B is NOT yet complete as of 2026-07-05. + +--- + +## Binary management quick-reference + +| Goal | Command | +|---|---| +| Refresh installed binary after src/ change | `cargo install --path .` | +| Fast refresh (target/release already built) | `cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs` | +| Bypass installed binary for one run | `cargo run --release --bin ddrs -- run --workflow train-and-test ...` | +| Check if stale binary ran | Look for FLAT checkpoint files `epoch_E_mb_M.mpk`; current binary writes DIRECTORIES `epoch_E_mb_M/head.mpk` | + +--- + +## Workspace flag gotcha + +`--workspace` takes the path to the `.ddrs` DIRECTORY ITSELF, not its parent. Experiment configs in `config/experiments/` default to `config/experiments/.ddrs` (wrong). Always pass the root workspace explicitly: + +```bash +ddrs run --config config/experiments/leakance_hourly_on.yaml \ + --workspace /home/tbindas/projects/ddrs/.ddrs \ + --workflow train-and-test +``` + +--- + +## Provenance and maintenance + +Files read to write this skill (re-read to verify any fact): + +```bash +# Core invariants and CLI behavior +cat /home/tbindas/projects/ddrs/CLAUDE.md + +# Architecture and per-timestep dataflow +cat /home/tbindas/projects/ddrs/.claude/ARCHITECTURE.md + +# Stale-binary incident + 2×2 experiment +cat /home/tbindas/projects/ddrs/docs/2026-07-01-leakance-hourly-experiment-handoff.md + +# 2×2 final findings and GO-marginal verdict +cat /home/tbindas/projects/ddrs/docs/2026-07-01-leakance-hourly-findings.md + +# Low-zeta diagnosis (H1-H7 verdicts) +cat /home/tbindas/projects/ddrs/docs/2026-07-02-leakance-diagnosis-findings.md + +# Gradient probe (P1-P3 verdicts) — worktree +cat /home/tbindas/projects/ddrs/.claude/worktrees/zeta-sensitivity/docs/2026-07-03-zeta-gradient-probe-findings.md + +# Recoverability positive control failure — worktree +cat /home/tbindas/projects/ddrs/.claude/worktrees/zeta-sensitivity/docs/2026-07-04-synthetic-recoverability-findings.md + +# rskan version pin +grep rskan /home/tbindas/projects/ddrs/Cargo.toml +``` + +Re-verification commands: +```bash +# Confirm invariant 1 still holds on current HEAD +cargo run --release --example compare_ddr_sandbox + +# Confirm rskan pin +grep rskan /home/tbindas/projects/ddrs/Cargo.toml + +# Confirm leakance tests pass +cargo test --test leakance_gradcheck --test leakance_off_parity --test zeta_accum +``` diff --git a/.claude/skills/ddrs-config-and-flags/SKILL.md b/.claude/skills/ddrs-config-and-flags/SKILL.md new file mode 100644 index 0000000..ff05bc1 --- /dev/null +++ b/.claude/skills/ddrs-config-and-flags/SKILL.md @@ -0,0 +1,400 @@ +--- +name: ddrs-config-and-flags +description: "Use when you need to add, change, or audit any ddrs YAML configuration key; diagnose a config-load error; understand what a parameter controls; add a new routing parameter or training flag; or decide which experiment config to use as a starting point. Also use when modifying params.use_leakance, params.use_cuda_graphs, kan_head.disaggregation, or experiment.loss." +--- + +# ddrs Config and Flags Reference + +**Jargon primer (defined once):** +- **BURN** — Rust deep-learning framework (like PyTorch for Rust). Used instead of PyTorch here. +- **KAN head** — Kolmogorov-Arnold Network; maps per-reach catchment attributes to routing parameters. Replaces an MLP. +- **MC routing** — Muskingum-Cunge, a 1-D river routing solver. The `params:` block controls it. +- **CONUS** — Contiguous US; the default training domain (346,321 reaches). +- **Q'** — lateral inflow (m³/s) per reach; the forcing signal from dHBV2. +- **CSR** — Compressed Sparse Row; the sparse matrix format used for the triangular network solve. +- **CUDA graph** — GPU kernel sequence baked at compile time and replayed cheaply each timestep. +- **ddrs.yaml** — the live workspace config; `ddrs plan` generates it from a template. NEVER committed. +- **config/merit_training.yaml** — the canonical production template; committed and kept in sync with DDR-Python. + +--- + +## When NOT to use this skill + +| Situation | Use instead | +|---|---| +| Debugging NaN loss or gradient explosion | `ddrs-systematic-debugging` | +| Adding a new data source format (new zarr reader, new fabric) | `ddrs-data-sources` | +| Understanding the sparse backward / autograd tape | `.claude/references/ddrs-burn-autograd.md` | +| Per-timestep routing math | `.claude/ARCHITECTURE.md` | +| CLI lifecycle (`ddrs plan`, `ddrs run`, `ddrs gc`) | `CLAUDE.md` §"ddrs CLI" | + +--- + +## Overview: config file anatomy + +A ddrs config is a single YAML file with six top-level sections: + +``` +mode / workflow / geodataset / device / seed / np_seed ← top-level scalars +data_sources: ← where inputs live on disk +experiment: ← training-loop hyperparameters +kan_head: ← KAN head architecture + which parameters it predicts +params: ← routing engine settings +testing: ← overlay applied in eval mode (overrides experiment: keys) +``` + +The Rust struct is `src/config.rs::Config`. Deserialization uses +`Config::from_yaml_file_with_mode(path, ConfigMode::Training|Testing)`. + +--- + +## Top-level scalars + +| Key | Type | Default | Notes | +|---|---|---|---| +| `mode` | `"training"` \| `"testing"` | `"training"` | Must agree with `workflow:` (see guard below). | +| `workflow` | `train` \| `eval` \| `train-and-test` | absent (None) | `train-and-test` runs both phases and computes the baseline comparison. | +| `geodataset` | `"merit"` | `"merit"` | Only value supported as of 2026-07-05. | +| `device` | integer | `0` | CUDA device ordinal. On multi-GPU hosts, pick a non-display GPU. | +| `seed` | integer | `42` | Controls KAN weight initialization. | +| `np_seed` | integer | `42` | Controls per-epoch gauge shuffle order. | + +**Guard:** `mode: training` requires `workflow ∈ {train, train-and-test}`. `mode: testing` requires `workflow: eval`. A contradiction is rejected at load time with a message containing `"conflicting top-level keys"`. + +--- + +## `data_sources:` section + +All data is read in place — no export step. Every path is a `PathBuf`. + +| Key | Required | Notes | +|---|---|---| +| `attributes` | Yes | NetCDF catchment attributes. Columns must match `kan_head.input_var_names`. | +| `streamflow` | Yes | dHBV2 lateral inflow Q'. Icechunk (`.ic`) for CONUS; zarr-v2 for global. | +| `observations` | Yes | USGS (or global) daily observed discharge; training targets. | +| `gages` | Yes | CSV with STAID and COMID columns. | +| `geospatial_fabric` | Conditional | `.shp`/`.dbf`/`.gpkg`; triggers managed adjacency build into `.ddrs/adjacency//`. | +| `geospatial_fabric_layer` | Optional | Layer name inside a multi-layer `.gpkg`; invalid for `.shp`/`.dbf`. | +| `conus_adjacency` | Conditional | Pre-built zarr. Must be paired with `gages_adjacency`. | +| `gages_adjacency` | Conditional | Pre-built zarr. Must be paired with `conus_adjacency`. | +| `aorc_precip` | Optional | Hourly AORC precip zarr v3 (`merit_unit_catchments.zarr`). Required when `kan_head.disaggregation.use_precip: true`. | + +**Adjacency rule (enforced at load time):** provide EITHER both `conus_adjacency` + `gages_adjacency`, OR `geospatial_fabric` (managed build). Providing only one of the two adjacency zarrs is rejected. Providing none of the three is rejected. + +**Production path (CONUS workstation, as of 2026-07-05):** +```yaml +data_sources: + attributes: /home/tbindas/projects/ddr/data/merit_global_attributes_v2.nc + geospatial_fabric: /projects/mhpi/data/MERIT/raw/continent/riv_pfaf_7_MERIT_Hydro_v07_Basins_v01_bugfix1.shp + streamflow: /mnt/ssd1/data/icechunk/merit_dhbv2_UH_retrospective.ic + observations: /mnt/ssd1/data/icechunk/usgs_daily_observations + gages: /home/tbindas/projects/ddr/references/gage_info/gages_3000.csv +``` + +--- + +## `experiment:` section (training mode) + +| Key | Type | Default | Production value | Notes | +|---|---|---|---|---| +| `batch_size` | integer | none | `64` | **Gauges** per mini-batch during training. Meaning shifts in `testing:` — see below. | +| `start_time` | `"YYYY/MM/DD"` | none | `"1981/10/01"` | Training window start. | +| `end_time` | `"YYYY/MM/DD"` | none | `"1995/09/30"` | Training window end (water year 1995). | +| `epochs` | integer | none | `5` | Total training epochs. | +| `rho` | integer \| null | none | `90` | Sequence length in days per mini-batch. Set `null` in testing overlay. | +| `shuffle` | bool | `false` | `true` | Re-shuffle gauge order each epoch (seeded by `np_seed`). | +| `warmup` | integer | none | `5` | Days excluded from loss at sequence start (routing spin-up). | +| `learning_rate` | map epoch→f32 | `{}` | `{1: 0.001, 3: 0.0005}` | Step decay; applies from that epoch onward. | +| `grad_clip_max_norm` | float \| absent | absent | `1.0` | Global gradient-norm clip. Omit to disable. | +| `checkpoint` | path \| absent | absent | absent | Directory path to resume from (e.g. `.ddrs/runs//checkpoints/epoch_5_mb_9`). | +| `loss` | block \| absent | L1 (see below) | absent | Training objective. Omit for historical L1. | + +### `experiment.loss:` sub-block + +Omit the entire `loss:` block to use the historical L1 objective. Including the block does NOT change behavior if `kind: l1`. + +| Key | Type | Default | Notes | +|---|---|---|---| +| `kind` | `l1` \| `nnse-kge` \| `kge` | `l1` | `l1` = mean absolute error. `nnse-kge` = composite NNSE + KGE. `kge` = component-weighted KGE (r, alpha, beta terms individually weighted). | +| `nnse_weight` | float | `1.0` | Weight on `1 - NNSE` term (all non-L1 kinds). | +| `kge_weight` | float | `1.0` | Weight on `1 - KGE` Euclidean term (`nnse-kge` only). | +| `r_weight` | float | `1.0` | Weight on `(r-1)²` correlation term (`kge` kind only). | +| `alpha_weight` | float | `1.0` | Weight on `(alpha-1)²` variance ratio (`kge` kind only). This is the restoring force against MC over-attenuation. | +| `beta_weight` | float | `1.0` | Weight on `(beta-1)²` mean ratio (`kge` kind only). | +| `kge_clamp` | float | `10.0` | Per-gauge upper bound on weighted KGE-component sum before averaging. Prevents near-constant gauges from hijacking the batch gradient. | +| `eps` | float | `0.1` | Stabilizes variance/mean denominators. Matches DDR `hydrograph_loss` default. | + +**Why L1 and NSE both fail KGE:** L1 and NSE are both maximized when simulated variance is below observed (NSE optimum is at `alpha = r < 1`). This rewards MC for over-attenuating flood peaks — the diagnosed cause of KGE regression vs the summed-Q' baseline in CONUS runs (median KGE 0.723→0.701 while NSE improved 0.639→0.684). The `(alpha-1)²` term in `nnse-kge` / `kge` supplies a restoring gradient. + +--- + +## `testing:` section (eval-mode overlay) + +These keys **replace** the matching `experiment:` keys when `mode: testing` is loaded. Absent keys inherit from `experiment:`. + +| Key | Default in testing | Notes | +|---|---|---| +| `start_time` | `"1995/10/01"` | Eval window start (water year 1996). | +| `end_time` | `"2010/09/30"` | Eval window end. | +| `batch_size` | `15` | **DAYS** per evaluation chunk — semantic shift from training's gauges-per-batch. | +| `rho` | `null` | Explicitly clears sequence sampling (null is distinct from absent). | + +**CAUTION:** `batch_size` changes meaning between modes. Training: gauges. Testing: days. This is not a typo — it's in the YAML comments. + +--- + +## `kan_head:` section + +The KAN head architecture: `Linear(F,H) → KanLayer(H,H) × num_hidden_layers → Linear(H,P) → Sigmoid`. + +| Key | Type | Code default | Production value | Notes | +|---|---|---|---|---| +| `hidden_size` | integer | none | `21` | Hidden dimension H. | +| `num_hidden_layers` | integer | none | `2` | Inner KanLayer repetitions. ALL receive the SAME seed (DDR `kan.py` quirk, preserved for parity). | +| `grid` | integer | `5` | `50` | B-spline grid intervals per KAN edge (`num` in pykan). Production overrides the code default. | +| `k` | integer | `3` | `2` | B-spline order. DDR overrides pykan's default of 3 to 2 in production; keep 2 for parity. | +| `input_var_names` | list of strings | none | 10 attributes (see below) | Column names in `attributes` NetCDF. | +| `learnable_parameters` | list of strings | none | `[n, q_spatial, p_spatial]` | Parameters the KAN head emits. Must have matching entries in `params.parameter_ranges`. | +| `disaggregation` | block \| absent | absent | absent | Enables the daily→hourly disaggregation head (see sub-block below). Absent = flat repeat-24. | + +**Production `input_var_names` (10 attributes):** +``` +SoilGrids1km_clay, aridity, meanelevation, meanP, NDVI, +meanslope, log10_uparea, SoilGrids1km_sand, ETPOT_Hargr, Porosity +``` + +### `kan_head.disaggregation:` sub-block + +Presence of this block enables the learnable daily→hourly disaggregation head (`src/nn/disagg_head.rs`). Absence = flat `repeat-24` (backward-compatible default). + +| Key | Type | Default | Notes | +|---|---|---|---| +| `hidden_size` | integer | `16` | Hidden dimension of the disagg MLP. | +| `use_attributes` | bool | `true` | Condition on catchment attributes. | +| `use_precip` | bool | `false` | Condition on 72-h AORC precip window `[d-1, d, d+1]`. Requires `data_sources.aorc_precip`. | +| `use_temp` | bool | `false` | Condition on 72-h AORC temperature window. Also requires `data_sources.aorc_precip`. | + +**If `use_precip: true` and `data_sources.aorc_precip` is absent:** `MeritGagesDataset::open` errors at runtime (not at config load time). The missing precip source cannot silently degrade. + +--- + +## `params:` section (routing engine) + +| Key | Type | Code default | Production value | Notes | +|---|---|---|---|---| +| `sparse_solver` | `cpu` \| `cuda` | `cpu` | `cuda` | `cuda` uses cuSPARSE for the triangular solve. Falls back to `cpu` on non-CUDA backends with a WARN log. | +| `use_cuda_graphs` | bool | `false` | `true` | Capture the routing forward as a CUDA graph; faster replay each timestep. **See guards below.** | +| `use_leakance` | bool | `false` | `false` | Enable the GW–SW water-loss term. **See guards below.** Experimental as of 2026-07-05. | +| `tau` | integer | `3` | `3` (not set in YAML) | Muskingum routing sub-step count. Rarely changed. | +| `log_space_parameters` | list of strings | `["p_spatial"]` | `["p_spatial"]` | Parameters whose range spans decades; KAN output is exp-scaled before routing. | +| `defaults` | map str→f32 | `{p_spatial: 21.0}` | `{p_spatial: 21.0}` | Fixed values for parameters NOT in `learnable_parameters`. | + +### `params.parameter_ranges:` sub-block + +Physical `[min, max]` each sigmoid-normalized KAN output maps onto. All defaults are defined in `src/config.rs::ParameterRanges::default()`. + +| Key | Default range | Log-space | Notes | +|---|---|---|---| +| `n` | `[0.015, 0.25]` | No | Manning's roughness coefficient. | +| `q_spatial` | `[0.0, 1.0]` | No | Leopold & Maddock width–depth exponent (`top_width = p·depth^q`). | +| `p_spatial` | `[1.0, 200.0]` | Yes | Leopold & Maddock width coefficient. In log-space by default. | +| `x_storage` | `[0.0, 0.5]` | No | Muskingum storage weight X. Only consumed when listed in `learnable_parameters`; otherwise routing uses constant 0.3. | +| `K_D` | `[1e-8, 1e-6]` | Yes (add to `log_space_parameters`) | Hydraulic exchange rate (1/s). Leakance only. Note: uppercase in YAML. | +| `d_gw` | `[-2.0, 2.0]` | No | Groundwater depth offset (m). Leakance only. | +| `leakance_factor` | `[0.0, 1.0]` | No | Dimensionless leakance scale. Leakance only. | + +### `params.attribute_minimums:` sub-block + +Physical floor applied during routing for numerical stability. + +| Key | Default | Units | +|---|---|---| +| `discharge` | `1.0e-4` | m³/s | +| `slope` | `1.0e-3` | m/m | +| `velocity` | `0.01` | m/s | +| `depth` | `0.01` | m | +| `bottom_width` | `0.01` | m | + +--- + +## Guards enforced at `Config::from_yaml_file` (load-time errors) + +| Guard | Trigger | Error message substring | +|---|---|---| +| Mode/workflow conflict | `mode: training` + `workflow: eval`, or `mode: testing` + `workflow: train` | `"conflicting top-level keys"` | +| Partial adjacency | Only one of `conus_adjacency` / `gages_adjacency` set | `"gages_adjacency\` is missing"` or `"conus_adjacency\` is missing"` | +| No adjacency sources | Neither adjacency zarrs nor `geospatial_fabric` | `"adjacency sources are missing"` | +| Fabric layer on non-gpkg | `geospatial_fabric_layer` set with `.shp`/`.dbf` fabric | `"geospatial_fabric_layer"` and `".gpkg"` | +| Leakance + CUDA graphs | `use_leakance: true` and `use_cuda_graphs: true` | `"use_leakance"` and `"use_cuda_graphs"` | + +--- + +## Production configs vs experimental configs + +| Config file | Status | Key differences from production | +|---|---|---| +| `config/merit_training.yaml` | Production template | `grid:50`, `k:2`, `use_cuda_graphs:true`, managed adjacency via `geospatial_fabric`, no leakance, no disaggregation, L1 loss | +| `config/experiments/leakance_hourly_on.yaml` | Experimental (2026-07-01, GO-marginal) | `use_leakance:true`, `use_cuda_graphs:false`, `K_D`/`d_gw`/`leakance_factor` in head, precip disaggregation enabled, explicit zarr adjacency paths | +| `config/experiments/leakance_daily_on.yaml` | Experimental (2026-07-01) | Same as `leakance_hourly_on.yaml` but NO disaggregation block, NO `aorc_precip` | +| `config/sources/conus.yaml` | Source group | CONUS workstation paths without AORC precip | +| `config/sources/conus-hourly.yaml` | Source group | CONUS + `aorc_precip: /mnt/ssd1/data/aorc/merit_unit_catchments.zarr` | +| `config/sources/global.yaml` | Source group | GPFS global paths | + +**Source groups** are text-spliced into `data_sources:` by `ddrs sources use `. They do not set `kan_head` or `params`. + +--- + +## Critical runtime traps + +### STALE-BINARY TRAP +`cargo build` does NOT update `~/.cargo/bin/ddrs`. After ANY change to `src/`, run: +```bash +cargo install --path . +# or faster if target/release is current: +cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs +``` +Self-check: current checkpoints are **directories** (`.ddrs/runs//checkpoints/epoch_E_mb_M/head.mpk`). Flat `.mpk` files mean a stale binary ran. + +### CUDA graphs mask NaN +`use_cuda_graphs: true` captures a finite forward pass graph. If a NaN appears in a subsequent forward (different inputs), the graph replays stale finite values — you get a finite loss with no error. To validate forwards, test with `use_cuda_graphs: false`. This is why `use_leakance: true` + `use_cuda_graphs: true` is rejected at config load time. + +### Leakance status (as of 2026-07-05) +The leakance 2×2 (forcing × leakance) returned a **GO-marginal** verdict: +- `|zeta| > 0.01 m³/s` on 10.4% of 64,892 eval reaches (meets ≥10% bar) +- Leakance helps under hourly forcing on the losing-stream subset (ΔNSE +0.0005, ΔKGE +0.0018, 55.5% of gauges improve) +- Leakance hurts under daily forcing (ΔNSE −0.0017, ΔKGE −0.0009, 35.6%) +- `K_D` pinned at ceiling `1e-6` (binding constraint; widening NOT recommended as of 2026-07-05) + +Diagnosis hypotheses (2026-07-02): +- H2 (head throttling): SUPPORTED +- H4 (gauge bias): SUPPORTED +- H5 (equifinality with `n`): SUPPORTED +- H1 (K_D box too narrow): REFUTED +- H3 (KAN capacity): REFUTED +- H6, H7: REFUTED + +**Leakance identifiability is NOT proven.** The positive-control synthetic recovery experiment (2026-07-04) FAILED: recovery ratio 0.009 vs ≥0.5 bar. Root cause: windowed training objective has ~130x hotstart-transient noise floor. Phase B (state-cache hotstart, ≤0.25 mean L1 noise floor target) is required before any identifiability claim. + +--- + +## How to add a new routing parameter (checklist) + +A "routing parameter" is a per-reach scalar the KAN head predicts and the MC solver consumes. Example: adding a new parameter `my_param`. + +- [ ] **1. Add to `ParameterRanges` struct** (`src/config.rs`): + ```rust + pub my_param: [f32; 2], + ``` + Add a default in `ParameterRanges::default()`. + +- [ ] **2. Add YAML key parsing** in `From for Params` (`src/config.rs`): + ```rust + if let Some(v) = r.parameter_ranges.get("my_param") { + p.parameter_ranges.my_param = *v; + } + ``` + +- [ ] **3. Add to `config/merit_training.yaml`** under `params.parameter_ranges:` (if it has a production-relevant range). + +- [ ] **4. Wire into routing** (`src/routing/mmc.rs` or a new module): consume `Params.parameter_ranges.my_param` in `setup_inputs` or `route_timestep`. Follow the `denormalize` pattern in `src/routing/utils.rs`. + +- [ ] **5. Add to `kan_head.learnable_parameters:`** in any experiment config that uses it. + +- [ ] **6. Add to `params.log_space_parameters:`** if the range spans decades. + +- [ ] **7. If log-space:** add to `log_space_parameters` list in `params.log_space_parameters` in YAML and ensure `denormalize` handles it. + +- [ ] **8. Write a gradient-exactness test** if the new parameter enters a custom backward op. Run: + ```bash + cargo test --test + cargo run --release --example compare_ddr_sandbox + ``` + The sandbox must still report `ABSOLUTE MATCH`. + +--- + +## How to add a new training-mode boolean flag (checklist) + +Example: adding `use_my_feature: bool` under `params:`. + +- [ ] **1. Add field to `Params` struct** (`src/config.rs`): + ```rust + pub use_my_feature: bool, + ``` + Add `use_my_feature: false` to `Params::default()`. + +- [ ] **2. Add to `ParamsRaw`** and parse in `From for Params`: + ```rust + // in ParamsRaw: + use_my_feature: Option, + // in From impl: + if let Some(b) = r.use_my_feature { p.use_my_feature = b; } + ``` + +- [ ] **3. Add validation if needed** (`validate_*` functions in `src/config.rs`). Call from `from_yaml_file_with_mode`. Validation errors must include the YAML key name and the reason. + +- [ ] **4. Add a test** in the `#[cfg(test)]` block at the bottom of `src/config.rs` covering: flag defaults to false, flag parses true, any guard is rejected. + +- [ ] **5. Thread through call sites**: training bootstrap (`src/training/bootstrap.rs`), eval (`src/cli/eval.rs`), and any other entrypoints that construct `MuskingumCunge` or read `Params`. + +- [ ] **6. Document in `config/merit_training.yaml`** as a commented-out key with explanation if it has production relevance. + +--- + +## Quick reference: minimal leakance-on config diff + +Starting from `config/merit_training.yaml`, three changes activate leakance: + +```yaml +# 1. Under params: +params: + use_leakance: true + use_cuda_graphs: false # REQUIRED — leakance + cuda_graphs is rejected at load time + parameter_ranges: + K_D: [1.0e-8, 1.0e-6] + d_gw: [-2.0, 2.0] + leakance_factor: [0.0, 1.0] + log_space_parameters: + - p_spatial + - K_D # ADD — K_D range spans decades + +# 2. Under kan_head.learnable_parameters: + learnable_parameters: + - n + - q_spatial + - p_spatial + - K_D + - d_gw + - leakance_factor +``` + +See `config/experiments/leakance_hourly_on.yaml` for the full working example. + +--- + +## Provenance and maintenance + +Ground truth files read to produce this skill (verify before re-editing): +```bash +# Config struct (all fields, defaults, guards): +grep -n "pub use_" /home/tbindas/projects/ddrs/src/config.rs + +# Production defaults verified from: +grep -n "fn default" /home/tbindas/projects/ddrs/src/config.rs + +# Production YAML (single source of truth for hyperparameter values): +cat /home/tbindas/projects/ddrs/config/merit_training.yaml + +# Leakance experiment config: +cat /home/tbindas/projects/ddrs/config/experiments/leakance_hourly_on.yaml + +# Load-time guard tests (exhaustive): +grep -n "#\[test\]" /home/tbindas/projects/ddrs/src/config.rs | head -40 +``` + +Config struct location: `/home/tbindas/projects/ddrs/src/config.rs` +Production template: `/home/tbindas/projects/ddrs/config/merit_training.yaml` +Experiment configs: `/home/tbindas/projects/ddrs/config/experiments/` +Source group configs: `/home/tbindas/projects/ddrs/config/sources/` + +Volatile facts date-stamped: 2026-07-05. Re-verify leakance status, CONUS metric baselines, and K_D ceiling diagnosis before citing in new experiments. diff --git a/.claude/skills/ddrs-debugging-playbook/SKILL.md b/.claude/skills/ddrs-debugging-playbook/SKILL.md new file mode 100644 index 0000000..20890dc --- /dev/null +++ b/.claude/skills/ddrs-debugging-playbook/SKILL.md @@ -0,0 +1,543 @@ +--- +name: ddrs-debugging-playbook +description: "Use when a ddrs run produces wrong results, silent failures, metric regressions, NaN loss, stale checkpoints, V1 mismatch, leakance anomalies, KAN head divergence, adjacency errors, data-source alignment issues, or any symptom that costs debug time. Also use before attributing a result to a code bug — many apparent bugs are operator error (stale binary, wrong fixture, config contradiction)." +--- + +# ddrs debugging playbook + +**Audience:** Sonnet-class AI or mid-level ML engineer who knows PyTorch but not Rust/BURN. +**Voice:** imperative runbook. Copy-paste every command. Verify before claiming. + +--- + +## Glossary (terms used throughout) + +| Term | Meaning | +|---|---| +| **ddrs** | BURN-based Rust port of DDR (Python/PyTorch Muskingum-Cunge routing solver) | +| **DDR** | Python reference: `~/projects/ddr/`. The gold standard for numerical parity | +| **BURN** | Rust deep-learning framework (version 0.21 in this project) | +| **MC solver** | Muskingum-Cunge routing: converts upstream + lateral inflow to routed discharge per reach per timestep | +| **V1 / ABSOLUTE MATCH** | Regression gate: `compare_ddr_sandbox` max abs diff < 1e-3 m³/s vs DDR | +| **KAN head** | The neural network head (`rskan::KanLayer` v0.1.3): maps catchment attributes → routing parameters | +| **f32 invariant** | All tensors in the routing core must stay float32; f64/bf16 casts break DDR parity | +| **lower-triangular adjacency** | The CSR sparse pattern has `rows[k] >= cols[k]`; the forward-sub solver requires this | +| **sparse backward** | Hand-written O(nnz) `CsrSolveOp: Backward` in `src/sparse.rs`; must not be replaced by autograd unrolling | +| **leakance** | Experimental GW–SW water-loss term (`src/routing/leakance.rs`); off by default | +| **zeta** | The per-reach per-timestep leakance flux (m³/s): `zeta = leakance_factor · area_z · K_D · (depth − d_gw)` | +| **summed-Q baseline** | No-routing reference: per-gauge sum of upstream divide Qr. CONUS: median NSE 0.689 / KGE 0.723 (as of 2026-07-05) | +| **CUDA Graphs** | CUDA kernel-replay optimization (`use_cuda_graphs: true`); incompatible with leakance; masks NaN loss | +| **Q'** | Lateral inflow (m³/s) from an upstream forcing model (DHBV, LSTM, etc.) | +| **worktree** | Git worktree at a branch tip — used for experimental campaigns without touching main tree | + +--- + +## When NOT to use this skill + +- You want to understand the routing math or architecture → read `.claude/ARCHITECTURE.md` and `.claude/references/ddrs-algorithm.md` +- You want to port or verify a new feature against DDR → use skill `ddrs-comparing-to-ddr` (`.claude/references/ddrs-comparing-to-ddr.md`) +- You want to set up a new experiment from scratch → read `CLAUDE.md` §"ddrs CLI" +- You are doing leakance identifiability research → see `docs/2026-07-02-leakance-diagnosis-findings.md` for the completed hypothesis battery + +--- + +## Part 1 — Symptom → triage table + +Scan this table first. Each row points to a Part 2 entry with the full story and fix. + +| Symptom | Most likely trap | Go to | +|---|---|---| +| Two runs that differ in config produce byte-identical predictions | Stale installed binary | [T1](#t1-stale-binary-trap) | +| Checkpoint files are flat `.mpk` (not a directory) | Stale binary | [T1](#t1-stale-binary-trap) | +| `manifest.json` shows current git SHA but behavior looks old | Stale binary (SHA stamps from `.git` at runtime, not the binary) | [T1](#t1-stale-binary-trap) | +| `ddrs run` silently ignores `disaggregation:` block | Stale binary (pre-disagg binary ignores unknown serde fields) | [T1](#t1-stale-binary-trap) | +| `compare_ddr_sandbox` reports diff > 1e-3 m³/s | V1 regression | [T2](#t2-v1-regression) | +| `compare_ddr_sandbox` fails after regenerating fixtures | Wrong DDR reference tree | [T2](#t2-v1-regression) | +| Loss goes NaN, but only with `use_cuda_graphs: true` | CUDA Graphs mask NaN | [T3](#t3-cuda-graphs-mask-nan) | +| Loss is finite but suspiciously constant across steps | CUDA Graphs returning stale capture | [T3](#t3-cuda-graphs-mask-nan) | +| Config parse error: "`use_leakance: true` requires `use_cuda_graphs: false`" | Config contradiction (intentional rejection) | [T4](#t4-leakance-config-contradictions) | +| Leakance run still uses CUDA graphs silently | Missing `use_cuda_graphs: false` in config | [T4](#t4-leakance-config-contradictions) | +| K_D pinned at ceiling (100% of reaches) | K_D box is binding — or head throttling (H2); see diagnosis | [T5](#t5-leakance-parameter-collapse-or-ceiling) | +| K_D collapsed to floor (sub-1e-8) | Replicates DDR's original revert failure; check forcing resolution | [T5](#t5-leakance-parameter-collapse-or-ceiling) | +| Gradient check fails on leakance op | Regression in `src/routing/leakance.rs` backward | [T6](#t6-leakance-gradient-correctness) | +| `zeta_accum` test fails | Accumulated zeta not matching headwater identity | [T6](#t6-leakance-gradient-correctness) | +| KAN head shape or init diverges from DDR | rskan version bump or inter-block ReLU accidentally re-added | [T7](#t7-kan-head-divergence) | +| KAN parity tests fail after `Cargo.toml` rskan bump | Fixture needs regeneration | [T7](#t7-kan-head-divergence) | +| Adjacency test fails or topological ordering wrong | `rows[k] < cols[k]` somewhere; lower-triangular invariant violated | [T8](#t8-adjacency-invariant) | +| `ddrs plan` hangs or errors on adjacency build | Bad fabric path or multi-layer gpkg needs `geospatial_fabric_layer` | [T8](#t8-adjacency-invariant) | +| Training NSE well below summed-Q baseline | Routing not earning its keep; loss or gradient issue | [T9](#t9-metric-regression-below-baseline) | +| KGE lower than baseline in every config | Expected — L1 loss penalizes variance; this is known behavior | [T9](#t9-metric-regression-below-baseline) | +| Hourly run produces same predictions as daily run | Stale binary (pre-disagg) or `aorc_precip` source missing | [T1](#t1-stale-binary-trap), [T10](#t10-disaggregation-no-op) | +| `MeritGagesDataset::open` errors with `use_precip: true` | `aorc_precip` source not configured | [T10](#t10-disaggregation-no-op) | +| Checkpoint resume trains zero batches | `experiment.epochs` not raised above checkpoint epoch | [T11](#t11-checkpoint-resume-issues) | +| Resumed run drifts from uninterrupted run | Expected: weights stored as f16 (CompactRecorder) | [T11](#t11-checkpoint-resume-issues) | +| `ddrs run --strict` exits with code 4 | Source fingerprint drift vs `.ddrs/sources.lock` | [T12](#t12-source-lock-drift) | +| Recoverability / identifiability experiment fails | Hotstart transient noise floor issue (Phase B not yet met) | [T13](#t13-leakance-identifiability-status) | + +--- + +## Part 2 — Trap stories and fixes + +### T1: Stale binary trap + +**Story (2026-07-01).** The leakance × hourly 2×2 experiment produced two runs that were byte-identical despite different configs (one with hourly disaggregation, one without). The `manifest.json` showed the current git SHA `2cdd341` — which made it look like a code bug. Root cause: `~/.cargo/bin/ddrs` had mtime 2026-06-03, predating both the disaggregation feature (June 19) and leakance (June 29). The installed binary silently ignored the `disaggregation:` config block (serde ignores unknown fields) and wrote flat `.mpk` checkpoints instead of the current directory format. + +**Discriminating test.** Check checkpoint format: +```bash +# Current binaries write DIRECTORIES: +ls .ddrs/runs//checkpoints/ +# → epoch_5_mb_9/ (directory = current binary) +# → epoch_5_mb_35.mpk (flat file = stale binary) +``` + +Check binary age: +```bash +stat ~/.cargo/bin/ddrs | grep Modify +# Should be >= your last src/ change date +``` + +**Fix (canonical):** +```bash +cargo install --path . +# OR faster if target/release/ is already built: +cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs +# OR bypass the installed binary entirely: +cargo run --release --bin ddrs -- run --workflow train-and-test +``` + +**Rule:** `cargo build` does NOT update `~/.cargo/bin/ddrs`. Re-install after every `src/` change before invoking `ddrs` by name. + +**Head size cross-check (CONUS, as of 2026-07-05):** +- No disagg, no leakance: ~103,459 B +- Disagg only: ~107,178 B +- Disagg + leakance (3 extra output cols): ~107,320 B + +--- + +### T2: V1 regression + +**What V1 is.** `examples/compare_ddr_sandbox` replays DDR's 5-reach RAPID2 sandbox through ddrs's MC solver. The threshold is `max abs diff < 1e-3 m³/s`. A passing run prints: +``` +verdict: ABSOLUTE MATCH (max abs < 1e-3 m³/s) +``` +Typical passing value is ~1.5e-5 m³/s — two orders of magnitude under the threshold. + +**Run it:** +```bash +mkdir -p output # required — the example does not mkdir -p +cargo run --release --example compare_ddr_sandbox +# Also test the CUDA + graph-capture path: +DDRS_FORCE_GRAPHS=1 cargo run --release --example compare_ddr_sandbox +``` + +**Triage checklist when V1 fails:** + +1. Inspect `output/ddrs_vs_ddr.csv` — which reaches are worst? Single bad reach = geometry/parameter bug. Global failure = solver or kernel issue. + +2. Check if fixtures are stale: + ```bash + cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/export_ddr_sandbox.py + cd ~/projects/ddrs && git diff fixtures/sandbox/ + ``` + **WARNING:** Only the local `~/projects/ddr` checkout is valid. The unpushed `geometry/trapezoidal.py` work is not in any public DDR commit. Regenerating from a clean DDR clone produces ~0.55 m³/s divergence at every ddrs commit — that is a wrong-reference artifact, NOT a port bug (as of 2026-06-06). + +3. Audit recent changes to the only paths that affect V1: + ```bash + git log -p -- src/routing/ src/geometry.rs src/sparse/ + ``` + +4. Look for precision leaks: + ```bash + grep -rn "f64\|bf16\|cast\|to_dtype" src/routing/ src/geometry.rs src/sparse/ + ``` + Any cast away from f32 in these paths breaks DDR parity. + +5. Cross-check with gradcheck: + ```bash + cargo test --test sparse_gradcheck + ``` + If gradcheck fails too, the algorithm changed. If only V1 fails, it's a kernel-ordering or arithmetic-fusion difference. + +**The threshold is non-negotiable.** Never relax `1e-3 m³/s`. Never declare "good enough." + +--- + +### T3: CUDA Graphs mask NaN + +**What happens.** When a forward pass produces NaN and `use_cuda_graphs: true`, the CUDA graph replays the stale pre-NaN capture instead of the live computation. The loss appears finite. You get silently wrong results with no error. + +**Confirmed behavior (as of 2026-07-05):** `use_cuda_graphs: true` returns stale finite loss on a NaN forward. + +**Discriminating test:** +```bash +# Reproduce the stale-loss symptom: +# Run once with cuda_graphs on vs off, inject a NaN input, compare loss values. +# If cuda_graphs=true gives finite loss and cuda_graphs=false gives NaN → confirmed. +``` + +**Fix:** Disable CUDA graphs when debugging any NaN or suspiciously-constant loss: +```yaml +# in ddrs.yaml or experiment config: +params: + use_cuda_graphs: false +``` + +**Rule:** Always debug loss anomalies with `use_cuda_graphs: false`. Re-enable only after confirming the forward is NaN-free. + +--- + +### T4: Leakance config contradictions + +**What happens.** Two config errors involving leakance are caught at load time. + +**Error 1 — leakance + CUDA graphs:** +``` +params: `use_leakance: true` requires `use_cuda_graphs: false` +``` +CUDA Graphs cannot capture the extra leakance kernel without a separate capture path. This is an intentional hard rejection in `src/config.rs:626`. + +**Fix:** +```yaml +params: + use_leakance: true + use_cuda_graphs: false # REQUIRED when use_leakance is true +``` + +**Error 2 — leakance parameters missing from KAN head.** +If `params.use_leakance: true` but `K_D`, `d_gw`, `leakance_factor` are not in `kan_head.learnable_parameters`, the head emits no leakance parameters and the routing silently has no exchange. No error is thrown — the leakance term gets a zero or garbage input. + +**Complete leakance config checklist:** +```yaml +params: + use_leakance: true + use_cuda_graphs: false + parameter_ranges: + K_D: [1.0e-8, 1.0e-6] # log-space; hydraulic exchange rate 1/s + d_gw: [-2.0, 2.0] # groundwater depth offset, m + leakance_factor: [0.0, 1.0] # dimensionless scale + +kan_head: + learnable_parameters: + - K_D + - d_gw + - leakance_factor + # ... plus your routing params (n, q_spatial, etc.) +``` + +--- + +### T5: Leakance parameter collapse or ceiling + +**Two failure modes:** + +**Mode A — K_D at ceiling (100% of reaches).** +Observed in both leakance-ON arms of the 2026-07-01 2×2 (hourly and daily). +- `K_D` median log10 = −5.999, IQR = 3.6e-4 (essentially a delta function at the `1e-6` upper bound). +- This is NOT the K_D box clipping the flux. Diagnosis (2026-07-02) showed median in-box utilization is only 3.4% — the optimizer maxes the rate constant and then throttles the product via the driving head (`d_gw` learned near typical depths, so `depth − d_gw ≈ 0`). +- **K_D widening is NOT recommended** — the Phase-3 gate failed because H1 (structural ceiling) was REFUTED. Root cause: H2 (head throttling) + H4 (gauge bias) + H5 (equifinality under daily forcing). + +**Mode B — K_D at floor (collapse to sub-1e-8).** +This replicates DDR's original revert failure (sub-0.01 m³/s exchange, physically negligible). If you see this, check: +- Is daily forcing being used? Under flat-daily forcing the depth dynamic range is too small for `zeta ∝ (depth − d_gw)` to be identifiable. +- Is hourly disaggregation actually running? Verify the binary is current (T1) and `aorc_precip` source is configured (T10). + +**Discriminating check after any leakance run:** +```bash +cargo build --release --bin dump_parameters +target/release/dump_parameters \ + --config \ + --checkpoint .ddrs/runs//checkpoints/epoch_E_mb_M/head \ + --output /tmp/kp.nc 2>&1 | grep -E "K_D|leakance_factor|d_gw|frac@" +``` +Expected for a non-collapsed run: `K_D` interior or at ceiling (not floor), `leakance_factor` interior (0.1–0.5), `d_gw` spatially varying. + +--- + +### T6: Leakance gradient correctness + +**Guard tests — run all four after any change to `src/routing/leakance.rs` or `mmc_op.rs`:** + +```bash +cargo test --test leakance_gradcheck # analytical ≈ finite-difference (8 params) +cargo test --test leakance_off_parity # byte-identical to no-leakance when off (3 tests) +cargo test --test zeta_accum # accumulated zeta == headwater identity +cargo run --release --example compare_ddr_sandbox # V1 must still pass +``` + +**If `leakance_gradcheck` fails:** The analytical backward in `TimestepLeakanceOp: Backward` is wrong. Compare against `src/routing/leakance.rs` math: `zeta = leakance_factor · area_z · K_D · (depth − d_gw)` where `area_z = (p · depth)^q_eps · length`. All partial derivatives are straightforward products/chains; check each of the 8 inputs. + +**If `zeta_accum` fails:** The accumulator in `evaluate` is not recomputing from the same primitives the backward used. The test verifies the headwater identity `q_no_leak[0] − q_leak[0] == zeta[0]`. + +**If `leakance_off_parity` fails:** The leakance gating is broken — the `None` path is not byte-identical to a run without leakance compiled in. + +--- + +### T7: KAN head divergence + +**Architecture (must not change without explicit intent):** +``` +Linear(F, H) → KanLayer(H, H) × num_hidden_layers → Linear(H, P) → Sigmoid +``` +- No inter-block ReLU. DDR's `kan.py` has none; adding one breaks parity. +- All `num_hidden_layers` inner KanLayers get the SAME initialization seed (DDR `kan.py:24-34` quirk — preserved for parity). +- rskan version: `v0.1.3` (as of 2026-07-05). Pinned in `Cargo.toml:27`. + +**Parity test suite:** +```bash +cargo test --features fixtures \ + --test kan_head_init_repro \ + --test kan_head_init_parity \ + --test kan_head_fixture_forward \ + --test kan_head_fixture_backward +``` + +**If tests fail after an rskan bump:** The fixtures need regeneration: +```bash +cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/dump_kan_weights.py +cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/dump_kan_forward.py +``` +Then re-run the test suite. If it still fails, the rskan API changed in a parity-breaking way — audit the diff and decide whether to update the DDR reference or roll back the bump. + +**If tests fail without an rskan bump:** Check for accidental re-introduction of inter-block ReLU in `src/nn/kan_head.rs`. + +--- + +### T8: Adjacency invariant + +**The invariant:** The CSR adjacency pattern must be lower-triangular: every nonzero at `(row, col)` must have `row >= col`. The forward-substitution solver reads rows in order and assumes all upstream contributions are already resolved. + +**Test:** +```bash +cargo test data_zarr_store::conus_adjacency_loads_real_merit_zarr +# Also: +cargo test --test adjacency_parity +``` + +**If adjacency build fails during `ddrs plan`:** +- Check the fabric path exists and is readable. +- For `.gpkg` files with multiple layers, set `geospatial_fabric_layer:` in config. +- The builder reads only the attribute table (`.dbf` or gpkg attributes) — never the geometry. Check `geospatial_fabric` points to the right file. +- On a fresh worktree, `.ddrs/adjacency/` does not exist — `ddrs plan` builds it on first run (~10 s for CONUS `.dbf`). + +**If topological ordering is wrong:** The managed builder replicates petgraph's deterministic DFS finish-time order. Check `src/adjacency/build.rs::topological_sort` against the engine's version. + +--- + +### T9: Metric regression below baseline + +**Baseline numbers (CONUS, as of 2026-07-05):** +| config | median NSE | median KGE | gauges | +|---|---|---|---| +| Summed-Q′ baseline, same-run (2,365-gauge eval set) | 0.678 | 0.717 | 2365 | +| Best result: precip-disagg + L1 | 0.715 | 0.711 | 2365 | + +**Critical known behavior:** KGE does NOT beat the summed-Q baseline in any trained config as of 2026-07-05. NSE beats it (+0.037 with precip disagg). The KGE regression is structural: the L1 loss maximizes at simulated variance below observed (α < 1), rewarding over-attenuation. The whole KGE drop is in the `α = σ_sim/σ_obs` term. + +**Triage if NSE is far below baseline:** +1. Is the binary current? (T1) +2. Is the loss descending? Check `run.log` for epoch-mean L1. +3. Is CUDA graphs masking NaN? (T3 — disable and re-check) +4. Are gauge batch sizes reasonable? Too few gauges per batch → noisy gradient. +5. Is the data source correct? `streamflow resolution: Daily|Hourly` is logged at dataset open — verify it. + +**To improve KGE above baseline:** Switch to `experiment.loss.kind: nnse-kge`. The `(α-1)²` term in KGE provides the restoring gradient. This requires explicit config: +```yaml +experiment: + loss: + kind: nnse-kge + nnse_weight: 1.0 + kge_weight: 1.0 +``` + +--- + +### T10: Disaggregation no-op + +**Symptoms:** +- Hourly and daily runs produce byte-identical predictions. +- Head file size is ~103,459 B (no-disagg size) even when config has `disaggregation: ...`. + +**Root causes (in priority order):** + +1. **Stale binary** — most likely. See T1. The pre-disagg binary silently ignores the `disaggregation:` block. + +2. **Missing `aorc_precip` source.** The AORC precip zarr at `/mnt/ssd1/data/aorc/merit_unit_catchments.zarr` must be in `data_sources:`. Without it, `MeritGagesDataset::open` errors when `use_precip: true`. Check: + ```bash + ddrs sources list # '*' marks active group + # conus-hourly group includes aorc_precip + ddrs sources use conus-hourly + ``` + +3. **`use_precip: false` in config.** The `aorc_precip` source must be present AND `kan_head.disaggregation.use_precip: true` must be set. The source group (`conus-hourly`) splices in the source; the disagg block must be in the experiment config separately. + +**Verification after a fix:** +```bash +# Binary check: head file should be ~107,320 B (disagg + leakance) or ~107,178 B (disagg only) +ls -la .ddrs/runs//checkpoints/epoch_5_mb_9/head.mpk + +# Dataset log at run start: +grep "AORC precip store" .ddrs/runs//run.log +# Should show: "AORC precip store: 290878 catchments" + +# Forcing verification: eval predictions should differ between hourly and daily runs: +md5sum .ddrs/runs//eval/predictions.zarr/predictions/0.0 +md5sum .ddrs/runs//eval/predictions.zarr/predictions/0.0 +# These must NOT be identical +``` + +--- + +### T11: Checkpoint resume issues + +**Resume requires three files in a directory:** +``` +.ddrs/runs//checkpoints/epoch_E_mb_M/ + head.mpk # KAN weights (f16, CompactRecorder) + optim.mpk # Adam moments (f16) + state.json # epoch, next mini-batch, rng state, sampler permutation + cursor +``` + +**Resume trains zero batches:** `experiment.epochs` is at or below the checkpoint epoch. Fix: raise `experiment.epochs` past `E` in `ddrs.yaml`. + +**Resumed trajectory drifts from uninterrupted run:** Expected. Weights and moments are stored as f16 (`CompactRecorder = HalfPrecisionSettings`). The resumed trajectory is numerically valid but will not be bit-identical to an uninterrupted run. + +**`dump_parameters --checkpoint` path gotcha:** Pass the HEAD BASE, not the directory: +```bash +# CORRECT (head base — CompactRecorder appends .mpk): +target/release/dump_parameters --checkpoint .ddrs/runs//checkpoints/epoch_5_mb_9/head ... +# WRONG (directory — will fail to find head.mpk): +target/release/dump_parameters --checkpoint .ddrs/runs//checkpoints/epoch_5_mb_9 ... +``` + +--- + +### T12: Source lock drift + +**What happens.** `ddrs run --strict` exits with code 4 when the data-source fingerprints in `.ddrs/sources.lock` differ from the current `ddrs.yaml`. This preserves evidence; re-locking would overwrite it. + +**Fix (normal):** +```bash +ddrs plan # re-locks sources.lock to match current ddrs.yaml +ddrs run --workflow +``` + +**Fix (investigate first):** +```bash +cat .ddrs/sources.lock # shows last-locked fingerprints +# Compare against current data_sources: paths in ddrs.yaml +# If a path moved or a store was updated, decide whether to re-plan or roll back +``` + +--- + +### T13: Leakance identifiability status (as of 2026-07-05) + +**This section describes an active research limitation — not a bug to fix.** + +**Positive control experiment (2026-07-04, worktree):** A synthetic recoverability test was run to check whether the gradient path can recover a known planted zeta through gauged-only observations. The experiment FAILED: recovery ratio 0.009 vs the >=0.5 bar. Root cause: the windowed training objective has a hotstart-transient noise floor approximately 130× larger than the leakance signal. + +**Implication:** Leakance identifiability is NOT proven. The 2×2 GO-marginal verdict (leakance helps skill on the losing-stream subset under hourly forcing) stands, but the mechanism cannot be confirmed as genuine GW–SW exchange recovery until Phase B is complete. + +**Phase B objective (NOT YET MET as of 2026-07-05):** noise floor <= 0.25 mean L1 (i.e., <= 10% of a converged run's loss). Requires a state-cache hotstart to eliminate the transient. Do not make identifiability claims until Phase B passes. + +**Gradient probe results (2026-07-03, worktree):** +- P1 (gradient starvation to leakance params): REFUTED +- P3 (detectability — signal vs 5% obs band): NO-GO, signal 53× smaller than detectability threshold + +**Summary of leakance diagnosis verdicts (as of 2026-07-02):** + +| Hypothesis | Verdict | Key evidence | +|---|---|---| +| H1: K_D box clips zeta | REFUTED | Median utilization 3.4%; 71.5% of reaches CAN exceed 0.01 m³/s in-box | +| H2: Driving-head starvation | SUPPORTED | Median head 0.021 m; 47% of reaches gaining at eval-window mean | +| H3: KAN variance collapse | REFUTED | Max Spearman(param, attribute) = 0.71 (strong spatial structure) | +| H4: Gauge bias / gradient starvation | SUPPORTED | zeta–uparea ρ +0.76; gauged 11× ungauged median zeta; dry/wet ratio inverted | +| H5: Equifinality with routing params | SUPPORTED (daily only) | Daily Δn = +0.012 (0.59 IQR); hourly Δn nil | +| H6: Wrong yardstick (absolute bar) | REFUTED | Fractional loss agrees: 8.4% lose >1% of local flow | +| H7: Model-form error (d_gw bounds) | REFUTED | 0.0% of d_gw at bounds in any aridity tercile | + +**Do not run K_D widening.** The Phase-3 gate for K_D widening FAILED because H1 was REFUTED. The constraint is the signal, not the box. + +--- + +## Part 3 — Pre-flight checklist before any training run + +Use this before starting a new experiment to prevent the most common traps: + +- [ ] `stat ~/.cargo/bin/ddrs` — mtime is newer than your last `src/` change +- [ ] `ddrs sources list` — active group (`*`) matches intended dataset +- [ ] `ddrs plan` — no source drift warnings; `mode:` and `workflow:` agree +- [ ] Config leakance consistency: if `use_leakance: true`, confirm `use_cuda_graphs: false` and all three params in `kan_head.learnable_parameters` +- [ ] If hourly disagg: config has `aorc_precip:` source AND `kan_head.disaggregation.use_precip: true` +- [ ] If resuming: `experiment.epochs` is greater than the checkpoint epoch; checkpoint path ends at `head` base (not the directory) +- [ ] If touching `src/routing/`, `src/geometry.rs`, or `src/sparse/`: run `cargo run --release --example compare_ddr_sandbox` and confirm ABSOLUTE MATCH + +--- + +## Part 4 — Quick-reference test commands + +```bash +# V1 regression gate (routing core, geometry, sparse): +mkdir -p output && cargo run --release --example compare_ddr_sandbox + +# V1 on CUDA + graph-capture path: +DDRS_FORCE_GRAPHS=1 cargo run --release --example compare_ddr_sandbox + +# Leakance gradient correctness: +cargo test --test leakance_gradcheck +cargo test --test leakance_off_parity +cargo test --test zeta_accum + +# Sparse backward correctness: +cargo test --test sparse_gradcheck + +# KAN head parity vs DDR: +cargo test --features fixtures \ + --test kan_head_init_repro \ + --test kan_head_init_parity \ + --test kan_head_fixture_forward \ + --test kan_head_fixture_backward + +# Adjacency ordering and builder parity: +cargo test data_zarr_store::conus_adjacency_loads_real_merit_zarr +cargo test --test adjacency_parity + +# All lib unit tests: +cargo test --lib + +# Full test suite: +cargo test +``` + +--- + +## Provenance and maintenance + +Ground truth for this skill (re-read these files to verify facts remain current): + +```bash +# V1 / comparing-to-DDR reference: +cat /home/tbindas/projects/ddrs/.claude/references/ddrs-comparing-to-ddr.md + +# Stale-binary trap story + leakance 2x2 re-run: +cat /home/tbindas/projects/ddrs/docs/2026-07-01-leakance-hourly-experiment-handoff.md + +# 2x2 findings (all four arms, GO verdict): +cat /home/tbindas/projects/ddrs/docs/2026-07-01-leakance-hourly-findings.md + +# Low-zeta diagnosis (H1–H7 hypothesis verdicts): +cat /home/tbindas/projects/ddrs/docs/2026-07-02-leakance-diagnosis-findings.md + +# Config rules (invariants, leakance enable, CLI lifecycle): +cat /home/tbindas/projects/ddrs/CLAUDE.md + +# CUDA graphs mask NaN (memory note): +cat /home/tbindas/projects/ddrs/.claude/memories/cuda-graphs-mask-nan.md + +# Re-verify rskan version: +grep rskan /home/tbindas/projects/ddrs/Cargo.toml + +# Re-verify config leakance rejection: +grep -n "use_leakance.*use_cuda_graphs\|use_cuda_graphs.*use_leakance" \ + /home/tbindas/projects/ddrs/src/config.rs +``` diff --git a/.claude/skills/ddrs-diagnostics-and-tooling/SKILL.md b/.claude/skills/ddrs-diagnostics-and-tooling/SKILL.md new file mode 100644 index 0000000..8640b60 --- /dev/null +++ b/.claude/skills/ddrs-diagnostics-and-tooling/SKILL.md @@ -0,0 +1,582 @@ +--- +name: ddrs-diagnostics-and-tooling +description: > + Use when you need to MEASURE a ddrs result rather than eyeball it — diagnosing + a failed run, verifying gradient health, interpreting leakance GO/NO-GO gates, + checking the summed-Q' baseline, probing identifiability, or deciding whether + a hypothesis is SUPPORTED/REFUTED/INCONCLUSIVE. Triggers: "why is zeta small", + "how do I tell if training is working", "what does K_D ceiling mean", + "compare two runs", "is my gradient alive", "how do I reproduce the 2x2 + verdict". Do NOT use for architecture changes, config authoring, or writing + new training code — use ddrs-change-control or ddrs-architecture-contract + instead. +--- + +# ddrs Diagnostics and Tooling + +## Glossary (jargon defined once) + +| Term | Definition | +|---|---| +| **ddrs** | BURN-0.21 Rust port of the DDR differentiable Muskingum-Cunge solver | +| **BURN** | Rust deep-learning framework (analogous to PyTorch); autograd tapes differ | +| **KAN head** | `rskan::KanLayer` network (`Linear→KanLayer×N→Linear→Sigmoid`); maps catchment attributes to per-reach routing parameters | +| **Q'** (Q-prime) | Upstream-summed divide streamflow forcing from a pre-computed DHBv2 retrospective | +| **zeta (ζ)** | Per-reach GW–SW exchange flux (m³/s): `leakance_factor · area_z · K_D · (depth − d_gw)`. Positive = losing reach | +| **K_D** | Hydraulic exchange rate (1/s); log-space parameter in `[1e-8, 1e-6]` by default | +| **Muskingum-Cunge (MC)** | Linear flood-routing method; ddrs solves a CSR lower-triangular system per timestep | +| **eval network** | Gauge-subgraph union used during evaluation (64,892 reaches for CONUS) | +| **NSE / KGE** | Nash-Sutcliffe Efficiency / Kling-Gupta Efficiency; standard hydrology skill scores | +| **NNSE** | Normalized NSE: `NSE/(2-NSE)`, range [0,1], avoids -∞ floor | +| **summed-Q' baseline** | Upper bound with no routing: sum of upstream Q' at each gauge; median NSE 0.689 / KGE 0.723 (CONUS, as of 2026-07-05) | +| **2×2** | Leakance ON/OFF × hourly/daily forcing factorial experiment | +| **rho-window** | Training sub-sequence length (default 90 days); sampled from the full training period | +| **hotstart transient** | Initial-condition mismatch at window start; big rivers carry memory >> warmup days | +| **CsrSolveOp** | Hand-written BURN autograd backward for the sparse triangular solve (invariant 4) | +| **dump_parameters** | Binary/CLI command that exports full-CONUS KAN outputs to `kan_parameters.nc` | +| **run-id** | `-[-]` directory name under `.ddrs/runs/` | + +--- + +## When NOT to use this skill + +- **Changing `src/routing/`, `src/sparse.rs`, or `src/geometry.rs`** — use ddrs-change-control (blast-radius analysis required) +- **KAN head architecture changes** — use ddrs-architecture-contract (invariants 5-6) +- **Writing new Python analysis scripts** — use ddrs-proof-and-analysis-toolkit +- **Interpreting the research roadmap / phase gating** — use ddrs-identifiability-campaign or ddrs-research-frontier + +--- + +## Part 1: Non-negotiable regression gates + +Run these before and after ANY change to `src/routing/`, `src/geometry.rs`, or `src/sparse.rs`. + +### 1.1 DDR parity gate (invariant 1) + +```bash +cargo run --release --example compare_ddr_sandbox +``` + +**Pass:** prints `ABSOLUTE MATCH` — max abs diff < 1e-3 m³/s on the 5-reach RAPID sandbox. +**Fail:** any diff >= 1e-3 m³/s means the port broke. Do NOT merge. + +**Caveat (as of 2026-06-06):** the reference fixture requires the desktop's `~/projects/ddr` with the unpushed `geometry/trapezoidal.py` changes. A clean DDR clone will diverge ~1%. That is a wrong-reference failure, not a port failure. See `.claude/references/ddrs-comparing-to-ddr.md` §Regenerating fixtures before concluding a real regression. + +### 1.2 Leakance gradient-exactness gates + +Run whenever `src/routing/leakance.rs` or its backward op changes: + +```bash +cargo test --test leakance_gradcheck # analytical grad ≈ finite-difference (8 cases) +cargo test --test leakance_off_parity # byte-identical to no-leakance when off (3 cases) +cargo test --test zeta_accum # accumulated zeta == b_rhs delta (6 cases) +cargo run --release --example compare_ddr_sandbox # still ABSOLUTE MATCH +``` + +**Interpretation:** `leakance_gradcheck` failing means the analytical backward diverges from finite-diff; this breaks training correctness. `leakance_off_parity` failing means leakance bleeds into non-leakance paths. + +### 1.3 KAN head parity gates + +Run when `src/nn/kan_head.rs`, `Cargo.toml` rskan pin, or DDR's `nn/kan.py` changes: + +```bash +cargo test --features fixtures \ + --test kan_head_init_repro \ + --test kan_head_init_parity \ + --test kan_head_fixture_forward \ + --test kan_head_fixture_backward +``` + +### 1.4 Sparse gradcheck + +```bash +cargo test --test sparse_gradcheck +``` + +Verifies the CSR backward (invariant 4) is gradient-exact. + +--- + +## Part 2: Stale binary trap (the most common failure mode) + +`~/.cargo/bin/ddrs` is installed once by `cargo install`. `cargo build` does NOT update it. + +**Symptom:** run looks fine, metrics make no sense, or a new feature is silently missing. + +**Check whether you have the right binary:** + +```bash +# Current checkpoints are DIRECTORIES: +ls .ddrs/runs//checkpoints/ +# Should show: epoch_5_mb_9/ (a directory) +# If you see: epoch_5_mb_9.mpk (a flat file) → stale binary +``` + +**Fix:** + +```bash +cargo install --path . # canonical, ~2 min +# or faster: +cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs +# or bypass installed binary entirely: +cargo run --release --bin ddrs -- run --workflow train-and-test +``` + +The stale-binary trap caused the 2026-07-01 2×2 to produce byte-identical hourly and daily cells (the installed binary predated the disaggregation feature). + +--- + +## Part 3: CUDA graphs masking NaN + +**Symptom:** training loss is finite and slowly decreasing, but intermediate checks show NaN activations. + +**Cause:** `use_cuda_graphs: true` replays a captured graph; a NaN in a subsequent forward returns stale (pre-NaN) finite values. The loss looks healthy while the model is broken. + +**Diagnosis:** + +```bash +# In ddrs.yaml, temporarily set: +use_cuda_graphs: false +# Then re-run one mini-batch and inspect: +# If loss is NaN → confirmed NaN forward; debug with use_cuda_graphs: false +# If loss is fine → not a NaN issue +``` + +**Note:** `use_leakance: true` combined with `use_cuda_graphs: true` is rejected at config load time with a hard error. + +--- + +## Part 4: Run inspection and workspace navigation + +### 4.1 Check run status and disk + +```bash +ddrs status # workspace summary + disk usage by run +ddrs show # full manifest: config, sources, git SHA, metrics +``` + +### 4.2 Read a run's log + +```bash +cat .ddrs/runs//run.log # timestamped stdout+stderr (fd-level capture) +``` + +Useful patterns in the log to check: + +| What to grep | Meaning | +|---|---| +| `streamflow resolution: Daily\|Hourly` | Confirms whether icechunk store was read as daily or hourly | +| `warm start: loaded KAN head` | Checkpoint resume loaded correctly | +| `no …/optim.mpk` | Adam starts cold (expected for head-only warm-start) | +| `ABSOLUTE MATCH` | Sandbox regression passed during this run | +| `precip loading` | AORC precip store opened (needed for disaggregation) | + +### 4.3 Inspect a run's config + +```bash +cat .ddrs/runs//config.yaml # exact config that produced this run +``` + +### 4.4 Compare two runs' metrics + +```bash +ddrs show | grep -E "nse|kge|loss" +ddrs show | grep -E "nse|kge|loss" +``` + +--- + +## Part 5: Summed-Q' baseline + +**What it is:** per-gauge sum of upstream Q' with NO routing or learning. It is the ceiling that trained routing must beat. + +**Reference numbers (CONUS, as of 2026-07-05):** +- Median NSE: 0.689 +- Median KGE: 0.723 + +**Best trained result (precip-driven disaggregation + L1, 2365 gauges, as of 2026-06-23):** +- Median NSE: 0.715 (+0.037 vs baseline — beats it) +- Median KGE: 0.711 (-0.012 vs baseline — does NOT beat it) + +**Critical:** KGE does NOT beat the summed-Q' baseline in any config as of 2026-07-05. NSE beats it with precip disaggregation. This is a known open problem (over-attenuation; L1 and NSE reward low variance). + +**Reproduce baseline:** + +```bash +ddrs plan # computes and caches baseline automatically on first run +# or read the cached version: +cat .ddrs/baselines//manifest.json # shows metrics, provenance +``` + +**Interpretation:** if your trained run's median NSE does NOT beat 0.689, the routing term earns nothing. Debug training loss curves and KAN head gradient stats before touching the sparse solver. + +--- + +## Part 6: dump_parameters — export learned KAN outputs to NetCDF + +```bash +# Via the legacy eval binary (required for leakance zeta export): +cargo build --release --bin eval +target/release/eval \ + --config config/experiments/leakance_hourly_on.yaml \ + --checkpoint .ddrs/runs//checkpoints/epoch_5_mb_9 \ + --output /tmp/eval.zarr \ + --zeta-output .ddrs/runs//kan_parameters.nc + +# Or (no zeta, just KAN params): +cargo build --release --bin dump_parameters +target/release/dump_parameters \ + --config ddrs.yaml \ + --checkpoint .ddrs/runs//checkpoints/epoch_5_mb_9/head \ + --output .ddrs/runs//plot/kan_parameters.nc +``` + +**Output file layout (`kan_parameters.nc`):** + +| Variable | Dimension | Unit | Notes | +|---|---|---|---| +| `COMID` | `(COMID,)` | — | Reach IDs for full-CONUS params | +| `n` | `(COMID,)` | — | Manning's roughness, denormalized | +| `q_spatial` | `(COMID,)` | — | Channel geometry exponent | +| `x_storage` | `(COMID,)` | — | Muskingum X (storage weighting) | +| `K_D` | `(COMID,)` | 1/s | Hydraulic exchange rate (leakance only) | +| `d_gw` | `(COMID,)` | m | GW depth threshold (leakance only) | +| `leakance_factor` | `(COMID,)` | — | Scale factor (leakance only) | +| `COMID_eval` | `(COMID_eval,)` | — | Eval-network reach IDs (leakance only) | +| `zeta` | `(COMID_eval,)` | m³/s | Mean \|zeta\| over eval window | +| `zeta_net` | `(COMID_eval,)` | m³/s | Signed mean; positive = losing reach | +| `depth_mean` | `(COMID_eval,)` | m | Eval-window mean routed depth | +| `area_z_mean` | `(COMID_eval,)` | m² | Eval-window mean plan-view wetted area | +| `q_mean` | `(COMID_eval,)` | m³/s | Eval-window mean routed discharge | + +**Load in Python:** + +```python +import xarray as xr +ds = xr.open_dataset(".ddrs/runs//kan_parameters.nc") +# for a quick K_D ceiling check: +kd = ds["K_D"].values +import numpy as np +print(f"K_D: min={kd.min():.2e} median={np.median(kd):.2e} max={kd.max():.2e}") +print(f"fraction at ceiling (1e-6): {(kd > 9.9e-7).mean():.1%}") +``` + +--- + +## Part 7: Leakance GO/NO-GO evaluation + +### 7.1 The three gate criteria (per spec) + +| Gate | Threshold | Interpretation | +|---|---|---| +| 1 | ΔNSE or ΔKGE > 0 (median) on losing-stream subset, hourly arm | Leakance improves skill where physics predicts it should | +| 2 | Effect absent or weaker in daily arm | Rules out fudge-factor behavior | +| 3 | \|zeta\| > 0.01 m³/s on ≥ 10% of eval reaches | Learned exchange is non-trivially active | + +**Current status (as of 2026-07-01):** GO — but marginal (10.4% vs 10% threshold for gate 3, no headroom). + +### 7.2 Running the full verdict script + +```bash +cd ~/projects/ddr +uv run python ~/projects/ddrs/scripts/leakance_subset_analysis.py \ + --hourly-on 2026-07-01T13-43-32Z-train-and-test \ + --daily-on 2026-07-01T21-20-27Z-train-and-test \ + --hourly-off 2026-06-23T02-49-12Z-conus-hourly-train-and-test \ + --daily-off 2026-06-05T01-41-16Z-train-and-test \ + --ddrs-runs-dir /home/tbindas/projects/ddrs/.ddrs/runs +``` + +**Prerequisites:** both ON arms must have `kan_parameters.nc` with the `zeta`/`COMID_eval` variables (produced by `--zeta-output` or `train-and-test` Phase 2). + +**Output block to look for:** + +``` +VERDICT: GO + Leakance improves skill on the losing-stream subset under hourly forcing ... +``` + +or `VERDICT: NO-GO` with reasons, or `VERDICT: NEEDS_ZETA_EXPORT`. + +**Losing-stream subset definition:** gauges where the summed-Q' baseline mean(pred)/mean(obs) > 1 on the hourly-OFF run. CONUS result: 1883/2365 gauges (79.6%). + +### 7.3 Interpreting leakance parameter outputs + +| Observation | Interpretation | +|---|---| +| `K_D` 100% at ceiling (1e-6) | Optimizer wants MORE exchange; box is binding. NOT a model failure (H1 REFUTED, as of 2026-07-02) | +| `leakance_factor` interior (≈0.33) | Gate is open; reaches are actively exchanging | +| `d_gw` near mean depth | Driving head throttled; ~47% of reaches gaining at eval-window mean | +| zeta–uparea Spearman +0.76 | Exchange tracks river size, not aridity — gauge bias, not starvation | +| dry-tercile zeta < wet-tercile | Inverse of physics; training signal concentrates near large gauged rivers | + +--- + +## Part 8: Seven-hypothesis diagnosis battery (leakance low-zeta) + +**When to run:** after any leakance experiment returns small zeta (median |zeta| < 0.01 m³/s). + +**Prereqs:** the ON run's `kan_parameters.nc` must contain `depth_mean`, `area_z_mean`, `q_mean` on `COMID_eval`. Requires the re-eval pass with the current binary. + +```bash +cd ~/projects/ddr +uv run python ~/projects/ddrs/scripts/leakance_diagnosis.py +# uses hardcoded run IDs in ARM_IDS dict; edit if using different runs +``` + +**Hypothesis reference table (results as of 2026-07-02):** + +| # | Hypothesis | Verdict (2026-07-02) | Key number | +|---|---|---|---| +| H1 | K_D box clips zeta below detectability | REFUTED | 71.5% of reaches CAN exceed 0.01 m³/s in-box; median utilization 3.4% | +| H2 | d_gw near depth → driving head ≈ 0 | SUPPORTED | 57.6% of reaches < 0.1 m mean driving head; 47.0% ≤ 0 | +| H3 | KAN variance collapse (original hypothesis) | REFUTED | K_D–aridity ρ = +0.61; d_gw–meanP ρ = +0.71 — strong spatial structure | +| H4 | Gauge bias / gradient starvation | SUPPORTED | gauged median |zeta| 6.7e-3 vs ungauged 5.9e-4 (11×); dry/wet ratio 0.40 (inverse of physics) | +| H5 | Equifinality with n/x_storage | SUPPORTED (daily only) | daily Δn = +0.012 (0.59 IQR); hourly Δn nil (0.05 IQR) | +| H6 | Wrong yardstick (absolute 0.01 bar) | REFUTED | 8.4% >1% fractional loss agrees with absolute bar | +| H7 | d_gw boundary pinning (disconnected regime) | REFUTED | 0.0% of d_gw at bounds | + +**Diagnosis conclusion:** zeta is small because the optimizer throttles through the driving head (H2) and the gradient only reaches gauged large rivers (H4), not because the K_D box or KAN architecture fails. Widening K_D past 1e-6 is NOT recommended (supersedes the "top follow-up" in `docs/2026-07-01-leakance-hourly-findings.md`). + +--- + +## Part 9: Gradient probe (adjoint reachability + detectability) + +**Location:** `origin/worktree-zeta-sensitivity` branch. +**When to run:** when you want to know whether the leakance gradient is alive at a reach, or whether a real-magnitude loss would be detectable at a downstream gauge. + +### 9.1 Stage 1 — adjoint reachability map + +```bash +# Trained checkpoint (use worktree binary): +WT=/home/tbindas/projects/ddrs/.claude/worktrees/zeta-sensitivity +nice -n 10 $WT/target/release/probe_zeta_gradient \ + --config config/experiments/leakance_hourly_on.yaml \ + --checkpoint .ddrs/runs/2026-07-01T13-43-32Z-train-and-test/checkpoints/epoch_5_mb_9 \ + --windows 96 --seed 42 \ + --output output/zeta_probe/grad_trained.nc + +# Cold head (omit --checkpoint): +nice -n 10 $WT/target/release/probe_zeta_gradient \ + --config config/experiments/leakance_hourly_on.yaml \ + --windows 96 --seed 42 \ + --output output/zeta_probe/grad_cold.nc +``` + +**Output:** per-reach `|∂L/∂factor|`, `∂L/∂factor`, coverage count in NetCDF. + +**Interpretation thresholds:** + +| Ratio (gauged/ungauged |g|) | Interpretation | +|---|---| +| ≥ 10× at both trained and cold points | SUPPORTED starvation — auxiliary supervision fills genuine gap | +| < 10× | REFUTED starvation — gradient reaches everywhere | + +**Measured result (2026-07-03, as of 2026-07-05):** gauged/ungauged ratio = 1.5× (trained), 2.9× (cold). P1 starvation REFUTED. The gradient is alive everywhere. + +### 9.2 Stage 2 — planted-delta detectability + +```bash +# Plan sites first (ddrs-py venv): +cd ddrs-py && uv run python ../scripts/zeta_probe_sites.py + +# Perturb pass: +nice -n 10 $WT/target/release/probe_zeta_gradient \ + --config config/experiments/leakance_hourly_on.yaml \ + --checkpoint .ddrs/runs/2026-07-01T13-43-32Z-train-and-test/checkpoints/epoch_5_mb_9 \ + --mode perturb \ + --probe-plan output/zeta_probe/probe_plan.csv \ + --eval-days 1095 \ + --output output/zeta_probe/perturb +``` + +**Detectability criterion:** `|mean ΔQ| > 99th-pct noise floor AND > 5% of gauge's mean flow`. + +**Measured result (2026-07-03, as of 2026-07-05):** + +| Delta | Reference (Ref) gauges | Non-reference | +|---|---|---| +| 0.01 m³/s (literature-magnitude) | 4.2% detectable | 0.0% | +| 0.1 m³/s (upper-literature) | 16.7% detectable | 2.1% | + +P3 detectability: NO-GO. The planted loss arrives at gauges at ~95% fidelity (transmission fine) but is 53× smaller than the median Ref gauge's 5% discharge-uncertainty band. Detection fails on dilution, not transmission. No gauge-only objective can learn real-world leakance. + +--- + +## Part 10: Synthetic recoverability control (Phase B) + +**Location:** `origin/worktree-zeta-sensitivity` branch. +**Status (as of 2026-07-05):** FAILED — positive control not passed. + +**One-line result:** recovery ratio median 0.009 (bar: ≥ 0.5). Root cause: the windowed training objective (rho-90, warmup-5) has a ~130× hotstart-transient noise floor relative to the planted signal. The signal is invisible even with zero observation noise and warm-started weights. + +**Phase B objective:** reduce the noise floor to ≤ 0.25 mean L1 (≤ 10% of a converged run). This is NOT YET MET as of 2026-07-05. Required before any identifiability claim for leakance. + +**Key decomposition:** + +| Quantity | Value | +|---|---| +| Planted signal (continuous residual, teacher weights + teacher obs) | 0.0076 mean L1 | +| Step-0 windowed training loss (warm-started run A) | 1.017 mean L1 | +| Noise floor / signal ratio | ~130× | +| Run A continuous residual after 5 epochs of training | 0.4431 (58× worse than start) | + +**Implication for leakance identifiability:** gauge-loss training cannot reward reach-scale leakance even with: detectable gauge signal (constructed), zero obs noise, expressible head, and warm-start from the answer. Leakance identifiability is NOT proven. Phase B (state-cache hotstart, ≤ 0.25 mean L1 target) is required before any identifiability claim. + +**Verdicts from the control run:** + +| # | Metric | Measured | Verdict | +|---|---|---|---| +| R1 | Recovery ratio median | 0.009 | FAILED (bar: ≥ 0.5) | +| R2 | Non-planted \|zeta_net\| A/baseline | 1.11 | PRECISE — trivially, nothing moved | +| R3 | Final-epoch loss A vs B (42.2% gap) | A < B | CONFOUNDED — B's handicap accounts for gap | +| R4 | Manning's n shift (run B) | Δn = −0.019 (global, not localized) | H5 equifinality confirmed at global scale | +| R5 | Cold emergence ratio | 1.20 | SUPPRESSED (bar: > 3) | + +--- + +## Part 11: Leakance configuration checklist + +Three config changes are ALL required together to enable leakance. Missing any one causes silent failure or a config-load error. + +```yaml +# 1. Activate the term (also disables CUDA graphs): +params: + use_leakance: true + +# 2. Tell the KAN head to emit leakance parameters: +kan_head: + learnable_parameters: + - K_D + - d_gw + - leakance_factor + - n # keep existing routing params + - q_spatial + # x_storage # optional + +# 3. Set parameter ranges: +params: + parameter_ranges: + K_D: [1.0e-8, 1.0e-6] # log-space; 1/s + d_gw: [-2.0, 2.0] # m + leakance_factor: [0.0, 1.0] # dimensionless +``` + +**Important:** `use_leakance: true` AND `use_cuda_graphs: true` is rejected at config load. The combination is not supported without a separate capture path. + +--- + +## Part 12: Training monitoring checklist + +Use this list when a run produces unexpected metrics. + +- [ ] **Check the binary is current** — flat checkpoint files mean stale binary (Part 2) +- [ ] **Check `streamflow resolution` in `run.log`** — `Daily` vs `Hourly` must match your intent +- [ ] **Check precip loaded** — if `use_precip: true` in disagg block, grep `precip loading` in run.log +- [ ] **Disable CUDA graphs if loss is suspiciously smooth** — guards against NaN masking (Part 3) +- [ ] **Check `kan_parameters.nc` K_D ceiling fraction** — 100% at ceiling means the box is binding +- [ ] **Run `leakance_subset_analysis.py`** for GO/NO-GO after any leakance experiment (Part 7.2) +- [ ] **Compare against baseline** — trained NSE should exceed 0.689; KGE may not (Part 5) +- [ ] **Check `ddrs show `** for final metrics in the manifest +- [ ] **Verify checkpoint format** — directory `epoch_E_mb_M/` with `head.mpk`, `optim.mpk`, `state.json` + +--- + +## Part 13: Run ID and workspace layout quick reference + +``` +.ddrs/ + system.json # GPU/driver probe result + sources.lock # fingerprints of data_sources paths + adjacency// # managed CONUS + gauge adjacency (content-addressed) + baselines// # summed-Q' baseline cache + manifest.json # metrics + gage_ids + predictions.f32 # row-major [n_gauges, n_days] + observations.f32 + runs// + manifest.json # config + sources + git SHA + output metrics + config.yaml # exact config snapshot + run.log # timestamped stdout+stderr + checkpoints/ + epoch_E_mb_M/ # DIRECTORY (flat .mpk = stale binary) + head.mpk + optim.mpk + state.json + eval/ + predictions.zarr/ # zarr-v3 group + predictions/ # float64 [n_gauges, n_days] + observations/ + gage_ids/ # uint8 [n_gauges, 8] fixed-width ASCII + baseline/ # copy of .ddrs/baselines// + kan_parameters.nc # KAN outputs + zeta (leakance) or plot params + plot/ + kan_parameters.nc # full-CONUS dump_parameters output +``` + +--- + +## Part 14: Zeta accumulator — what it measures and how to verify + +The zeta accumulator is enabled automatically during eval when `use_leakance: true`. It recomputes per-step zeta from the SAME saved primitives the backward reads, then accumulates per-reach means over the eval window. + +**Correctness identity (from `tests/zeta_accum.rs`):** + +``` +q_no_leak[0] − q_leak[0] == zeta[0] (headwater reach; exact equality) +``` + +**Verify the identity is preserved:** + +```bash +cargo test --test zeta_accum +``` + +**Check zeta is non-trivial after a leakance run:** + +```python +import xarray as xr, numpy as np +ds = xr.open_dataset(".ddrs/runs//kan_parameters.nc") +z = np.abs(ds["zeta"].values) +print(f"median |zeta|: {np.median(z):.4e} m3/s") +print(f"|zeta| > 0.01 on {np.mean(z > 0.01):.1%} of eval reaches") +# CONUS target (as of 2026-07-01): 10.4% of 64,892 reaches +``` + +**Zeta dimensionality:** `COMID_eval` dimension = gauge-subgraph union = eval network (64,892 reaches for CONUS). NOT the full 346,321-reach CONUS. + +--- + +## Provenance and maintenance + +```bash +# Re-verify Part 1 regression gates: +cargo run --release --example compare_ddr_sandbox # ABSOLUTE MATCH +cargo test --test leakance_gradcheck # 8/8 +cargo test --test zeta_accum # 6/6 +cargo test --test sparse_gradcheck # pass + +# Re-verify leakance 2x2 results: +cd ~/projects/ddr && uv run python \ + ~/projects/ddrs/scripts/leakance_subset_analysis.py \ + --hourly-on 2026-07-01T13-43-32Z-train-and-test \ + --daily-on 2026-07-01T21-20-27Z-train-and-test \ + --hourly-off 2026-06-23T02-49-12Z-conus-hourly-train-and-test \ + --daily-off 2026-06-05T01-41-16Z-train-and-test \ + --ddrs-runs-dir /home/tbindas/projects/ddrs/.ddrs/runs + +# Re-verify 7-hypothesis diagnosis: +cd ~/projects/ddr && uv run python \ + ~/projects/ddrs/scripts/leakance_diagnosis.py + +# Source files for this skill: +# /home/tbindas/projects/ddrs/CLAUDE.md +# /home/tbindas/projects/ddrs/scripts/leakance_subset_analysis.py +# /home/tbindas/projects/ddrs/scripts/leakance_diagnosis.py +# /home/tbindas/projects/ddrs/docs/2026-07-01-leakance-hourly-findings.md +# /home/tbindas/projects/ddrs/docs/2026-07-02-leakance-diagnosis-findings.md +# origin/worktree-zeta-sensitivity:docs/2026-07-03-zeta-gradient-probe-findings.md +# origin/worktree-zeta-sensitivity:docs/2026-07-04-synthetic-recoverability-findings.md +# Skill last verified: 2026-07-05 +# Volatile facts: summed-Q baseline metrics, leakance GO/NO-GO verdict, +# recoverability Phase B status — re-verify after any new experiment run +``` diff --git a/.claude/skills/ddrs-docs-and-writing/SKILL.md b/.claude/skills/ddrs-docs-and-writing/SKILL.md new file mode 100644 index 0000000..d5df37a --- /dev/null +++ b/.claude/skills/ddrs-docs-and-writing/SKILL.md @@ -0,0 +1,366 @@ +--- +name: ddrs-docs-and-writing +description: "Use when writing, updating, or reviewing any ddrs project document: findings reports, session handoffs, experiment specs, implementation plans, or the paper at ddr_equifinality/paper.tex. Also use when asking what the correct doc type is for a given output, how to connect an experiment result to the paper narrative, or whether a hypothesis verdict belongs in a findings doc or a spec. Do NOT use for running experiments, editing Rust code, debugging builds, or plotting — use ddrs-run-and-operate, ddrs-debugging-playbook, or ddrs-eval-plots instead." +--- + +# ddrs docs and writing + +This skill covers the documentation conventions, doc-type taxonomy, template structures, house style, and paper-to-experiment-log connection for the ddrs project. + +## Glossary (defined once) + +| Term | Meaning | +|---|---| +| **ddrs** | BURN-0.21 Rust port of the Python DDR (Differentiable Discharge Routing) solver using Muskingum-Cunge | +| **DDR** | Python/PyTorch reference at `~/projects/ddr/`. ddrs must stay gradient-exact against it | +| **KAN head** | Kolmogorov-Arnold Network head (`rskan::KanLayer`) that maps catchment attributes → routing parameters | +| **summed-Q′** | No-routing baseline: per-gauge sum of upstream divide Qr. Any trained model must beat this to prove routing earns its keep | +| **CONUS** | Contiguous United States; 346,321 MERIT reaches, 338,814 edges | +| **eval network** | The gauge-subgraph union used at evaluation time (64,892 reaches in the 2×2 experiments) | +| **zeta** | Per-reach GW–SW exchange flux (m³/s); positive = losing stream | +| **leakance** | Experimental GW–SW water-loss term; off by default; controlled by `use_leakance: true` in config | +| **disagg head** | Daily→hourly disaggregation sub-head inside KanHead; driven by AORC hourly precip | +| **spec** | Design doc written BEFORE an experiment. Lives in `docs/superpowers/specs/` | +| **plan** | Implementation task list derived from a spec. Lives in `docs/superpowers/plans/` | +| **findings doc** | Post-experiment narrative with hypothesis table, results, and verdict. Lives in `docs/` | +| **handoff** | Session-boundary doc with the action list for the next session. Lives in `docs/` | +| **journal** | Multi-experiment chronological record (used for cross-cutting investigations). Lives in `docs/` | + +--- + +## Doc-type taxonomy + +Choose the correct doc type before writing anything. + +| Situation | Doc type | Location | Naming pattern | +|---|---|---|---| +| Planning a new experiment before any code runs | **spec** | `docs/superpowers/specs/` | `YYYY-MM-DD--design.md` | +| Breaking a spec into implementation tasks | **plan** | `docs/superpowers/plans/` | `YYYY-MM-DD-.md` (same slug) | +| Summarizing what an experiment found after it ran | **findings doc** | `docs/` | `YYYY-MM-DD--findings.md` | +| Handing state to the next session mid-experiment | **handoff** | `docs/` | `YYYY-MM-DD--handoff.md` | +| Documenting a cross-cutting investigation that spans multiple runs | **journal** | `docs/` | `_journal.md` or a named findings doc | +| Documenting a data source, contract, or external API | **reference** | `docs/reference/` or `docs/` | descriptive name, no date required | + +### When NOT to write a new doc + +- If a findings doc for that experiment already exists: update it in place (add a datestamped section) rather than creating a duplicate. +- Handoffs and journals are living docs — append, do not replace. +- The spec/plan pair is mandatory before running expensive GPU jobs. Do not skip the spec to save time: the pre-registered hypotheses and falsification criteria protect against HARKing (Hypothesizing After Results are Known). + +--- + +## Findings doc template + +Every experiment that runs on real data gets a findings doc. Use this structure exactly. + +```markdown +# — findings () + +Spec: `docs/superpowers/specs/.md` +Plan: `docs/superpowers/plans/.md` (if applicable) +Script: `scripts/