diff --git a/.claude/skills/ddrs-architecture-contract/SKILL.md b/.claude/skills/ddrs-architecture-contract/SKILL.md new file mode 100644 index 0000000..2b2493a --- /dev/null +++ b/.claude/skills/ddrs-architecture-contract/SKILL.md @@ -0,0 +1,428 @@ +--- +name: ddrs-architecture-contract +description: "Use when you are about to touch src/routing/, src/sparse.rs, src/geometry.rs, src/nn/kan_head.rs, src/cuda_graph/, Cargo.toml's rskan pin, or any training/eval path; when you need to know which invariants are load-bearing and why; when a test fails and you need to triage against known weak points; when you are running an experiment and need to know the current performance baseline and which claims are proven vs open; or when setting up the binary install / CLI workflow." +--- + +# ddrs Architecture Contract + +## When NOT to use this skill + +- For CLI lifecycle / workflow orchestration details → read `docs/superpowers/specs/2026-05-30-ddrs-cli-lifecycle-design.md` +- For the BURN 0.21 autograd API recipe → `.claude/references/ddrs-burn-autograd.md` +- For data-source path details (zarr layout, icechunk sniffing) → `.claude/references/ddrs-reading-inputs.md` +- For eval output format / zeta netcdf schema → `.claude/references/ddrs-reading-outputs.md` + +--- + +## 1. What ddrs is (two sentences) + +`ddrs` is a **BURN-0.21 Rust port** of DDR, a differentiable Muskingum-Cunge routing solver originally in Python/PyTorch. The port must produce **gradient-exact** outputs against DDR's reference on the 5-reach RAPID sandbox; that guarantee is the V1 invariant and must hold after every commit. + +**BURN** = Rust deep-learning framework (analogous to PyTorch). **Muskingum-Cunge** (MC) = a linear reservoir routing scheme where each reach has coefficients c1–c4 derived from channel geometry and Manning's equation. **Gradient-exact** = `max(|ddrs_output - ddr_output|) < 1e-3 m³/s` on that sandbox. + +--- + +## 2. The seven invariants — break any of these and the port is meaningless + +| # | Invariant | File(s) | Test / guard | +|---|---|---|---| +| 1 | `examples/compare_ddr_sandbox` reports **ABSOLUTE MATCH** (max abs diff < 1e-3 m³/s) | `src/routing/`, `src/geometry.rs`, `src/sparse.rs` | `cargo run --release --example compare_ddr_sandbox` | +| 2 | **f32 throughout the routing core** — no f64 or bf16 casts inside the timestep chain | `src/routing/mmc_op.rs`, `src/sparse/mod.rs`, `src/geometry.rs` | V1 test; any precision drift breaks DDR parity at the f32 floor (~1e-7 rel diff per reach) | +| 3 | **Adjacency is topologically ordered, lower-triangular** (`rows[k] >= cols[k]`) | `src/sparse/`, `src/adjacency/build.rs` | `cargo test data_zarr_store::conus_adjacency_loads_real_merit_zarr` | +| 4 | **Do NOT replace the hand-written sparse backward** in `src/sparse/mod.rs` (`CsrSolveOp impl Backward`) | `src/sparse/mod.rs` | `cargo test --test sparse_gradcheck` | +| 5 | **Routing head is `rskan::KanLayer`** via `src/nn/kan_head.rs` — `Linear(F,H) → KanLayer(H,H)×N → Linear(H,P) → Sigmoid`, no inter-block ReLU | `src/nn/kan_head.rs` | `cargo test --test kan_head` | +| 6 | **rskan pinned to a tag** in `Cargo.toml` — bump tag, re-run KAN parity tests, validate before merging | `Cargo.toml` | `cargo test --features fixtures --test kan_head_init_repro --test kan_head_init_parity --test kan_head_fixture_forward --test kan_head_fixture_backward` | +| 7 | **leakance + `use_cuda_graphs: true` is a config error** — config load rejects this combination | `src/routing/leakance.rs`, `src/config.rs` | Config validation at `ddrs plan` / `ddrs run` | + +### Why invariant 4 matters (O(nnz) vs O(n²)) + +BURN's default autograd records one node per tensor operation. If you replace `CsrSolveOp`'s hand-written backward with plain tensor unrolling, the tape grows O(n²) per timestep (n = 346,321 CONUS reaches). The custom backward keeps it O(nnz) = O(338,814 edges). Same logic applies to `TimestepOp` in `src/routing/mmc_op.rs` — one node per timestep, not ~33. + +### Why invariant 3 matters + +Forward substitution on `A = I − c1·N` requires N to be strictly lower-triangular (every reach appears after all its upstream neighbors in the sorted order). If any `rows[k] < cols[k]` entry exists, the solver silently produces wrong answers with no error. + +--- + +## 3. Source tree in one screen + +``` +src/ +├── routing/ +│ ├── mmc.rs MuskingumCunge: setup_inputs, forward, route_timestep +│ ├── mmc_op.rs TimestepOp — single Backward per timestep; saves 23 intermediates +│ ├── leakance.rs GW–SW loss term; TimestepLeakanceOp: Backward (experimental) +│ └── utils.rs denormalize, hotstart, dense helpers +├── sparse/ +│ ├── mod.rs CsrPattern (Arc-shared), CsrSolveOp + hand-written Backward +│ ├── cusparse.rs cuSPARSE SpMV + SpSV FFI wrappers (SP-9) +│ └── dispatch.rs CPU forward-sub vs cuSPARSE SpSV selector +├── cuda_graph/ SP-10 CUDA Graph capture/replay (forward-only; backward not yet captured) +│ ├── capture.rs +│ ├── geometry_kernel.rs fused #[cube] kernels K1/K2/K3 +│ └── scratch.rs +├── geometry.rs Trapezoidal channel geometry (Leopold & Maddock) +├── config.rs YAML config, parameter ranges, log-space flags, SparseSolver enum +├── nn/kan_head.rs KAN head via rskan — matches DDR's kan.py exactly +├── adjacency/build.rs Managed adjacency builder (topological_sort matches petgraph DFS) +├── data/ Live zarr/netcdf/icechunk readers — no export step +│ ├── ids.rs Comid(i64), Staid(String) newtypes; IdIndex +│ ├── dates.rs TimeAxis + rho-window sampler +│ └── store/ zarr.rs, zarr_obs.rs, zarr_qprime.rs, obs_writer.rs +├── training/ +│ ├── loss.rs L1 (default) or nnse-kge; config-selectable +│ ├── forward.rs LeakanceOverride seam (eval path only) +│ ├── bootstrap.rs Checkpoint resume: weights + optim + RNG state +│ └── probe.rs lift_leaf, probe_forward, GradAccum (gradient probe instruments) +└── bin/ + ├── ddrs.rs Primary CLI: plan / run / show / status / gc / sources / import + ├── probe_zeta_gradient.rs gradient probe + synthetic teacher (--mode grad|perturb|teacher|floor|state-cache) + ├── train.rs Legacy (deprecated, removed in 0.4) + └── eval.rs Legacy; still used for --zeta-output on existing checkpoints +``` + +--- + +## 4. Per-timestep dataflow (the MC routing step) + +Everything below runs inside `forward_chain_inner` in `src/routing/mmc_op.rs` at the **inner-backend primitive level** — no autograd nodes are created inside this function. One `TimestepOp` node wraps the entire chain. + +``` +inputs: (n, q_spatial, p_spatial, q_t, q_prime_t) +fixed: (length, slope, x_storage, dt=3600 s) + +K1 — geometry + Muskingum coefficients (one fused #[cube] kernel on CUDA): + depth = ((Q·n·(q+1)) / (p·√slope))^(3/(3q+5)) + top_width = p · depth^q + side_slope = clamp(top_width·q / (2·depth), 0.5, 50) + bottom_width = clamp(top_width − 2·side_slope·depth, bw_lb) + hyd_radius = ((top_width+bottom_width)·depth/2) / (bottom_width + 2·depth·√(ss²+1)) + velocity = clamp((1/n)·R^(2/3)·√slope, v_lb, 15) + celerity = velocity · 5/3 + k_musk = length / celerity + denom = 2·k·(1−x) + dt + c1..c4 = Muskingum coefficients + +SpMV: i_t = N · q_t (cuSPARSE SpMV on GPU; scatter on CPU) + +K2 — RHS assembly: + b_rhs = c2·i_t + c3·q_t + c4·q_prime_t + +[optional leakance, when params.use_leakance: true] + area_z = (p · depth)^q_eps · length + zeta = leakance_factor · area_z · K_D · (depth − d_gw) + b_rhs = b_rhs − zeta + +A-values: a_values = assemble_primitive(c1) [CSR values of A = I − c1·N] + +SpSV: x_sol = triangular_csr_solve(a_values, b_rhs) [lower-triangular] + +K3 — clamp: + q_next = clamp_min(x_sol, discharge_lb) +``` + +On the CUDA path (SP-10), K1+K2+K3 are fused `#[cube]` kernels, and the captured per-step sequence is **K1 → SpMV → K2 → assemble → SpSV → K3** — six kernel launches replayed as one `cuGraphLaunch`. + +**Cold start (t = 0):** solves `(I − N)·Q_0 = q'_0`. On a linear chain this reduces to `Q_0[i] = Σ_{j ≤ i} q'_0[j]` (cumulative sum). + +--- + +## 5. KAN head architecture + +``` +Linear(F, H) + → KanLayer(H, H) × num_hidden_layers [ALL layers receive the SAME init seed — DDR kan.py :24-34 quirk] + → Linear(H, P) + → Sigmoid +→ output in [0, 1] (denormalized to physical units in setup_inputs via config.rs bounds) +``` + +- **F** = number of catchment attributes, **H** = hidden size, **P** = number of learnable routing parameters +- **No inter-block ReLU** — DDR's `kan.py` has none; adding one breaks parity +- `rskan` version as of 2026-07-05: **v0.1.3** (verify with `grep rskan Cargo.toml`) +- All `num_hidden_layers` KanLayers use the **same seed** (a DDR quirk preserved intentionally for parity — see `src/nn/kan_head.rs`) + +--- + +## 6. Operational traps and known weak points + +### STALE-BINARY TRAP (high severity — has caused silent wrong results) + +`cargo build` and `cargo run` do NOT update `~/.cargo/bin/ddrs`. If you type `ddrs run` after editing `src/`, you silently execute the old binary. The manifest's `git.sha` is stamped from `.git` at runtime, not from the binary, so the run log looks current. + +**After any `src/` change, do ONE of:** +```bash +cargo install --path . # canonical refresh +# or, faster if target/release is warm: +cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs +# or bypass the installed copy entirely: +cargo run --release --bin ddrs -- run --workflow … +``` + +**Self-check:** current checkpoints are **directories** (`.ddrs/runs//checkpoints/epoch_E_mb_M/head.mpk`). A stale pre-checkpoint-resume binary writes flat files (`epoch_E_mb_M.mpk`). Flat files = stale binary. + +This trap caused the 2026-07-01 leakance×hourly 2×2 first run: the hourly cell silently ran flat-repeat-24 because the installed binary predated the disaggregation feature. + +### CUDA Graphs mask NaN (high severity) + +`use_cuda_graphs: true` can return stale finite loss when the forward produces NaN. Always validate new forward-path changes with `use_cuda_graphs: false` before benchmarking with graphs on. + +`leakance + use_cuda_graphs: true` is rejected at config load time — this combination is a hard error, not a silent wrong result. + +### Checkpoint resume drifts slightly + +Checkpoints store weights/moments in **f16** (`CompactRecorder = HalfPrecisionSettings`). A resumed trajectory diverges slowly from an uninterrupted one. Exact state (epoch, mini-batch cursor, RNG permutation) is preserved, but weight precision is not. See `docs/2026-06-07-checkpoint-resume-handoff.md` follow-up #1. + +### Fixture regeneration caveat (as of 2026-07-05) + +The DDR reference state used to validate V1 lives only in the desktop's `~/projects/ddr` working tree (unpushed `geometry/trapezoidal.py` work). A fixture regenerated from a clean DDR clone diverges ~1% at every ddrs commit — that is a wrong reference, not a port bug. See `.claude/references/ddrs-comparing-to-ddr.md` §Regenerating fixtures. + +### Worktree binary path + +Fresh worktrees lack the gitignored `output/` and fixture directories. Relative `target/release` resolves to the main tree's stale binary in some shells. Always use absolute paths or `cargo run` in worktrees. See `.claude/memory/ddrs-worktree-gotchas.md`. + +--- + +## 7. Performance numbers (as of 2026-07-05) + +### CONUS network + +| Metric | Value | +|---|---| +| Reaches (CONUS MERIT) | 346,321 | +| Edges | 338,814 | +| Gauge training set | ~2,365 (CONUS) | + +### Summed-Q' baseline (no routing, no learned params) + +This is the sanity floor: per-gauge sum of upstream divide Q' over the eval window. + +| Metric | Value | +|---|---| +| Median NSE | 0.689 | +| Median KGE | 0.723 | + +**If a trained run does not beat NSE 0.689, routing is not earning its keep. Check training loss curves and KAN gradient stats first.** + +### Best trained result (as of 2026-07-05) + +Precip-driven disaggregation + L1 loss, 2,365 CONUS gauges, eval 2026-06-23: + +| Metric | Value | +|---|---| +| Median NSE | 0.715 (+0.037 vs baseline) | +| Median KGE | 0.711 (−0.012 vs baseline) | + +**KGE does NOT beat the summed-Q' baseline in any config as of 2026-07-05.** NSE does (+0.037 with precip disagg). The NSE gain is real; the KGE regression traces to over-attenuation of flood peaks (the L1 / NSE gradient rewards the MC solver for attenuating, reducing `α = σ_sim/σ_obs` below 1). + +The `nnse-kge` loss mode (`experiment.loss.kind: nnse-kge`) exists to restore the KGE gradient, but no validated CONUS result with this mode is available as of 2026-07-05. + +--- + +## 8. Leakance — experimental GW–SW water-loss term + +### What it is + +A losing-stream correction subtracted from the routing RHS `b` at each timestep: + +``` +zeta = leakance_factor · area_z · K_D · (depth − d_gw) +area_z = (p · depth)^q_eps · length (plan-view wetted area, m²) +b ← b − zeta positive zeta = losing reach +``` + +Implementation: `src/routing/leakance.rs`. Gradient is analytical via `TimestepLeakanceOp: Backward`. + +### How to enable (three required config changes) + +```yaml +params: + use_leakance: true # activates term; forces use_cuda_graphs: false + parameter_ranges: + K_D: [1.0e-8, 1.0e-5] # log-space; hydraulic exchange rate, 1/s + d_gw: [-2.0, 2.0] # groundwater depth offset, m + leakance_factor: [0.0, 1.0] # dimensionless scale + +kan_head: + learnable_parameters: [n, q_spatial, x_storage, K_D, d_gw, leakance_factor] +``` + +Note: original K_D range was `[1e-8, 1e-6]`. The recoverability experiment (2026-07-04) widened to `[1e-8, 1e-5]` to achieve 58/96 expressible sites (vs 23/96 at the original ceiling). Use `[1e-8, 1e-5]` for any future leakance work. + +### Gradient-exactness guard (run after any change to leakance.rs) + +```bash +cargo test --test leakance_gradcheck # analytical ≈ finite-difference (8/8) +cargo test --test leakance_off_parity # byte-identical to no-leakance when off (3/3) +cargo test --test zeta_accum # eval zeta == what was subtracted from b (6/6) +cargo run --release --example compare_ddr_sandbox # must still report ABSOLUTE MATCH +``` + +### Leakance status summary (as of 2026-07-05) + +**2×2 verdict (leakance × forcing, 2026-07-01): GO-marginal.** +- Leakance + hourly: ΔNSE +0.0005, ΔKGE +0.0018 on the losing-stream subset (55.5% of gauges improve) +- Leakance + daily: ΔNSE −0.0017, ΔKGE −0.0009 (35.6% improve — hurts) +- Zeta gate: |zeta| > 0.01 m³/s on 10.4% of 64,892 eval reaches (bar: ≥10%) + +**Low-zeta diagnosis (2026-07-02):** + +| Hypothesis | Verdict | Key evidence | +|---|---|---| +| H1 — K_D box clips flux | REFUTED | 71.5% of reaches CAN exceed 0.01 m³/s inside the current box; median utilization 3.4% | +| H2 — driving-head starvation | SUPPORTED | median head `(depth − d_gw)` = 0.02 m; 47% of reaches gaining at eval-window mean | +| H3 — KAN variance collapse | REFUTED | d_gw–meanP Spearman +0.71; K_D–aridity +0.61 — strong learned structure | +| H4 — gauge bias / gradient starvation | SUPPORTED | zeta–uparea ρ +0.76; gauged median |zeta| 6.7e-3 vs ungauged 5.9e-4; dry/wet ratio 0.40 (inverted from physics) | +| H5 — equifinality (n absorbs loss) | SUPPORTED (daily only) | daily Δn = +0.012 (0.59 IQR, ~20%); hourly Δn nil (0.05 IQR) | +| H6 — wrong yardstick | REFUTED | fractional loss agrees: 8.4% lose >1% of local flow | +| H7 — d_gw model-form error | REFUTED | 0.0% of d_gw at bounds, incl. dry tercile | + +**Implication of diagnosis: K_D widening alone is NOT recommended.** The binding constraint is the training signal (H2 + H4), not the parameter box. The pre-registered Phase-3 gate FAILED; the widened-K_D retrain was not run. This supersedes the "widen K_D — top follow-up" recommendation from the 2026-07-01 findings. + +**Gradient probe (2026-07-03, worktree `origin/worktree-zeta-sensitivity`):** + +| Hypothesis | Bar | Measured | Verdict | +|---|---|---|---| +| P1 — starvation (gradient dead off-gauge) | gauged/ungauged |g| ≥ 10× | 1.5× (trained), 2.9× (cold) | REFUTED | +| P2 — rejection (gradient pushes zeta down) | >67% dry-tercile push-down | 52.5% (≈ neutral) | REFUTED | +| P3 — detectability (real-magnitude loss visible at gauge) | ≥10% of Ref δ=0.01 probes detectable | 4.2% (4/96); delta is 53× smaller than median 5% discharge-uncertainty band | NO-GO | + +**Synthetic recoverability positive control (2026-07-04, worktree):** + +The positive control FAILED: median recovery ratio = 0.009 (bar: ≥0.5). Root cause: the windowed training objective has a ~130× hotstart-transient noise floor. Continuous residual with teacher weights on teacher obs = 0.0076 mean L1; step-0 windowed training loss = 1.017. The planted signal (0.8% of training loss) is invisible. After 5 epochs, Adam actively degrades the model (continuous residual grew from 0.0076 to 0.4431 — 58× worse than not training). + +**Leakance identifiability is NOT proven. The positive control must pass (Phase B objective: windowed training loss ≤ 0.25 mean L1, i.e. ≤10% of a converged run's loss) before any identifiability claim can be made.** + +--- + +## 9. Phase B objective and current state (as of 2026-07-05) + +**Phase B goal:** state-cache hotstart — inject continuous-run discharge state at each training window boundary to eliminate the hotstart-transient noise floor. + +**Target:** windowed training loss ≤ 0.25 mean L1 (≤10% of a converged run's 1.017 step-0 loss). + +**Status:** NOT YET MET as of 2026-07-05. The state-cache infrastructure (`experiment.state_cache`, `src/data/store/obs_writer.rs`, `src/training/forward.rs` injection seam, `--mode state-cache` in probe binary) is implemented in `origin/worktree-zeta-sensitivity` but the floor validation target has not been hit. + +**Until Phase B passes, do not claim leakance is learnable from gauge-only supervision.** + +--- + +## 10. CLI quick-reference + +### First-time setup + +```bash +cargo install --path . # installs ddrs to ~/.cargo/bin/ +ddrs plan # GPU probe + smoke test + writes ddrs.yaml (opens $EDITOR) +ddrs run --workflow train-and-test # train + eval + write manifest +``` + +### After any src/ change + +```bash +cargo install --path . # or: cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs +``` + +### Key commands + +```bash +ddrs sources use conus-hourly # switch to hourly AORC precip source group +ddrs sources list # show active group (* = match) +ddrs show # inspect run manifest +ddrs status # workspace summary + disk usage +ddrs gc --keep 5 --keep-successful # prune old runs + +# Hourly forcing requires BOTH: +# 1. ddrs sources use conus-hourly (adds aorc_precip path) +# 2. kan_head.disaggregation.use_precip: true in ddrs.yaml +# Without aorc_precip, config with use_precip: true is a hard error. + +# Resume from checkpoint: +# Set experiment.checkpoint: .ddrs/runs//checkpoints/epoch_E_mb_M in ddrs.yaml +# Then raise experiment.epochs above E. +``` + +### Regression tests (run after touching core routing) + +```bash +cargo run --release --example compare_ddr_sandbox # V1: must print ABSOLUTE MATCH +DDRS_FORCE_GRAPHS=1 cargo run --release --example compare_ddr_sandbox # V9: CUDA graphs bit-match +cargo test --test mmc # hotstart, coefficients, forward, autodiff +cargo test --test sparse_gradcheck # CsrSolveOp backward +cargo test --test sp8_gradcheck # fused TimestepOp backward +cargo test --test leakance_gradcheck # leakance backward (8/8) +cargo test --test leakance_off_parity # byte-identical to no-leakance (3/3) +cargo test --test zeta_accum # zeta diagnostic identity (6/6) +``` + +--- + +## 11. Workspace layout + +| Path | Purpose | +|---|---| +| `ddrs.yaml` | Workflow + experiment config (gitignored) | +| `.ddrs/system.json` | GPU/driver/smoke-test record | +| `.ddrs/sources.lock` | Fingerprints of data_sources paths | +| `.ddrs/adjacency//` | Cached CONUS + gauges adjacency zarr stores (content-addressed) | +| `.ddrs/baselines//` | Cached summed-Q' baseline (blake3 of data sources + time window) | +| `.ddrs/runs//manifest.json` | Per-run manifest (config + sources + git SHA + outputs) | +| `.ddrs/runs//config.yaml` | Snapshot of the config that produced this run | +| `.ddrs/runs//run.log` | Timestamped stdout+stderr (fd-level tee) | +| `.ddrs/runs//checkpoints/epoch_E_mb_M/` | Checkpoint directory: `head.mpk`, `optim.mpk`, `state.json` | +| `.ddrs/runs//kan_parameters.nc` | Eval-window per-reach zeta/zeta_net/depth_mean/area_z_mean/q_mean | + +--- + +## 12. Data sources summary + +| Source | Type | Path (as of 2026-07-05) | +|---|---|---| +| MERIT adjacency | managed zarr (built from fabric) | `.ddrs/adjacency//` | +| Streamflow Q' (CONUS) | icechunk | `/mnt/ssd1/data/icechunk/merit_dhbv2_UH_retrospective.ic` | +| USGS observations (CONUS) | icechunk | `/mnt/ssd1/data/icechunk/usgs_daily_observations` | +| AORC precip (hourly) | zarr-v3, catchment-major, mm/hr | `/mnt/ssd1/data/aorc/merit_unit_catchments.zarr` | +| Global streamflow Q' | zarr-v2 multi-zone (60 zones) | `/gpfs/hjj5218/data/dmc_forcing/streamflow/zarr/8km/merit_global_v2.7` | +| Global observations | zarr-v2, one array per `Provider__GageId` | `/gpfs/hjj5218/data/dmc_forcing/observation/dMC_global_v3.1` | + +Global Q' units: m³/s (confirmed empirically — no units attribute on the zarr). Time axis: CF `days since 1980-01-01`. ~42k fabric reaches lack predictions → 0.001 fill at read. + +Hourly AORC precip: zarr-v3, catchment-major (COMID-first), mm/hr, starts 1980-01-01 UTC. Experiment windows using hourly forcing must not reach into 1980 (hourly-lstm store starts 1981-01-01). + +--- + +## 13. Branches + +| Branch | Description | +|---|---| +| `master` | Main integration branch | +| `unit_catchments` | Current working branch (as of 2026-07-05) | +| `origin/worktree-zeta-sensitivity` | Most advanced — Phase B state-cache hotstart, gradient probe, recoverability control, unit-catchment attribute wiring | + +--- + +## Provenance and maintenance + +Re-verify commands (copy-pasteable, all from project root): + +```bash +# V1 invariant +cargo run --release --example compare_ddr_sandbox + +# Leakance gradient-exactness suite +cargo test --test leakance_gradcheck && cargo test --test leakance_off_parity && cargo test --test zeta_accum + +# KAN head parity +cargo test --features fixtures --test kan_head_init_repro --test kan_head_init_parity --test kan_head_fixture_forward --test kan_head_fixture_backward + +# Sparse backward +cargo test --test sparse_gradcheck && cargo test --test sp8_gradcheck + +# Check rskan version +grep rskan Cargo.toml + +# Check current binary is fresh (flat files = stale) +ls ~/.cargo/bin/ddrs -la && ls .ddrs/runs/ 2>/dev/null | tail -3 +``` + +Ground-truth sources read to produce this skill: `CLAUDE.md`, `.claude/ARCHITECTURE.md`, `.claude/references/ddrs-burn-autograd.md`, `.claude/references/ddrs-architecture.md`, `docs/2026-07-02-leakance-diagnosis-findings.md`, `origin/worktree-zeta-sensitivity:docs/2026-07-03-zeta-gradient-probe-findings.md`, `origin/worktree-zeta-sensitivity:docs/2026-07-04-synthetic-recoverability-findings.md`. Volatile facts dated 2026-07-05. Re-read those files when key numbers or experiment verdicts change. diff --git a/.claude/skills/ddrs-build-and-env/SKILL.md b/.claude/skills/ddrs-build-and-env/SKILL.md new file mode 100644 index 0000000..2fad55a --- /dev/null +++ b/.claude/skills/ddrs-build-and-env/SKILL.md @@ -0,0 +1,439 @@ +--- +name: ddrs-build-and-env +description: Use when setting up a fresh ddrs checkout, diagnosing build failures, hitting the forked-dependency trap, missing fixture errors, static netcdf/HDF5 cmake issues, stale-binary symptoms, or CUDA graphs masking NaN. Also use when cargo build succeeds but runtime behaves as an old version, or when fixture regeneration is needed after DDR solver changes. +--- + +# ddrs Build and Environment Runbook + +## Glossary (read once; terms used throughout) + +| Term | Meaning | +|---|---| +| **BURN** | Rust deep-learning framework (like PyTorch but for Rust). Version 0.21 in ddrs. | +| **DDR** | The Python/PyTorch reference implementation at `~/projects/ddr`. ddrs is its Rust port. | +| **V1 gate** | The regression test that must always pass: `compare_ddr_sandbox` reports ABSOLUTE MATCH (max abs diff < 1e-3 m³/s). | +| **`[patch.crates-io]`** | Cargo mechanism to globally replace a dependency's source. ddrs uses this to swap crates.io `cubecl`/`burn` for forked GitHub branches. | +| **KAN head** | Kolmogorov-Arnold Network routing head (`rskan::KanLayer`). Replaces MLP. Must stay at tag v0.1.3. | +| **f32 invariant** | All routing-core tensors stay float32. No f64, bf16, or mixed precision. | +| **uv** | Python package manager (like pip + venv). DDR's venv is managed by uv; needed for fixture regeneration only. | +| **icechunk** | Transactional Zarr-over-filesystem store for streamflow + observations data. | +| **cuSPARSE** | NVIDIA sparse linear algebra library. Used for the GPU triangular solve in `src/sparse.rs`. | + +--- + +## When NOT to use this skill + +| If you need... | Use instead | +|---|---| +| MC routing algorithm math | `.claude/references/ddrs-algorithm.md` | +| Autograd / sparse backward internals | `.claude/references/ddrs-burn-autograd.md` | +| DDR parity / V1 failure debugging | `.claude/references/ddrs-comparing-to-ddr.md` | +| Training a run from scratch | `CLAUDE.md` §"ddrs CLI" or README §"Getting started" | +| Leakance experiment status | `docs/2026-07-01-leakance-hourly-findings.md` | +| Architecture diagram | `.claude/ARCHITECTURE.md` | + +--- + +## 1. Prerequisites Checklist + +Before `cargo build` can succeed, verify every item below. + +### 1a. Rust toolchain + +```bash +rustc --version # must be >= 1.80; tested on 1.94.0 as of 2026-07-05 +cargo --version +``` + +Install or update via rustup: + +```bash +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +rustup update stable +``` + +### 1b. cmake (required for static netcdf/HDF5) + +`Cargo.toml` declares `netcdf = { version = "0.12", features = ["static"] }`. The `static` feature compiles bundled HDF5 and netcdf-c from source during `cargo build`. This avoids depending on system-installed dev packages (HPC hosts often have only Intel-MPI-flavored shared libs that would break `ddrs` outside `module load`). + +**Consequence:** cmake must be available on your PATH before building. + +```bash +cmake --version # must exist; any reasonably recent version works +``` + +If missing: +```bash +# Debian/Ubuntu +sudo apt-get install cmake +# Arch +sudo pacman -S cmake +# HPC (check module system) +module load cmake +``` + +### 1c. CUDA Toolkit + +Required only for the GPU path (`sparse_solver: cuda`, `use_cuda_graphs: true`). The CPU/NdArray backend path builds and runs without CUDA. + +```bash +nvcc --version # want CUDA 12+ (13.2 validated as of 2026-07-05) +nvidia-smi # driver must support the CUDA version +``` + +Validated configurations: +- RTX 4080 SUPER, driver 610.43.02, CUDA 13.2 (desktop, as of 2026-07-05) +- 8× A100, driver 575.57.08, CUDA 12, sm_80 (HPC) + +For CPU-only machines: skip CUDA setup. Override the default GPU config at runtime (see §5 "CPU-only override"). + +### 1d. git (for fork resolution) + +The forked cubecl and burn are fetched via HTTPS git by Cargo automatically. No local clones of cubecl or burn are needed — `git clone ddrs && cargo build` is sufficient. + +```bash +git --version # any recent version +``` + +--- + +## 2. Build Steps + +```bash +# Clone ddrs +git clone git@github.com:taddyb/ddrs ~/projects/ddrs +cd ~/projects/ddrs + +# Release build (LTO=thin; matches what ddrs CLI uses) +cargo build --release + +# Sanity check: V1 gate +mkdir -p output +cargo run --release --example compare_ddr_sandbox +# Expected last line: "verdict: ABSOLUTE MATCH (max abs < 1e-3 m³/s)" +``` + +The first build fetches: +- `github.com/taddyb/cubecl` branch `ddrs-release` (all cubecl-* crates) +- `github.com/taddyb/burn` branch `ddrs-sp7-primitive-ctor` (all burn-* crates) +- `github.com/taddyb/rskan` tag `v0.1.3` + +This takes several minutes on first run; subsequent builds use the Cargo registry cache. + +--- + +## 3. The Forked-Dependency Trap + +### What is patched and why + +`Cargo.toml` contains a `[patch.crates-io]` block that replaces the published crates.io versions of cubecl and burn with fork branches on github.com/taddyb: + +``` +[patch.crates-io] +cubecl = { git = "https://github.com/taddyb/cubecl.git", branch = "ddrs-release" } +cubecl-cuda = { git = "..." } # ... and 8 more cubecl-* crates +burn-cubecl = { git = "https://github.com/taddyb/burn.git", branch = "ddrs-sp7-primitive-ctor" } +burn-autodiff = { git = "..." } # ... and 12 more burn-* crates +``` + +The patches add exactly two `pub` accessors needed by ddrs's cuSPARSE GPU solve (SP-7): + +| Crate | Added accessor | +|---|---| +| `cubecl-cuda` 0.10 | `pub fn CudaServer::stream() -> CUstream` | +| `burn-cubecl` 0.21 | `pub fn CubeTensor::from_handle(...) -> Self` | + +These were `pub(crate)` in the upstream releases. The plan is to upstream them as SP-8 and remove `[patch.crates-io]` once merged. + +### Rules when working with forks + +1. **All burn-* crates must come from the same fork branch.** If Cargo resolves any burn-* crate from crates.io while another resolves from the fork, you get duplicate `Device` trait objects and cryptic link errors. The `[patch.crates-io]` block covers all 13 burn sub-crates; do not add a direct dependency on a crates.io burn sub-crate that would escape the patch. + +2. **Same rule for cubecl-***: all 10 crates in the monorepo must come from `ddrs-release`. + +3. **To iterate on the fork locally**: push your changes to the fork branch, then pull into ddrs with `cargo update -p cubecl` (or the changed crate). Do NOT commit `path = "..."` overrides — they break the public build. + +4. **rskan is pinned to a tag**, not a branch: `rskan = { git = "https://github.com/taddyb/rskan.git", tag = "v0.1.3" }`. Bumping the tag requires re-running the KAN parity sweep (CLAUDE.md invariant 6-7). + +### Diagnosing fork resolution failures + +``` +error[E0277]: the trait `Device` is not implemented for ... +``` +→ burn crate split across crates.io and fork. Run `cargo tree -p burn-std` to find which sub-crate is from crates.io. Add it to `[patch.crates-io]`. + +``` +error: failed to resolve patches for ... +``` +→ The fork branch was renamed or deleted. Check `github.com/taddyb/cubecl` or `github.com/taddyb/burn` for current branch name. + +--- + +## 4. Static netcdf/HDF5 Build Details + +`netcdf = { version = "0.12", features = ["static"] }` causes Cargo's build script to: +1. Download HDF5 and netcdf-c sources. +2. Compile them via cmake during `cargo build`. +3. Link them statically into the final binary. + +This is intentional: the HPC hosts lack usable `libnetcdf-dev`/`libhdf5-dev` packages (the module-provided ones are Intel-MPI-flavored and break `ddrs` outside `module load`). + +**Build time impact**: first build takes several extra minutes for the cmake compile. Subsequent builds are cached by Cargo. + +**If cmake is not found**: +``` +error: failed to execute process `cmake`: No such file or directory +``` +Install cmake (see §1b) and retry `cargo build`. + +**If cmake finds system HDF5 but produces link errors**: the `static` feature should bypass system HDF5. Ensure you have not set `HDF5_DIR` in your environment pointing at an incompatible installation: +```bash +unset HDF5_DIR +unset NETCDF_DIR +cargo build --release +``` + +--- + +## 5. CPU-only Override + +On a machine without CUDA, the default config (`config/merit_training.yaml`) fails because it requests `sparse_solver: cuda`. Override it via a minimal config file: + +```yaml +# cpu_override.yaml (do not commit) +sparse_solver: cpu +use_cuda_graphs: false +``` + +```bash +ddrs --config cpu_override.yaml plan +ddrs --config cpu_override.yaml run --workflow train +# or for just the V1 gate (V1 always defaults to NdArray/CPU): +cargo run --release --example compare_ddr_sandbox # no override needed +``` + +The CPU NdArray backend is the default for `compare_ddr_sandbox`; V1 always passes on CPU. Only training-scale runs need the GPU config override. + +--- + +## 6. Gitignored Fixtures and Outputs + +Three categories of files are gitignored and must be recreated after a fresh clone: + +### 6a. Sandbox fixtures (`/fixtures/`, `examples/fixtures/`) + +These are the V1 gate inputs generated by DDR's Python solver. They are gitignored because they are derived artifacts. `tests/fixtures/` IS tracked; only root-level and examples-level fixtures are excluded. + +```bash +# Regenerate after DDR's solver changes, or on a fresh clone +cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/export_ddr_sandbox.py +``` + +CRITICAL CAVEAT (as of 2026-07-05): The valid V1 fixture lives ONLY in the desktop's `~/projects/ddr` working tree. That tree contains unpushed work — `src/ddr/geometry/trapezoidal.py` — that does not exist in any DDR public commit. Regenerating from a clean DDR clone produces a ~1%-divergent reference (max abs ≈ 0.55 m³/s) that would make V1 fail at every ddrs commit. This is a wrong-reference artifact, not a port bug. Until DDR's geometry work is pushed, only the desktop DDR tree produces a valid V1 fixture. + +### 6b. Output directory (`output/`) + +The `compare_ddr_sandbox` example writes to `output/ddrs_vs_ddr.{csv,png}` using `File::create`, which does NOT `mkdir -p`. A fresh clone will panic on the file create. + +```bash +mkdir -p output +cargo run --release --example compare_ddr_sandbox +``` + +### 6c. Workspace artifacts (`.ddrs/`) + +The entire `.ddrs/` directory is gitignored. This includes adjacency caches, baseline caches, run manifests, checkpoints, and run logs. They are rebuilt by `ddrs plan` on first run. + +--- + +## 7. Stale-Binary Trap + +This is one of the most common sources of silent wrong behavior. + +**The problem**: `ddrs` on your PATH is `~/.cargo/bin/ddrs`. `cargo build` and `cargo run` compile into `target/release/ddrs` but do NOT copy it to `~/.cargo/bin/`. If you edit `src/` and then type `ddrs run`, you silently execute the old binary. + +The manifest's `git.sha` field is stamped from `.git` at runtime, NOT from the binary. A run can look like current code in the manifest while a weeks-old binary actually executed. This caused the 2026-07-01 leakance×hourly 2×2 to produce byte-identical hourly cells — the installed binary predated the disaggregation feature. + +**Self-check**: current checkpoints are DIRECTORIES (`epoch_E_mb_M/head.mpk` etc.). If you see flat `.mpk` files at `epoch_E_mb_M.mpk`, you ran a pre-checkpoint-resume binary. + +**Fix after every `src/` change**: + +```bash +# Canonical (always correct): +cargo install --path . + +# Faster when target/release is already built: +cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs + +# Bypass the installed binary entirely (safest during development): +cargo run --release --bin ddrs -- run --workflow train +``` + +--- + +## 8. CUDA Graphs Masking NaN + +**The trap**: `use_cuda_graphs: true` records a graph on the first forward pass and replays it on subsequent steps. If the first forward passes with a finite loss but a later one would produce NaN (e.g., from a bad parameter initialization or data batch), the graph replay returns the stale finite loss from the captured pass. You see a training run that appears to converge smoothly but is actually computing nothing. + +**Affected config key**: `use_cuda_graphs: true` in `ddrs.yaml` or `config/merit_training.yaml`. + +**Config rejection rule**: `use_leakance: true` + `use_cuda_graphs: true` is REJECTED at config load time (no exception path). These cannot be used together; the leakance kernel has no separate CUDA graph capture path. + +**Diagnosis**: + +```bash +# 1. Disable graphs and rerun the suspicious config +# Add to ddrs.yaml: +# use_cuda_graphs: false + +# 2. Watch for NaN in the loss log — if it now appears, graphs were masking it + +# 3. Find the NaN source: typically a parameter initialized to zero or +# a data batch with all-NaN streamflow +``` + +**Rule**: always validate new configs and new parameter initializations with `use_cuda_graphs: false` first. Only re-enable graphs after confirming the forward is NaN-free. + +--- + +## 9. Critical Invariants (Do Not Break) + +Breaking any of these makes the ddrs port meaningless or incorrect. + +| # | Invariant | Test | +|---|---|---| +| 1 | `compare_ddr_sandbox` must report ABSOLUTE MATCH (max abs < 1e-3 m³/s) | `cargo run --release --example compare_ddr_sandbox` | +| 2 | f32 throughout routing core; no f64/bf16 casts in `src/routing/`, `src/geometry.rs`, `src/sparse.rs` | `grep -rn 'f64\|bf16\|cast\|to_dtype' src/routing/ src/geometry.rs src/sparse.rs` | +| 3 | Adjacency is topologically ordered, lower-triangular (`rows[k] >= cols[k]`) | `cargo test data_zarr_store::conus_adjacency_loads_real_merit_zarr` | +| 4 | Hand-written sparse backward in `src/sparse.rs` must NOT be replaced with tape unrolling | `cargo test --test sparse_gradcheck` | +| 5 | KAN head = `rskan::KanLayer` via `src/nn/kan_head.rs`; no MLP placeholder; no inter-block ReLU | `cargo test --test kan_head` | +| 6 | rskan pinned to tag `v0.1.3` in Cargo.toml | `grep 'rskan.*tag' Cargo.toml` | +| 7 | KAN parity vs DDR must pass on every PR touching `src/nn/`, rskan pin, or DDR's `nn/kan.py` | See §11 KAN parity command | + +--- + +## 10. Full Verification Command Set + +Run these in order after a fresh build or after touching `src/`: + +```bash +# V1 gate — must report ABSOLUTE MATCH +mkdir -p output +cargo run --release --example compare_ddr_sandbox + +# V1 gate on CUDA + graph-capture path (if GPU available) +DDRS_FORCE_GRAPHS=1 cargo run --release --example compare_ddr_sandbox + +# Sparse gradient correctness +cargo test --test sparse_gradcheck + +# Routing correctness (linear chain) +cargo test --test mmc mc_routes_linear_chain + +# Leakance gradient-exactness (if leakance was touched) +cargo test --test leakance_gradcheck +cargo test --test leakance_off_parity +cargo test --test zeta_accum +``` + +--- + +## 11. KAN Head Parity (after touching `src/nn/`, rskan pin, or DDR's `nn/kan.py`) + +```bash +cargo test --features fixtures \ + --test kan_head_init_repro \ + --test kan_head_init_parity \ + --test kan_head_fixture_forward \ + --test kan_head_fixture_backward +``` + +If DDR's `nn/kan.py` changed: regenerate fixtures first, then re-validate: +```bash +cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/dump_kan_fixture.py +cd ~/projects/ddrs && cargo test --features fixtures --test kan_head_fixture_forward --test kan_head_fixture_backward +``` + +--- + +## 12. DDR Reference Clone (for fixture regeneration) + +```bash +git clone git@github.com:mhpi/ddr ~/projects/ddr +cd ~/projects/ddr && uv sync --all-packages +``` + +uv creates a `.venv` automatically. Fixtures are generated by running scripts under this venv: + +```bash +cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/export_ddr_sandbox.py +``` + +The uv venv must remain at `~/projects/ddr/.venv` — ddrs scripts import from DDR's Python packages. + +--- + +## 13. Data File Paths + +`config/merit_training.yaml`'s `data_sources:` block references these paths. Edit the YAML to match your machine if they live elsewhere. + +| Source | Default path | +|---|---| +| Geospatial fabric | `riv_pfaf_7_MERIT_Hydro_v07_Basins_v01_bugfix1.shp` (+ sibling `.dbf`), or a `.gpkg` | +| MERIT adjacency | Managed — built by `ddrs plan` into `.ddrs/adjacency//` | +| Per-gauge subgraphs | Managed — same directory | +| Catchment attributes | `~/projects/ddr/data/merit_global_attributes_v2.nc` | +| Streamflow forcing | `/mnt/ssd1/data/icechunk/merit_dhbv2_UH_retrospective.ic` | +| USGS observations | `/mnt/ssd1/data/icechunk/usgs_daily_observations` | +| Gauges list | `~/projects/ddr/references/gage_info/gages_3000.csv` | + +To skip the managed adjacency build (e.g., you have pre-built zarr stores): +```yaml +# in ddrs.yaml, replace geospatial_fabric with: +conus_adjacency: /path/to/merit_conus_adjacency.zarr +gages_adjacency: /path/to/merit_gages_conus_adjacency.zarr +``` + +--- + +## 14. Common Failure Modes at a Glance + +| Symptom | Root cause | Fix | +|---|---|---| +| `cmake: No such file or directory` during `cargo build` | cmake not on PATH; needed for static netcdf | Install cmake (§1b) | +| `the trait Device is not implemented` link error | burn crate split across crates.io and fork | Check `cargo tree -p burn-std`; add missing crate to `[patch.crates-io]` | +| `failed to resolve patches` | Fork branch renamed or deleted | Check `github.com/taddyb/{cubecl,burn}` for current branch | +| `thread 'main' panicked at 'No such file or directory' (output/...)` | `output/` missing on fresh clone | `mkdir -p output` | +| V1 fails with max abs ≈ 0.55 m³/s | Fixtures regenerated from wrong DDR clone (no trapezoidal.py) | Use desktop's `~/projects/ddr` working tree | +| Training appears to converge but loss never moves | CUDA graphs masking NaN | Set `use_cuda_graphs: false` and check for NaN (§8) | +| `ddrs run` uses wrong/old feature after `cargo build` | Stale installed binary at `~/.cargo/bin/ddrs` | `cargo install --path .` (§7) | +| Checkpoint is a flat `.mpk` file, not a directory | Stale pre-checkpoint-resume binary executed | `cargo install --path .` and re-run | +| `use_leakance + use_cuda_graphs rejected at config load` | These two are mutually exclusive by design | Remove `use_cuda_graphs: true` from leakance configs | +| `fixtures/sandbox/` missing, V1 panics on CSV read | gitignored artifact not regenerated | `cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/export_ddr_sandbox.py` | + +--- + +## Provenance and maintenance + +Skill written 2026-07-05 from `Cargo.toml`, `CLAUDE.md`, `README.md`, `.claude/references/ddrs-setup.md`, `.claude/references/ddrs-comparing-to-ddr.md`, and `vendor/README.md`. + +Re-verification commands: +```bash +# Confirm fork branches still exist +git ls-remote https://github.com/taddyb/cubecl.git ddrs-release +git ls-remote https://github.com/taddyb/burn.git ddrs-sp7-primitive-ctor +git ls-remote https://github.com/taddyb/rskan.git refs/tags/v0.1.3 + +# Confirm static netcdf feature still declared +grep 'netcdf.*static' /home/tbindas/projects/ddrs/Cargo.toml + +# Confirm gitignore entries +grep -E 'fixtures|output|\.ddrs' /home/tbindas/projects/ddrs/.gitignore + +# Confirm Rust version meets minimum +rustc --version # must be >= 1.80 + +# V1 gate +mkdir -p /home/tbindas/projects/ddrs/output +cargo run --release --example compare_ddr_sandbox 2>&1 | grep -E 'verdict|ABSOLUTE|FAIL' +``` diff --git a/.claude/skills/ddrs-change-control/SKILL.md b/.claude/skills/ddrs-change-control/SKILL.md new file mode 100644 index 0000000..0c3464b --- /dev/null +++ b/.claude/skills/ddrs-change-control/SKILL.md @@ -0,0 +1,413 @@ +--- +name: ddrs-change-control +description: "Use when reviewing, gating, or merging a change to ddrs source code, config files, or Cargo dependencies; when assessing whether a modification to src/routing/, src/sparse.rs, src/geometry.rs, src/nn/kan_head.rs, or Cargo.toml is safe; when a run produced unexpected results and binary staleness or an invariant violation may be the cause; or when designing an experiment that touches leakance, CUDA graphs, or the routing core." +--- + +# ddrs change-control runbook + +## Overview + +`ddrs` is a BURN-0.21 Rust port of the DDR differentiable Muskingum-Cunge routing model (Python/PyTorch reference at `~/projects/ddr/`). The port must remain **gradient-exact** against DDR. Breaking any of the seven invariants below makes the port meaningless; every PR that touches the affected files must clear its gate before merge. + +**Glossary for PyTorch engineers:** +- `BURN` — Rust deep-learning framework, analogous to PyTorch. BURN 0.21 is pinned. +- `Backward` — BURN's trait for a custom autograd function (analogous to `torch.autograd.Function`). `I` = backend (CPU/CUDA), `N` = number of saved tensors. +- `CsrPattern` — the sparsity structure of the river network adjacency matrix, stored as a Rust struct (row/col index arrays). Analogous to `torch.sparse_csr_tensor`. +- `KanLayer` — Kolmogorov-Arnold Network layer from the `rskan` crate (Rust equivalent of DDR's `kan.py`). +- `CompactRecorder` / `HalfPrecisionSettings` — BURN serializer for checkpoints. Saves weights as f16. +- `COMID` — unique 64-bit integer ID for each river reach in the MERIT-Hydro fabric. +- `Q'` (Qr) — lateral inflow forcing (divide-level runoff, m³/s) from the pre-trained DHBV2 model. Not observed discharge. +- `zeta` — leakance flux (m³/s), the GW–SW water-loss term. Subtracted from the routing RHS at every timestep. +- `rho-window` — a training mini-batch: a contiguous time slice of length `rho` (default 90 days) sampled from the training period. + +--- + +## When NOT to use this skill + +Do not use this skill for: +- **Plotting or analysis scripts only** (no `src/` change) — use `ddrs-eval-plots` instead. +- **Config tuning within documented safe ranges** (changing `experiment.epochs`, `learning_rate`, `batch_size`, loss weights) — no gate applies; these do not affect the routing core or port invariants. +- **Data source path changes only** — consult `CLAUDE.md §Data sources` directly. +- **CLI / workspace questions** — consult `CLAUDE.md §ddrs CLI`. + +--- + +## Change classification matrix + +Every change falls into one of four tiers. Look up the modified file(s) in the left column; the tier determines which gate checklist you must run. + +| Modified file(s) | Tier | Rationale | +|---|---|---| +| `src/routing/mmc.rs`, `src/routing/mmc_op.rs`, `src/routing/utils.rs` | **A — routing core** | Directly implements the Muskingum-Cunge timestep; must remain gradient-exact vs DDR | +| `src/routing/leakance.rs` | **A — routing core + leakance** | Custom `Backward` for the GW–SW term; both the forward kernel and analytical gradients must stay exact | +| `src/geometry.rs` | **A — routing core** | Trapezoidal geometry; changes cascade into every geometry-dependent variable | +| `src/sparse.rs` | **A — routing core** | Hand-written CSR triangular solve + custom `CsrSolveOp: Backward`; O(nnz) autograd tape invariant | +| `src/nn/kan_head.rs` | **B — KAN head** | Must match DDR `kan.py` exactly; rskan version pin governs this | +| `Cargo.toml` (rskan tag) | **B — KAN head** | rskan pin is the single authoritative version for KAN parity | +| `src/config.rs` | **C — config/ranges** | Parameter ranges and log-space flags affect denormalization; wrong range silently mis-scales gradients | +| `src/training/loss.rs` | **C — objective** | Autograd is unchanged (invariant 4 intact) but loss changes affect all metrics comparisons | +| `src/training/forward.rs` | **C — training path** | Disaggregation, leakance threading; changes can silently no-op features (see STALE-BINARY TRAP) | +| `config/experiments/*.yaml`, `config/sources/*.yaml` | **D — config only** | No Rust changes; validate with `ddrs plan` before running | +| Any other `src/` file | **C — default** | Run full test suite and DDR regression | + +--- + +## Tier A gate — routing core + +Run ALL of the following. A single failure is a merge blocker. + +```bash +# 1. DDR parity — THE non-negotiable regression +cargo run --release --example compare_ddr_sandbox +# Must print: "ABSOLUTE MATCH" with max abs diff < 1e-3 m³/s + +# 2. Core unit + integration tests +cargo test --lib +cargo test --test mmc +cargo test --test sparse_gradcheck + +# 3. Leakance gates (required even if you did not touch leakance.rs, +# because any routing change can disturb the leakance OFF parity) +cargo test --test leakance_gradcheck # 8/8 analytical ≈ finite-difference +cargo test --test leakance_off_parity # 3/3 byte-identical to no-leakance when off +cargo test --test zeta_accum # 6/6 accumulated zeta == headwater q difference +``` + +If you touched `src/routing/leakance.rs` specifically: +```bash +# Confirm gradient-exactness for all 8 leakance backward inputs +cargo test --test leakance_gradcheck -- --nocapture +``` + +**After passing all Tier A gates, run the binary self-check:** +```bash +# Stale-binary check: directory checkpoints = current binary +ls .ddrs/runs//checkpoints/ +# Must show directories like epoch_5_mb_9/, NOT flat files like epoch_5_mb_9.mpk +# Flat files = stale binary ran. Refresh: cargo install --path . +``` + +--- + +## Tier B gate — KAN head + +```bash +# Full KAN parity sweep (required on every PR touching src/nn/ or Cargo.toml rskan pin) +cargo test --features fixtures \ + --test kan_head_init_repro \ + --test kan_head_init_parity \ + --test kan_head_fixture_forward \ + --test kan_head_fixture_backward + +# Then run Tier A DDR parity to confirm the head change did not break routing +cargo run --release --example compare_ddr_sandbox +``` + +If a DDR-side change to `kan.py` broke the fixture: +```bash +# Regenerate under DDR's venv, then re-validate +cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/dump_kan_head.py +# Re-run the fixture tests above +``` + +--- + +## Tier C gate — config, training path, other src/ + +```bash +cargo test --lib +cargo test # full suite +cargo run --release --example compare_ddr_sandbox # DDR parity +``` + +For `src/training/forward.rs` changes that affect disaggregation or leakance threading, also run: +```bash +cargo test --test leakance_off_parity # leakance OFF must stay byte-identical +``` + +--- + +## Tier D gate — config files only + +```bash +# Validate the config parses and data sources resolve +ddrs plan --config config/experiments/.yaml \ + --workspace /home/tbindas/projects/ddrs/.ddrs +# Must exit 0 with no "drift" warnings +``` + +--- + +## The 7 non-negotiables with rationale and incidents + +### Invariant 1 — DDR sandbox ABSOLUTE MATCH + +**Rule.** `cargo run --release --example compare_ddr_sandbox` must print "ABSOLUTE MATCH" (max abs diff < 1e-3 m³/s on the 5-reach RAPID sandbox). Re-run after every change to `src/routing/`, `src/geometry.rs`, or `src/sparse.rs`. + +**Rationale.** The port exists to be gradient-exact against DDR. Any drift makes subsequent metric comparisons meaningless — you cannot tell whether a difference is a port bug or a genuine model improvement. + +**Caveat (as of 2026-06-06).** The reference DDR state lives only in the desktop's `~/projects/ddr` working tree (contains unpushed `geometry/trapezoidal.py` changes). A fixture regenerated from a clean DDR clone diverges ~1% per commit — that is a wrong reference, not a port bug. See `.claude/references/ddrs-comparing-to-ddr.md §Regenerating fixtures` before regenerating. + +--- + +### Invariant 2 — f32 throughout routing core + +**Rule.** No casts to f64 or bf16 inside `src/routing/`, `src/geometry.rs`, or `src/sparse.rs`. The DDR comparison sits at the f32 precision floor (~1e-7 relative difference per reach); any precision change breaks reproducibility. + +**Rationale.** Mixed precision introduces per-reach rounding that accumulates across the 346,321-reach CONUS network; the 1e-3 m³/s sandbox tolerance is calibrated for f32-only arithmetic. + +--- + +### Invariant 3 — lower-triangular adjacency + +**Rule.** The adjacency matrix must be topologically sorted and lower-triangular: `rows[k] >= cols[k]` for every non-zero entry. The forward-substitution solver (`triangular_solve_lower`) assumes no upstream values are uncomputed when it processes a reach. + +**Rationale.** Forward substitution over a topological order is the entire basis for the O(n) per-timestep solve. A non-lower-triangular entry means a downstream reach tries to read an upstream value before that upstream reach is solved — silent wrong output, no error. + +**Test.** `cargo test data_zarr_store::conus_adjacency_loads_real_merit_zarr` verifies the invariant on the real CONUS zarr store. + +--- + +### Invariant 4 — hand-written sparse backward + +**Rule.** Do NOT replace the hand-written `CsrSolveOp impl Backward` in `src/sparse.rs` with autograd-tape unrolling. + +**Rationale.** The entire point of the custom backward is O(nnz) tape entries per timestep. Tape unrolling would be O(n²) for a triangular solve — quadratic memory and time, infeasible at CONUS scale (346,321 reaches). The analytical backward is `∇A = -gradb[rows]·x[cols]`, exactly as in DDR's `torch.autograd.Function`. + +**Reference.** `.claude/references/ddrs-burn-autograd.md` has the full BURN-0.21 recipe. + +--- + +### Invariant 5 — KAN head architecture matches DDR + +**Rule.** The routing head is `rskan::KanLayer` via `src/nn/kan_head.rs`. The architecture is `Linear(F, H) → KanLayer(H, H) × num_hidden_layers → Linear(H, P) → Sigmoid`. No inter-block ReLU. All `num_hidden_layers` inner KanLayers receive the SAME seed (DDR's `kan.py` lines 24–34 quirk — preserved for parity). + +**Rationale.** DDR parity requires identical weight initialization. A ReLU between KAN blocks or different per-layer seeds changes the initialization and breaks the fixture tests. + +**What NOT to do.** Do not reintroduce the prior MLP placeholder. + +--- + +### Invariant 6 — rskan pinned to a tag + +**Rule.** `rskan` in `Cargo.toml` must be a git dependency pinned to a tag, currently `v0.1.3`. When updating, bump the tag, then re-run all Tier B tests and the Tier A DDR regression before merging. + +**Rationale.** An unpinned git dependency (`branch = "main"`) can change silently on `cargo update`, breaking KAN parity without any local code change. + +**Current pin (as of 2026-07-05):** +```toml +rskan = { git = "https://github.com/taddyb/rskan.git", tag = "v0.1.3" } +``` + +--- + +### Invariant 7 — KAN head parity on every relevant PR + +**Rule.** Any PR touching `src/nn/`, `Cargo.toml`'s rskan pin, or DDR's `nn/kan.py` must pass the full KAN parity suite (see Tier B gate above). + +**Rationale.** Invariant 5 is not self-enforcing at the compiler level. The fixture tests are the only automated check that the architecture, seed, and weight initialization actually match DDR. + +--- + +## The STALE-BINARY TRAP (historical incident: 2026-07-01) + +**Rule.** After touching ANY file under `src/`, refresh the installed binary before running experiments. + +```bash +# Canonical (always correct): +cargo install --path . + +# Faster if target/release is already built: +cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs + +# Bypass entirely (safest for one-off experiments): +cargo run --release --bin ddrs -- run --workflow train-and-test ... +``` + +**Why this matters.** `ddrs` on your PATH is `~/.cargo/bin/ddrs`. `cargo build` and `cargo run` do NOT update it. The manifest's `git.sha` is stamped from `.git` at runtime, so a run appears to have the correct SHA while silently executing a weeks-old binary. + +**What happened (2026-07-01).** The installed `ddrs` was dated 2026-06-03, before disaggregation (landed 2026-06-19) and leakance (landed 2026-06-29). The hourly-forcing cell silently ran flat repeat-24 with no leakance. Both hourly-ON and daily-ON cells produced byte-identical eval predictions (`52ec721`). The manifest showed `git.sha = 2cdd341` (correct HEAD), masking the stale binary completely. + +**Self-check.** Current checkpoints are DIRECTORIES: `.ddrs/runs//checkpoints/epoch_E_mb_M/head.mpk`. A stale pre-checkpoint-resume binary writes FLAT files: `epoch_E_mb_M.mpk`. Flat files = stale binary. + +--- + +## CUDA graphs + NaN masking (known gotcha) + +**Rule.** Validate model forwards with `use_cuda_graphs: false` when debugging NaN loss or unexpected constant loss. + +**Why.** `use_cuda_graphs: true` captures a kernel graph on the first forward pass and replays it on subsequent passes. If the first forward produces a NaN (e.g., during early training on bad data), the captured graph replays stale finite values rather than recomputing and propagating the NaN. The result is a constant finite loss that does not go to NaN even when the actual computation is invalid. See memory file `cuda-graphs-mask-nan.md` for the full diagnosis. + +**Hard constraint.** `params.use_leakance: true` combined with `use_cuda_graphs: true` is REJECTED at config load time. The leakance kernel is not captured in the current CUDA graph implementation; the rejection prevents silent wrong results. + +--- + +## Leakance-specific gates + +Leakance (`params.use_leakance: true`) is experimental and off by default. Any change that enables, modifies, or interacts with leakance must satisfy these gates in addition to the appropriate tier gates. + +### Enabling leakance requires three config changes together + +Missing any one causes either a config-load error or silent wrong behavior: + +| Config key | Required value | Why | +|---|---|---| +| `params.use_leakance` | `true` | Activates the leakance kernel in `route_timestep` | +| `kan_head.learnable_parameters` | Include `K_D`, `d_gw`, `leakance_factor` | Without these, the KAN head does not emit leakance params → all-zero zeta | +| `params.parameter_ranges.K_D` | `[1e-8, 1e-6]` (log-space) | Range gate; current recommendation is `[1e-8, 1e-5]` for recoverability experiments (see §Research status) | +| `params.parameter_ranges.d_gw` | `[-2, 2]` | Groundwater depth offset (m) | +| `params.parameter_ranges.leakance_factor` | `[0, 1]` | Dimensionless scale | +| `use_cuda_graphs` | `false` | Enforced by config load; leakance + graphs = rejected | + +### Leakance gradient-exactness gate + +```bash +cargo test --test leakance_gradcheck # 8/8 — all analytical grads match finite-diff +cargo test --test leakance_off_parity # 3/3 — OFF is byte-identical to no-leakance +cargo test --test zeta_accum # 6/6 — accumulated zeta == headwater q difference +cargo run --release --example compare_ddr_sandbox # must still say ABSOLUTE MATCH +``` + +### Leakance eval-time zeta diagnostic + +`dump_parameters` exports learned `K_D`/`d_gw`/`leakance_factor` per COMID but NOT the actual zeta flux (which depends on routed depth, only available during eval). The zeta diagnostic runs during `ddrs run --workflow train-and-test` Phase 2 automatically. For an existing checkpoint: + +```bash +cargo build --release --bin eval +target/release/eval \ + --config config/experiments/leakance_hourly_on.yaml \ + --checkpoint .ddrs/runs//checkpoints/epoch_5_mb_9 \ + --output /tmp/eval.zarr \ + --zeta-output .ddrs/runs//kan_parameters.nc +``` + +Output variables in `kan_parameters.nc` (dimension `COMID_eval`, 64,892 reaches on the CONUS eval network): +- `zeta` — mean |zeta| (m³/s) over eval window +- `zeta_net` — signed mean zeta (positive = losing reach) + +**GO/NO-GO bar.** |zeta| > 0.01 m³/s on at least 10% of eval reaches = zeta is physically active. + +--- + +## Research-status facts (as of 2026-07-05) + +These facts govern what claims can be made about leakance. Cite dates; do not generalize beyond what is measured. + +### Leakance 2×2 (as of 2026-07-01) — DONE + +Four valid arms, seed 42, eval window 1995/10/01–2010/09/30, 2,365 gauges: + +| arm | run id | NSE med | KGE med | +|---|---|---|---| +| hourly-OFF | `2026-06-23T02-49-12Z-conus-hourly-train-and-test` | 0.7153 | 0.7104 | +| hourly-ON | `2026-07-01T13-43-32Z-train-and-test` | 0.7145 | 0.7150 | +| daily-OFF | `2026-06-05T01-41-16Z-train-and-test` | 0.7004 | 0.7244 | +| daily-ON | `2026-07-01T21-20-27Z-train-and-test` | 0.6963 | 0.7250 | + +Losing-stream subset (1,883/2,365 gauges): hourly leakance ΔNSE +0.0005, ΔKGE +0.0018, 55.5% of gauges improve. Daily leakance ΔNSE −0.0017, ΔKGE −0.0009. Verdict: **GO — marginal** (3/3 gates met; zeta |>0.01| on 10.4% of 64,892 eval reaches). + +**Summed-Q' baseline (CONUS):** median NSE 0.689, KGE 0.723. Best trained result (precip-driven disagg + L1): median NSE 0.715, KGE 0.711 (2,365 gauges). NSE beats the baseline by +0.026; KGE does NOT beat the summed-Q' baseline in any config as of 2026-07-05. + +### Low-zeta diagnosis (as of 2026-07-02) + +| Hypothesis | Verdict | +|---|---| +| H1 — K_D ceiling clips zeta | REFUTED (71.5% of reaches CAN exceed 0.01 m³/s in-box; utilization median 3.4%) | +| H2 — driving-head starvation | SUPPORTED (median head 0.021 m; 47% of reaches gaining at eval-window mean) | +| H3 — KAN variance collapse | REFUTED (K_D–aridity ρ +0.61, d_gw–meanP ρ +0.71 — strong learned structure) | +| H4 — gauge bias / gradient starvation | SUPPORTED (gauged median |zeta| 11× ungauged; dry/wet zeta ratio 0.40, inverse of physics) | +| H5 — equifinality (daily only) | SUPPORTED (daily Δn +0.012, 0.59 IQR; hourly Δn nil) | +| H6 — wrong yardstick | REFUTED (fractional loss agrees: 8.4% of reaches lose >1% of local flow) | +| H7 — model-form error | REFUTED (0.0% of d_gw at bounds) | + +**Implication.** The K_D-widening follow-up (`[1e-8, 1e-5]`) recommended in the 2×2 findings is NOT recommended by the diagnosis. The diagnosis shows the K_D box is not the binding constraint. Widening K_D alone is expected to re-pin at the new ceiling with negligible zeta or skill change. + +### Gradient probe (as of 2026-07-03, worktree: zeta-sensitivity) + +| Probe | Verdict | Key number | +|---|---|---| +| P1 — gradient starvation | REFUTED | gauged/ungauged \|g\| ratio 1.5× trained, 2.9× cold (bar: ≥10×) | +| P2 — rejection at trained point | REFUTED | 52.5% of dry-tercile grads push zeta down (bar: >67%) | +| P3 — detectability | NO-GO | 4.2% of Ref probes detectable at δ=0.01 m³/s (bar: ≥10%); median 5%-band 0.531 m³/s vs planted signal 0.01 m³/s = **53× dilution** | + +P3 NO-GO means: gauge-only discharge supervision cannot distinguish real-world leakance magnitudes from measurement uncertainty. Transmission is fine (~95% fidelity); the problem is signal-to-noise at the sensor. + +### Synthetic recoverability positive control (as of 2026-07-04, worktree: zeta-sensitivity) + +| Metric | Measured | Verdict | +|---|---|---| +| R1 — recovery ratio median (n=58) | 0.009 | FAILED (bar: ≥0.5) | +| R2 — non-planted spatial precision | 1.11× baseline | PRECISE (trivial: model didn't move) | +| R3 — loss gap A vs B | A=1.339 vs B=2.317, +42% | A3) | + +**Root cause.** Windowed training objective (rho=90, warmup=5) has a ~130× hotstart-transient noise floor vs the planted signal. The continuous residual with teacher weights + teacher obs is 0.0076 mean L1; step-0 windowed training loss is 1.017. The optimizer chases irreducible initial-condition noise; after 5 epochs the continuous residual degrades from 0.0076 to 0.4431. + +**Implication.** Leakance identifiability is NOT proven. Phase B (state-cache hotstart, target: windowed loss ≤ 0.25 mean L1, ≤10% of a converged run) is required before any identifiability claim. Phase B is NOT yet complete as of 2026-07-05. + +--- + +## Binary management quick-reference + +| Goal | Command | +|---|---| +| Refresh installed binary after src/ change | `cargo install --path .` | +| Fast refresh (target/release already built) | `cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs` | +| Bypass installed binary for one run | `cargo run --release --bin ddrs -- run --workflow train-and-test ...` | +| Check if stale binary ran | Look for FLAT checkpoint files `epoch_E_mb_M.mpk`; current binary writes DIRECTORIES `epoch_E_mb_M/head.mpk` | + +--- + +## Workspace flag gotcha + +`--workspace` takes the path to the `.ddrs` DIRECTORY ITSELF, not its parent. Experiment configs in `config/experiments/` default to `config/experiments/.ddrs` (wrong). Always pass the root workspace explicitly: + +```bash +ddrs run --config config/experiments/leakance_hourly_on.yaml \ + --workspace /home/tbindas/projects/ddrs/.ddrs \ + --workflow train-and-test +``` + +--- + +## Provenance and maintenance + +Files read to write this skill (re-read to verify any fact): + +```bash +# Core invariants and CLI behavior +cat /home/tbindas/projects/ddrs/CLAUDE.md + +# Architecture and per-timestep dataflow +cat /home/tbindas/projects/ddrs/.claude/ARCHITECTURE.md + +# Stale-binary incident + 2×2 experiment +cat /home/tbindas/projects/ddrs/docs/2026-07-01-leakance-hourly-experiment-handoff.md + +# 2×2 final findings and GO-marginal verdict +cat /home/tbindas/projects/ddrs/docs/2026-07-01-leakance-hourly-findings.md + +# Low-zeta diagnosis (H1-H7 verdicts) +cat /home/tbindas/projects/ddrs/docs/2026-07-02-leakance-diagnosis-findings.md + +# Gradient probe (P1-P3 verdicts) — worktree +cat /home/tbindas/projects/ddrs/.claude/worktrees/zeta-sensitivity/docs/2026-07-03-zeta-gradient-probe-findings.md + +# Recoverability positive control failure — worktree +cat /home/tbindas/projects/ddrs/.claude/worktrees/zeta-sensitivity/docs/2026-07-04-synthetic-recoverability-findings.md + +# rskan version pin +grep rskan /home/tbindas/projects/ddrs/Cargo.toml +``` + +Re-verification commands: +```bash +# Confirm invariant 1 still holds on current HEAD +cargo run --release --example compare_ddr_sandbox + +# Confirm rskan pin +grep rskan /home/tbindas/projects/ddrs/Cargo.toml + +# Confirm leakance tests pass +cargo test --test leakance_gradcheck --test leakance_off_parity --test zeta_accum +``` diff --git a/.claude/skills/ddrs-config-and-flags/SKILL.md b/.claude/skills/ddrs-config-and-flags/SKILL.md new file mode 100644 index 0000000..49b0274 --- /dev/null +++ b/.claude/skills/ddrs-config-and-flags/SKILL.md @@ -0,0 +1,403 @@ +--- +name: ddrs-config-and-flags +description: "Use when you need to add, change, or audit any ddrs YAML configuration key; diagnose a config-load error; understand what a parameter controls; add a new routing parameter or training flag; or decide which experiment config to use as a starting point. Also use when modifying params.use_leakance, params.use_cuda_graphs, kan_head.disaggregation, or experiment.loss." +--- + +# ddrs Config and Flags Reference + +**Jargon primer (defined once):** +- **BURN** — Rust deep-learning framework (like PyTorch for Rust). Used instead of PyTorch here. +- **KAN head** — Kolmogorov-Arnold Network; maps per-reach catchment attributes to routing parameters. Replaces an MLP. +- **MC routing** — Muskingum-Cunge, a 1-D river routing solver. The `params:` block controls it. +- **CONUS** — Contiguous US; the default training domain (346,321 reaches). +- **Q'** — lateral inflow (m³/s) per reach; the forcing signal from dHBV2. +- **CSR** — Compressed Sparse Row; the sparse matrix format used for the triangular network solve. +- **CUDA graph** — GPU kernel sequence baked at compile time and replayed cheaply each timestep. +- **ddrs.yaml** — the live workspace config; `ddrs plan` generates it from a template. NEVER committed. +- **config/merit_training.yaml** — the canonical production template; committed and kept in sync with DDR-Python. + +--- + +## When NOT to use this skill + +| Situation | Use instead | +|---|---| +| Debugging NaN loss or gradient explosion | `ddrs-systematic-debugging` | +| Adding a new data source format (new zarr reader, new fabric) | `ddrs-data-sources` | +| Understanding the sparse backward / autograd tape | `.claude/references/ddrs-burn-autograd.md` | +| Per-timestep routing math | `.claude/ARCHITECTURE.md` | +| CLI lifecycle (`ddrs plan`, `ddrs run`, `ddrs gc`) | `CLAUDE.md` §"ddrs CLI" | + +--- + +## Overview: config file anatomy + +A ddrs config is a single YAML file with six top-level sections: + +``` +mode / workflow / geodataset / device / seed / np_seed ← top-level scalars +data_sources: ← where inputs live on disk +experiment: ← training-loop hyperparameters +kan_head: ← KAN head architecture + which parameters it predicts +params: ← routing engine settings +testing: ← overlay applied in eval mode (overrides experiment: keys) +``` + +The Rust struct is `src/config.rs::Config`. Deserialization uses +`Config::from_yaml_file_with_mode(path, ConfigMode::Training|Testing)`. + +--- + +## Top-level scalars + +| Key | Type | Default | Notes | +|---|---|---|---| +| `mode` | `"training"` \| `"testing"` | `"training"` | Must agree with `workflow:` (see guard below). | +| `workflow` | `train` \| `eval` \| `train-and-test` | absent (None) | `train-and-test` runs both phases and computes the baseline comparison. | +| `geodataset` | `"merit"` | `"merit"` | Only value supported as of 2026-07-05. | +| `device` | integer | `0` | CUDA device ordinal. On multi-GPU hosts, pick a non-display GPU. | +| `seed` | integer | `42` | Controls KAN weight initialization. | +| `np_seed` | integer | `42` | Controls per-epoch gauge shuffle order. | + +**Guard:** `mode: training` requires `workflow ∈ {train, train-and-test}`. `mode: testing` requires `workflow: eval`. A contradiction is rejected at load time with a message containing `"conflicting top-level keys"`. + +--- + +## `data_sources:` section + +All data is read in place — no export step. Every path is a `PathBuf`. + +| Key | Required | Notes | +|---|---|---| +| `attributes` | Yes | NetCDF catchment attributes. Columns must match `kan_head.input_var_names`. `attributes` can be a single path OR a list of paths — they are feature-concatenated on COMID (NaN-filled for missing COMIDs, inner join on COMID dimension). Example: `[merit_global_attributes_v2.nc, merit_channel_attributes_v1.nc]`. | +| `streamflow` | Yes | dHBV2 lateral inflow Q'. Icechunk (`.ic`) for CONUS; zarr-v2 for global. | +| `observations` | Yes | USGS (or global) daily observed discharge; training targets. | +| `gages` | Yes | CSV with STAID and COMID columns. | +| `geospatial_fabric` | Conditional | `.shp`/`.dbf`/`.gpkg`; triggers managed adjacency build into `.ddrs/adjacency//`. | +| `geospatial_fabric_layer` | Optional | Layer name inside a multi-layer `.gpkg`; invalid for `.shp`/`.dbf`. | +| `conus_adjacency` | Conditional | Pre-built zarr. Must be paired with `gages_adjacency`. | +| `gages_adjacency` | Conditional | Pre-built zarr. Must be paired with `conus_adjacency`. | +| `aorc_precip` | Optional | Hourly AORC precip zarr v3 (`merit_unit_catchments.zarr`). Required when `kan_head.disaggregation.use_precip: true`. | + +**Adjacency rule (enforced at load time):** provide EITHER both `conus_adjacency` + `gages_adjacency`, OR `geospatial_fabric` (managed build). Providing only one of the two adjacency zarrs is rejected. Providing none of the three is rejected. + +**Production path (CONUS workstation, as of 2026-07-05):** +```yaml +data_sources: + attributes: /home/tbindas/projects/ddr/data/merit_global_attributes_v2.nc + geospatial_fabric: /projects/mhpi/data/MERIT/raw/continent/riv_pfaf_7_MERIT_Hydro_v07_Basins_v01_bugfix1.shp + streamflow: /mnt/ssd1/data/icechunk/merit_dhbv2_UH_retrospective.ic + observations: /mnt/ssd1/data/icechunk/usgs_daily_observations + gages: /home/tbindas/projects/ddr/references/gage_info/gages_3000.csv +``` + +--- + +## `experiment:` section (training mode) + +| Key | Type | Default | Production value | Notes | +|---|---|---|---|---| +| `batch_size` | integer | none | `64` | **Gauges** per mini-batch during training. Meaning shifts in `testing:` — see below. | +| `start_time` | `"YYYY/MM/DD"` | none | `"1981/10/01"` | Training window start. | +| `end_time` | `"YYYY/MM/DD"` | none | `"1995/09/30"` | Training window end (water year 1995). | +| `epochs` | integer | none | `5` | Total training epochs. | +| `rho` | integer \| null | none | `90` | Sequence length in days per mini-batch. Set `null` in testing overlay. | +| `shuffle` | bool | `false` | `true` | Re-shuffle gauge order each epoch (seeded by `np_seed`). | +| `warmup` | integer | none | `5` | Days excluded from loss at sequence start (routing spin-up). | +| `learning_rate` | map epoch→f32 | `{}` | `{1: 0.001, 3: 0.0005}` | Step decay; applies from that epoch onward. | +| `grad_clip_max_norm` | float \| absent | absent | `1.0` | Global gradient-norm clip. Omit to disable. | +| `checkpoint` | path \| absent | absent | absent | Directory path to resume from (e.g. `.ddrs/runs//checkpoints/epoch_5_mb_9`). | +| `state_cache` | path \| absent | absent | absent | Path to a continuous-run state zarr store (from `probe_zeta_gradient --mode state-cache`). Injects window-start routing states (hotstarted from a continuous run rather than cold zero-flow start). Reduces the hotstart-transient noise floor. Optional; leave blank for standard cold-start training. | +| `loss` | block \| absent | L1 (see below) | absent | Training objective. Omit for historical L1. | + +### `experiment.loss:` sub-block + +Omit the entire `loss:` block to use the historical L1 objective. Including the block does NOT change behavior if `kind: l1`. + +| Key | Type | Default | Notes | +|---|---|---|---| +| `kind` | `l1` \| `nnse-kge` \| `kge` | `l1` | `l1` = mean absolute error. `nnse-kge` = composite NNSE + KGE. `kge` = component-weighted KGE (r, alpha, beta terms individually weighted). | +| `nnse_weight` | float | `1.0` | Weight on `1 - NNSE` term (all non-L1 kinds). | +| `kge_weight` | float | `1.0` | Weight on `1 - KGE` Euclidean term (`nnse-kge` only). | +| `r_weight` | float | `1.0` | Weight on `(r-1)²` correlation term (`kge` kind only). | +| `alpha_weight` | float | `1.0` | Weight on `(alpha-1)²` variance ratio (`kge` kind only). This is the restoring force against MC over-attenuation. | +| `beta_weight` | float | `1.0` | Weight on `(beta-1)²` mean ratio (`kge` kind only). | +| `kge_clamp` | float | `10.0` | Per-gauge upper bound on weighted KGE-component sum before averaging. Prevents near-constant gauges from hijacking the batch gradient. | +| `eps` | float | `0.1` | Stabilizes variance/mean denominators. Matches DDR `hydrograph_loss` default. | + +**Why L1 and NSE both fail KGE:** L1 and NSE are both maximized when simulated variance is below observed (NSE optimum is at `alpha = r < 1`). This rewards MC for over-attenuating flood peaks — the diagnosed cause of KGE regression vs the summed-Q' baseline in CONUS runs (median KGE 0.723→0.701 while NSE improved 0.639→0.684). The `(alpha-1)²` term in `nnse-kge` / `kge` supplies a restoring gradient. + +--- + +## `testing:` section (eval-mode overlay) + +These keys **replace** the matching `experiment:` keys when `mode: testing` is loaded. Absent keys inherit from `experiment:`. + +| Key | Default in testing | Notes | +|---|---|---| +| `start_time` | `"1995/10/01"` | Eval window start (water year 1996). | +| `end_time` | `"2010/09/30"` | Eval window end. | +| `batch_size` | `15` | **DAYS** per evaluation chunk — semantic shift from training's gauges-per-batch. | +| `rho` | `null` | Explicitly clears sequence sampling (null is distinct from absent). | + +**CAUTION:** `batch_size` changes meaning between modes. Training: gauges. Testing: days. This is not a typo — it's in the YAML comments. + +--- + +## `kan_head:` section + +The KAN head architecture: `Linear(F,H) → KanLayer(H,H) × num_hidden_layers → Linear(H,P) → Sigmoid`. + +| Key | Type | Code default | Production value | Notes | +|---|---|---|---|---| +| `hidden_size` | integer | none | `21` | Hidden dimension H. | +| `num_hidden_layers` | integer | none | `2` | Inner KanLayer repetitions. ALL receive the SAME seed (DDR `kan.py` quirk, preserved for parity). | +| `grid` | integer | `5` | `50` | B-spline grid intervals per KAN edge (`num` in pykan). Production overrides the code default. | +| `k` | integer | `3` | `2` | B-spline order. DDR overrides pykan's default of 3 to 2 in production; keep 2 for parity. | +| `input_var_names` | list of strings | none | 10 attributes (see below) | Column names in `attributes` NetCDF. | +| `learnable_parameters` | list of strings | none | `[n, q_spatial, p_spatial]` | Parameters the KAN head emits. Must have matching entries in `params.parameter_ranges`. | +| `disaggregation` | block \| absent | absent | absent | Enables the daily→hourly disaggregation head (see sub-block below). Absent = flat repeat-24. | + +**Production `input_var_names` (10 attributes):** +``` +SoilGrids1km_clay, aridity, meanelevation, meanP, NDVI, +meanslope, log10_uparea, SoilGrids1km_sand, ETPOT_Hargr, Porosity +``` + +### `kan_head.disaggregation:` sub-block + +Presence of this block enables the learnable daily→hourly disaggregation head (`src/nn/disagg_head.rs`). Absence = flat `repeat-24` (backward-compatible default). + +| Key | Type | Default | Notes | +|---|---|---|---| +| `hidden_size` | integer | `16` | Hidden dimension of the disagg MLP. | +| `use_attributes` | bool | `true` | Condition on catchment attributes. | +| `use_precip` | bool | `false` | Condition on 72-h AORC precip window `[d-1, d, d+1]`. Requires `data_sources.aorc_precip`. | +| `use_temp` | bool | `false` | Condition on 72-h AORC temperature window. Also requires `data_sources.aorc_precip`. | + +**If `use_precip: true` and `data_sources.aorc_precip` is absent:** `MeritGagesDataset::open` errors at runtime (not at config load time). The missing precip source cannot silently degrade. + +--- + +## `params:` section (routing engine) + +| Key | Type | Code default | Production value | Notes | +|---|---|---|---|---| +| `sparse_solver` | `cpu` \| `cuda` | `cpu` | `cuda` | `cuda` uses cuSPARSE for the triangular solve. Falls back to `cpu` on non-CUDA backends with a WARN log. | +| `use_cuda_graphs` | bool | `false` | `true` | Capture the routing forward as a CUDA graph; faster replay each timestep. **See guards below.** | +| `use_leakance` | bool | `false` | `false` | Enable the GW–SW water-loss term. **See guards below.** Experimental as of 2026-07-05. | +| `leakance_losing_only` | bool | `true` | `true` | When true, applies `max(0, depth − d_gw)` clamp so zeta is zero for gaining reaches. Config-gated; added in Phase C. No-op when `use_leakance: false`. | +| `leakance_impervious_threshold` | float | `0.7` | `0.7` | Hard-zero mask: reaches with `corridor_impervious` attribute ≥ this threshold get zeta = 0 regardless of other params. No-op when `corridor_impervious` is absent from attributes. | +| `tau` | integer | `3` | `3` (not set in YAML) | Muskingum routing sub-step count. Rarely changed. | +| `log_space_parameters` | list of strings | `["p_spatial"]` | `["p_spatial"]` | Parameters whose range spans decades; KAN output is exp-scaled before routing. | +| `defaults` | map str→f32 | `{p_spatial: 21.0}` | `{p_spatial: 21.0}` | Fixed values for parameters NOT in `learnable_parameters`. | + +### `params.parameter_ranges:` sub-block + +Physical `[min, max]` each sigmoid-normalized KAN output maps onto. All defaults are defined in `src/config.rs::ParameterRanges::default()`. + +| Key | Default range | Log-space | Notes | +|---|---|---|---| +| `n` | `[0.015, 0.25]` | No | Manning's roughness coefficient. | +| `q_spatial` | `[0.0, 1.0]` | No | Leopold & Maddock width–depth exponent (`top_width = p·depth^q`). | +| `p_spatial` | `[1.0, 200.0]` | Yes | Leopold & Maddock width coefficient. In log-space by default. | +| `x_storage` | `[0.0, 0.5]` | No | Muskingum storage weight X. Only consumed when listed in `learnable_parameters`; otherwise routing uses constant 0.3. | +| `K_D` | `[1e-8, 1e-6]` | Yes (add to `log_space_parameters`) | Hydraulic exchange rate (1/s). Leakance only. Note: uppercase in YAML. | +| `d_gw` | `[-2.0, 2.0]` | No | Groundwater depth offset (m). Leakance only. | +| `leakance_factor` | `[0.0, 1.0]` | No | Dimensionless leakance scale. Leakance only. | + +### `params.attribute_minimums:` sub-block + +Physical floor applied during routing for numerical stability. + +| Key | Default | Units | +|---|---|---| +| `discharge` | `1.0e-4` | m³/s | +| `slope` | `1.0e-3` | m/m | +| `velocity` | `0.01` | m/s | +| `depth` | `0.01` | m | +| `bottom_width` | `0.01` | m | + +--- + +## Guards enforced at `Config::from_yaml_file` (load-time errors) + +| Guard | Trigger | Error message substring | +|---|---|---| +| Mode/workflow conflict | `mode: training` + `workflow: eval`, or `mode: testing` + `workflow: train` | `"conflicting top-level keys"` | +| Partial adjacency | Only one of `conus_adjacency` / `gages_adjacency` set | `"gages_adjacency\` is missing"` or `"conus_adjacency\` is missing"` | +| No adjacency sources | Neither adjacency zarrs nor `geospatial_fabric` | `"adjacency sources are missing"` | +| Fabric layer on non-gpkg | `geospatial_fabric_layer` set with `.shp`/`.dbf` fabric | `"geospatial_fabric_layer"` and `".gpkg"` | +| Leakance + CUDA graphs | `use_leakance: true` and `use_cuda_graphs: true` | `"use_leakance"` and `"use_cuda_graphs"` | + +--- + +## Production configs vs experimental configs + +| Config file | Status | Key differences from production | +|---|---|---| +| `config/merit_training.yaml` | Production template | `grid:50`, `k:2`, `use_cuda_graphs:true`, managed adjacency via `geospatial_fabric`, no leakance, no disaggregation, L1 loss | +| `config/experiments/leakance_hourly_on.yaml` | Experimental (2026-07-01, GO-marginal) | `use_leakance:true`, `use_cuda_graphs:false`, `K_D`/`d_gw`/`leakance_factor` in head, precip disaggregation enabled, explicit zarr adjacency paths | +| `config/experiments/leakance_daily_on.yaml` | Experimental (2026-07-01) | Same as `leakance_hourly_on.yaml` but NO disaggregation block, NO `aorc_precip` | +| `config/sources/conus.yaml` | Source group | CONUS workstation paths without AORC precip | +| `config/sources/conus-hourly.yaml` | Source group | CONUS + `aorc_precip: /mnt/ssd1/data/aorc/merit_unit_catchments.zarr` | +| `config/sources/global.yaml` | Source group | GPFS global paths | + +**Source groups** are text-spliced into `data_sources:` by `ddrs sources use `. They do not set `kan_head` or `params`. + +--- + +## Critical runtime traps + +### STALE-BINARY TRAP +`cargo build` does NOT update `~/.cargo/bin/ddrs`. After ANY change to `src/`, run: +```bash +cargo install --path . +# or faster if target/release is current: +cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs +``` +Self-check: current checkpoints are **directories** (`.ddrs/runs//checkpoints/epoch_E_mb_M/head.mpk`). Flat `.mpk` files mean a stale binary ran. + +### CUDA graphs mask NaN +`use_cuda_graphs: true` captures a finite forward pass graph. If a NaN appears in a subsequent forward (different inputs), the graph replays stale finite values — you get a finite loss with no error. To validate forwards, test with `use_cuda_graphs: false`. This is why `use_leakance: true` + `use_cuda_graphs: true` is rejected at config load time. + +### Leakance status (as of 2026-07-05) +The leakance 2×2 (forcing × leakance) returned a **GO-marginal** verdict: +- `|zeta| > 0.01 m³/s` on 10.4% of 64,892 eval reaches (meets ≥10% bar) +- Leakance helps under hourly forcing on the losing-stream subset (ΔNSE +0.0005, ΔKGE +0.0018, 55.5% of gauges improve) +- Leakance hurts under daily forcing (ΔNSE −0.0017, ΔKGE −0.0009, 35.6%) +- `K_D` pinned at ceiling `1e-6` (binding constraint; widening NOT recommended as of 2026-07-05) + +Diagnosis hypotheses (2026-07-02): +- H2 (head throttling): SUPPORTED +- H4 (gauge bias): SUPPORTED +- H5 (equifinality with `n`): SUPPORTED +- H1 (K_D box too narrow): REFUTED +- H3 (KAN capacity): REFUTED +- H6, H7: REFUTED + +**Leakance identifiability is NOT proven.** The positive-control synthetic recovery experiment (2026-07-04) FAILED: recovery ratio 0.009 vs ≥0.5 bar. Root cause: windowed training objective has ~130x hotstart-transient noise floor. Phase B (state-cache hotstart, ≤0.25 mean L1 noise floor target) is required before any identifiability claim. + +--- + +## How to add a new routing parameter (checklist) + +A "routing parameter" is a per-reach scalar the KAN head predicts and the MC solver consumes. Example: adding a new parameter `my_param`. + +- [ ] **1. Add to `ParameterRanges` struct** (`src/config.rs`): + ```rust + pub my_param: [f32; 2], + ``` + Add a default in `ParameterRanges::default()`. + +- [ ] **2. Add YAML key parsing** in `From for Params` (`src/config.rs`): + ```rust + if let Some(v) = r.parameter_ranges.get("my_param") { + p.parameter_ranges.my_param = *v; + } + ``` + +- [ ] **3. Add to `config/merit_training.yaml`** under `params.parameter_ranges:` (if it has a production-relevant range). + +- [ ] **4. Wire into routing** (`src/routing/mmc.rs` or a new module): consume `Params.parameter_ranges.my_param` in `setup_inputs` or `route_timestep`. Follow the `denormalize` pattern in `src/routing/utils.rs`. + +- [ ] **5. Add to `kan_head.learnable_parameters:`** in any experiment config that uses it. + +- [ ] **6. Add to `params.log_space_parameters:`** if the range spans decades. + +- [ ] **7. If log-space:** add to `log_space_parameters` list in `params.log_space_parameters` in YAML and ensure `denormalize` handles it. + +- [ ] **8. Write a gradient-exactness test** if the new parameter enters a custom backward op. Run: + ```bash + cargo test --test + cargo run --release --example compare_ddr_sandbox + ``` + The sandbox must still report `ABSOLUTE MATCH`. + +--- + +## How to add a new training-mode boolean flag (checklist) + +Example: adding `use_my_feature: bool` under `params:`. + +- [ ] **1. Add field to `Params` struct** (`src/config.rs`): + ```rust + pub use_my_feature: bool, + ``` + Add `use_my_feature: false` to `Params::default()`. + +- [ ] **2. Add to `ParamsRaw`** and parse in `From for Params`: + ```rust + // in ParamsRaw: + use_my_feature: Option, + // in From impl: + if let Some(b) = r.use_my_feature { p.use_my_feature = b; } + ``` + +- [ ] **3. Add validation if needed** (`validate_*` functions in `src/config.rs`). Call from `from_yaml_file_with_mode`. Validation errors must include the YAML key name and the reason. + +- [ ] **4. Add a test** in the `#[cfg(test)]` block at the bottom of `src/config.rs` covering: flag defaults to false, flag parses true, any guard is rejected. + +- [ ] **5. Thread through call sites**: training bootstrap (`src/training/bootstrap.rs`), eval (`src/cli/eval.rs`), and any other entrypoints that construct `MuskingumCunge` or read `Params`. + +- [ ] **6. Document in `config/merit_training.yaml`** as a commented-out key with explanation if it has production relevance. + +--- + +## Quick reference: minimal leakance-on config diff + +Starting from `config/merit_training.yaml`, three changes activate leakance: + +```yaml +# 1. Under params: +params: + use_leakance: true + use_cuda_graphs: false # REQUIRED — leakance + cuda_graphs is rejected at load time + parameter_ranges: + K_D: [1.0e-8, 1.0e-6] + d_gw: [-2.0, 2.0] + leakance_factor: [0.0, 1.0] + log_space_parameters: + - p_spatial + - K_D # ADD — K_D range spans decades + +# 2. Under kan_head.learnable_parameters: + learnable_parameters: + - n + - q_spatial + - p_spatial + - K_D + - d_gw + - leakance_factor +``` + +See `config/experiments/leakance_hourly_on.yaml` for the full working example. + +--- + +## Provenance and maintenance + +Ground truth files read to produce this skill (verify before re-editing): +```bash +# Config struct (all fields, defaults, guards): +grep -n "pub use_" /home/tbindas/projects/ddrs/src/config.rs + +# Production defaults verified from: +grep -n "fn default" /home/tbindas/projects/ddrs/src/config.rs + +# Production YAML (single source of truth for hyperparameter values): +cat /home/tbindas/projects/ddrs/config/merit_training.yaml + +# Leakance experiment config: +cat /home/tbindas/projects/ddrs/config/experiments/leakance_hourly_on.yaml + +# Load-time guard tests (exhaustive): +grep -n "#\[test\]" /home/tbindas/projects/ddrs/src/config.rs | head -40 +``` + +Config struct location: `/home/tbindas/projects/ddrs/src/config.rs` +Production template: `/home/tbindas/projects/ddrs/config/merit_training.yaml` +Experiment configs: `/home/tbindas/projects/ddrs/config/experiments/` +Source group configs: `/home/tbindas/projects/ddrs/config/sources/` + +Volatile facts date-stamped: 2026-07-05. Re-verify leakance status, CONUS metric baselines, and K_D ceiling diagnosis before citing in new experiments. diff --git a/.claude/skills/ddrs-debugging-playbook/SKILL.md b/.claude/skills/ddrs-debugging-playbook/SKILL.md new file mode 100644 index 0000000..20890dc --- /dev/null +++ b/.claude/skills/ddrs-debugging-playbook/SKILL.md @@ -0,0 +1,543 @@ +--- +name: ddrs-debugging-playbook +description: "Use when a ddrs run produces wrong results, silent failures, metric regressions, NaN loss, stale checkpoints, V1 mismatch, leakance anomalies, KAN head divergence, adjacency errors, data-source alignment issues, or any symptom that costs debug time. Also use before attributing a result to a code bug — many apparent bugs are operator error (stale binary, wrong fixture, config contradiction)." +--- + +# ddrs debugging playbook + +**Audience:** Sonnet-class AI or mid-level ML engineer who knows PyTorch but not Rust/BURN. +**Voice:** imperative runbook. Copy-paste every command. Verify before claiming. + +--- + +## Glossary (terms used throughout) + +| Term | Meaning | +|---|---| +| **ddrs** | BURN-based Rust port of DDR (Python/PyTorch Muskingum-Cunge routing solver) | +| **DDR** | Python reference: `~/projects/ddr/`. The gold standard for numerical parity | +| **BURN** | Rust deep-learning framework (version 0.21 in this project) | +| **MC solver** | Muskingum-Cunge routing: converts upstream + lateral inflow to routed discharge per reach per timestep | +| **V1 / ABSOLUTE MATCH** | Regression gate: `compare_ddr_sandbox` max abs diff < 1e-3 m³/s vs DDR | +| **KAN head** | The neural network head (`rskan::KanLayer` v0.1.3): maps catchment attributes → routing parameters | +| **f32 invariant** | All tensors in the routing core must stay float32; f64/bf16 casts break DDR parity | +| **lower-triangular adjacency** | The CSR sparse pattern has `rows[k] >= cols[k]`; the forward-sub solver requires this | +| **sparse backward** | Hand-written O(nnz) `CsrSolveOp: Backward` in `src/sparse.rs`; must not be replaced by autograd unrolling | +| **leakance** | Experimental GW–SW water-loss term (`src/routing/leakance.rs`); off by default | +| **zeta** | The per-reach per-timestep leakance flux (m³/s): `zeta = leakance_factor · area_z · K_D · (depth − d_gw)` | +| **summed-Q baseline** | No-routing reference: per-gauge sum of upstream divide Qr. CONUS: median NSE 0.689 / KGE 0.723 (as of 2026-07-05) | +| **CUDA Graphs** | CUDA kernel-replay optimization (`use_cuda_graphs: true`); incompatible with leakance; masks NaN loss | +| **Q'** | Lateral inflow (m³/s) from an upstream forcing model (DHBV, LSTM, etc.) | +| **worktree** | Git worktree at a branch tip — used for experimental campaigns without touching main tree | + +--- + +## When NOT to use this skill + +- You want to understand the routing math or architecture → read `.claude/ARCHITECTURE.md` and `.claude/references/ddrs-algorithm.md` +- You want to port or verify a new feature against DDR → use skill `ddrs-comparing-to-ddr` (`.claude/references/ddrs-comparing-to-ddr.md`) +- You want to set up a new experiment from scratch → read `CLAUDE.md` §"ddrs CLI" +- You are doing leakance identifiability research → see `docs/2026-07-02-leakance-diagnosis-findings.md` for the completed hypothesis battery + +--- + +## Part 1 — Symptom → triage table + +Scan this table first. Each row points to a Part 2 entry with the full story and fix. + +| Symptom | Most likely trap | Go to | +|---|---|---| +| Two runs that differ in config produce byte-identical predictions | Stale installed binary | [T1](#t1-stale-binary-trap) | +| Checkpoint files are flat `.mpk` (not a directory) | Stale binary | [T1](#t1-stale-binary-trap) | +| `manifest.json` shows current git SHA but behavior looks old | Stale binary (SHA stamps from `.git` at runtime, not the binary) | [T1](#t1-stale-binary-trap) | +| `ddrs run` silently ignores `disaggregation:` block | Stale binary (pre-disagg binary ignores unknown serde fields) | [T1](#t1-stale-binary-trap) | +| `compare_ddr_sandbox` reports diff > 1e-3 m³/s | V1 regression | [T2](#t2-v1-regression) | +| `compare_ddr_sandbox` fails after regenerating fixtures | Wrong DDR reference tree | [T2](#t2-v1-regression) | +| Loss goes NaN, but only with `use_cuda_graphs: true` | CUDA Graphs mask NaN | [T3](#t3-cuda-graphs-mask-nan) | +| Loss is finite but suspiciously constant across steps | CUDA Graphs returning stale capture | [T3](#t3-cuda-graphs-mask-nan) | +| Config parse error: "`use_leakance: true` requires `use_cuda_graphs: false`" | Config contradiction (intentional rejection) | [T4](#t4-leakance-config-contradictions) | +| Leakance run still uses CUDA graphs silently | Missing `use_cuda_graphs: false` in config | [T4](#t4-leakance-config-contradictions) | +| K_D pinned at ceiling (100% of reaches) | K_D box is binding — or head throttling (H2); see diagnosis | [T5](#t5-leakance-parameter-collapse-or-ceiling) | +| K_D collapsed to floor (sub-1e-8) | Replicates DDR's original revert failure; check forcing resolution | [T5](#t5-leakance-parameter-collapse-or-ceiling) | +| Gradient check fails on leakance op | Regression in `src/routing/leakance.rs` backward | [T6](#t6-leakance-gradient-correctness) | +| `zeta_accum` test fails | Accumulated zeta not matching headwater identity | [T6](#t6-leakance-gradient-correctness) | +| KAN head shape or init diverges from DDR | rskan version bump or inter-block ReLU accidentally re-added | [T7](#t7-kan-head-divergence) | +| KAN parity tests fail after `Cargo.toml` rskan bump | Fixture needs regeneration | [T7](#t7-kan-head-divergence) | +| Adjacency test fails or topological ordering wrong | `rows[k] < cols[k]` somewhere; lower-triangular invariant violated | [T8](#t8-adjacency-invariant) | +| `ddrs plan` hangs or errors on adjacency build | Bad fabric path or multi-layer gpkg needs `geospatial_fabric_layer` | [T8](#t8-adjacency-invariant) | +| Training NSE well below summed-Q baseline | Routing not earning its keep; loss or gradient issue | [T9](#t9-metric-regression-below-baseline) | +| KGE lower than baseline in every config | Expected — L1 loss penalizes variance; this is known behavior | [T9](#t9-metric-regression-below-baseline) | +| Hourly run produces same predictions as daily run | Stale binary (pre-disagg) or `aorc_precip` source missing | [T1](#t1-stale-binary-trap), [T10](#t10-disaggregation-no-op) | +| `MeritGagesDataset::open` errors with `use_precip: true` | `aorc_precip` source not configured | [T10](#t10-disaggregation-no-op) | +| Checkpoint resume trains zero batches | `experiment.epochs` not raised above checkpoint epoch | [T11](#t11-checkpoint-resume-issues) | +| Resumed run drifts from uninterrupted run | Expected: weights stored as f16 (CompactRecorder) | [T11](#t11-checkpoint-resume-issues) | +| `ddrs run --strict` exits with code 4 | Source fingerprint drift vs `.ddrs/sources.lock` | [T12](#t12-source-lock-drift) | +| Recoverability / identifiability experiment fails | Hotstart transient noise floor issue (Phase B not yet met) | [T13](#t13-leakance-identifiability-status) | + +--- + +## Part 2 — Trap stories and fixes + +### T1: Stale binary trap + +**Story (2026-07-01).** The leakance × hourly 2×2 experiment produced two runs that were byte-identical despite different configs (one with hourly disaggregation, one without). The `manifest.json` showed the current git SHA `2cdd341` — which made it look like a code bug. Root cause: `~/.cargo/bin/ddrs` had mtime 2026-06-03, predating both the disaggregation feature (June 19) and leakance (June 29). The installed binary silently ignored the `disaggregation:` config block (serde ignores unknown fields) and wrote flat `.mpk` checkpoints instead of the current directory format. + +**Discriminating test.** Check checkpoint format: +```bash +# Current binaries write DIRECTORIES: +ls .ddrs/runs//checkpoints/ +# → epoch_5_mb_9/ (directory = current binary) +# → epoch_5_mb_35.mpk (flat file = stale binary) +``` + +Check binary age: +```bash +stat ~/.cargo/bin/ddrs | grep Modify +# Should be >= your last src/ change date +``` + +**Fix (canonical):** +```bash +cargo install --path . +# OR faster if target/release/ is already built: +cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs +# OR bypass the installed binary entirely: +cargo run --release --bin ddrs -- run --workflow train-and-test +``` + +**Rule:** `cargo build` does NOT update `~/.cargo/bin/ddrs`. Re-install after every `src/` change before invoking `ddrs` by name. + +**Head size cross-check (CONUS, as of 2026-07-05):** +- No disagg, no leakance: ~103,459 B +- Disagg only: ~107,178 B +- Disagg + leakance (3 extra output cols): ~107,320 B + +--- + +### T2: V1 regression + +**What V1 is.** `examples/compare_ddr_sandbox` replays DDR's 5-reach RAPID2 sandbox through ddrs's MC solver. The threshold is `max abs diff < 1e-3 m³/s`. A passing run prints: +``` +verdict: ABSOLUTE MATCH (max abs < 1e-3 m³/s) +``` +Typical passing value is ~1.5e-5 m³/s — two orders of magnitude under the threshold. + +**Run it:** +```bash +mkdir -p output # required — the example does not mkdir -p +cargo run --release --example compare_ddr_sandbox +# Also test the CUDA + graph-capture path: +DDRS_FORCE_GRAPHS=1 cargo run --release --example compare_ddr_sandbox +``` + +**Triage checklist when V1 fails:** + +1. Inspect `output/ddrs_vs_ddr.csv` — which reaches are worst? Single bad reach = geometry/parameter bug. Global failure = solver or kernel issue. + +2. Check if fixtures are stale: + ```bash + cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/export_ddr_sandbox.py + cd ~/projects/ddrs && git diff fixtures/sandbox/ + ``` + **WARNING:** Only the local `~/projects/ddr` checkout is valid. The unpushed `geometry/trapezoidal.py` work is not in any public DDR commit. Regenerating from a clean DDR clone produces ~0.55 m³/s divergence at every ddrs commit — that is a wrong-reference artifact, NOT a port bug (as of 2026-06-06). + +3. Audit recent changes to the only paths that affect V1: + ```bash + git log -p -- src/routing/ src/geometry.rs src/sparse/ + ``` + +4. Look for precision leaks: + ```bash + grep -rn "f64\|bf16\|cast\|to_dtype" src/routing/ src/geometry.rs src/sparse/ + ``` + Any cast away from f32 in these paths breaks DDR parity. + +5. Cross-check with gradcheck: + ```bash + cargo test --test sparse_gradcheck + ``` + If gradcheck fails too, the algorithm changed. If only V1 fails, it's a kernel-ordering or arithmetic-fusion difference. + +**The threshold is non-negotiable.** Never relax `1e-3 m³/s`. Never declare "good enough." + +--- + +### T3: CUDA Graphs mask NaN + +**What happens.** When a forward pass produces NaN and `use_cuda_graphs: true`, the CUDA graph replays the stale pre-NaN capture instead of the live computation. The loss appears finite. You get silently wrong results with no error. + +**Confirmed behavior (as of 2026-07-05):** `use_cuda_graphs: true` returns stale finite loss on a NaN forward. + +**Discriminating test:** +```bash +# Reproduce the stale-loss symptom: +# Run once with cuda_graphs on vs off, inject a NaN input, compare loss values. +# If cuda_graphs=true gives finite loss and cuda_graphs=false gives NaN → confirmed. +``` + +**Fix:** Disable CUDA graphs when debugging any NaN or suspiciously-constant loss: +```yaml +# in ddrs.yaml or experiment config: +params: + use_cuda_graphs: false +``` + +**Rule:** Always debug loss anomalies with `use_cuda_graphs: false`. Re-enable only after confirming the forward is NaN-free. + +--- + +### T4: Leakance config contradictions + +**What happens.** Two config errors involving leakance are caught at load time. + +**Error 1 — leakance + CUDA graphs:** +``` +params: `use_leakance: true` requires `use_cuda_graphs: false` +``` +CUDA Graphs cannot capture the extra leakance kernel without a separate capture path. This is an intentional hard rejection in `src/config.rs:626`. + +**Fix:** +```yaml +params: + use_leakance: true + use_cuda_graphs: false # REQUIRED when use_leakance is true +``` + +**Error 2 — leakance parameters missing from KAN head.** +If `params.use_leakance: true` but `K_D`, `d_gw`, `leakance_factor` are not in `kan_head.learnable_parameters`, the head emits no leakance parameters and the routing silently has no exchange. No error is thrown — the leakance term gets a zero or garbage input. + +**Complete leakance config checklist:** +```yaml +params: + use_leakance: true + use_cuda_graphs: false + parameter_ranges: + K_D: [1.0e-8, 1.0e-6] # log-space; hydraulic exchange rate 1/s + d_gw: [-2.0, 2.0] # groundwater depth offset, m + leakance_factor: [0.0, 1.0] # dimensionless scale + +kan_head: + learnable_parameters: + - K_D + - d_gw + - leakance_factor + # ... plus your routing params (n, q_spatial, etc.) +``` + +--- + +### T5: Leakance parameter collapse or ceiling + +**Two failure modes:** + +**Mode A — K_D at ceiling (100% of reaches).** +Observed in both leakance-ON arms of the 2026-07-01 2×2 (hourly and daily). +- `K_D` median log10 = −5.999, IQR = 3.6e-4 (essentially a delta function at the `1e-6` upper bound). +- This is NOT the K_D box clipping the flux. Diagnosis (2026-07-02) showed median in-box utilization is only 3.4% — the optimizer maxes the rate constant and then throttles the product via the driving head (`d_gw` learned near typical depths, so `depth − d_gw ≈ 0`). +- **K_D widening is NOT recommended** — the Phase-3 gate failed because H1 (structural ceiling) was REFUTED. Root cause: H2 (head throttling) + H4 (gauge bias) + H5 (equifinality under daily forcing). + +**Mode B — K_D at floor (collapse to sub-1e-8).** +This replicates DDR's original revert failure (sub-0.01 m³/s exchange, physically negligible). If you see this, check: +- Is daily forcing being used? Under flat-daily forcing the depth dynamic range is too small for `zeta ∝ (depth − d_gw)` to be identifiable. +- Is hourly disaggregation actually running? Verify the binary is current (T1) and `aorc_precip` source is configured (T10). + +**Discriminating check after any leakance run:** +```bash +cargo build --release --bin dump_parameters +target/release/dump_parameters \ + --config \ + --checkpoint .ddrs/runs//checkpoints/epoch_E_mb_M/head \ + --output /tmp/kp.nc 2>&1 | grep -E "K_D|leakance_factor|d_gw|frac@" +``` +Expected for a non-collapsed run: `K_D` interior or at ceiling (not floor), `leakance_factor` interior (0.1–0.5), `d_gw` spatially varying. + +--- + +### T6: Leakance gradient correctness + +**Guard tests — run all four after any change to `src/routing/leakance.rs` or `mmc_op.rs`:** + +```bash +cargo test --test leakance_gradcheck # analytical ≈ finite-difference (8 params) +cargo test --test leakance_off_parity # byte-identical to no-leakance when off (3 tests) +cargo test --test zeta_accum # accumulated zeta == headwater identity +cargo run --release --example compare_ddr_sandbox # V1 must still pass +``` + +**If `leakance_gradcheck` fails:** The analytical backward in `TimestepLeakanceOp: Backward` is wrong. Compare against `src/routing/leakance.rs` math: `zeta = leakance_factor · area_z · K_D · (depth − d_gw)` where `area_z = (p · depth)^q_eps · length`. All partial derivatives are straightforward products/chains; check each of the 8 inputs. + +**If `zeta_accum` fails:** The accumulator in `evaluate` is not recomputing from the same primitives the backward used. The test verifies the headwater identity `q_no_leak[0] − q_leak[0] == zeta[0]`. + +**If `leakance_off_parity` fails:** The leakance gating is broken — the `None` path is not byte-identical to a run without leakance compiled in. + +--- + +### T7: KAN head divergence + +**Architecture (must not change without explicit intent):** +``` +Linear(F, H) → KanLayer(H, H) × num_hidden_layers → Linear(H, P) → Sigmoid +``` +- No inter-block ReLU. DDR's `kan.py` has none; adding one breaks parity. +- All `num_hidden_layers` inner KanLayers get the SAME initialization seed (DDR `kan.py:24-34` quirk — preserved for parity). +- rskan version: `v0.1.3` (as of 2026-07-05). Pinned in `Cargo.toml:27`. + +**Parity test suite:** +```bash +cargo test --features fixtures \ + --test kan_head_init_repro \ + --test kan_head_init_parity \ + --test kan_head_fixture_forward \ + --test kan_head_fixture_backward +``` + +**If tests fail after an rskan bump:** The fixtures need regeneration: +```bash +cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/dump_kan_weights.py +cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/dump_kan_forward.py +``` +Then re-run the test suite. If it still fails, the rskan API changed in a parity-breaking way — audit the diff and decide whether to update the DDR reference or roll back the bump. + +**If tests fail without an rskan bump:** Check for accidental re-introduction of inter-block ReLU in `src/nn/kan_head.rs`. + +--- + +### T8: Adjacency invariant + +**The invariant:** The CSR adjacency pattern must be lower-triangular: every nonzero at `(row, col)` must have `row >= col`. The forward-substitution solver reads rows in order and assumes all upstream contributions are already resolved. + +**Test:** +```bash +cargo test data_zarr_store::conus_adjacency_loads_real_merit_zarr +# Also: +cargo test --test adjacency_parity +``` + +**If adjacency build fails during `ddrs plan`:** +- Check the fabric path exists and is readable. +- For `.gpkg` files with multiple layers, set `geospatial_fabric_layer:` in config. +- The builder reads only the attribute table (`.dbf` or gpkg attributes) — never the geometry. Check `geospatial_fabric` points to the right file. +- On a fresh worktree, `.ddrs/adjacency/` does not exist — `ddrs plan` builds it on first run (~10 s for CONUS `.dbf`). + +**If topological ordering is wrong:** The managed builder replicates petgraph's deterministic DFS finish-time order. Check `src/adjacency/build.rs::topological_sort` against the engine's version. + +--- + +### T9: Metric regression below baseline + +**Baseline numbers (CONUS, as of 2026-07-05):** +| config | median NSE | median KGE | gauges | +|---|---|---|---| +| Summed-Q′ baseline, same-run (2,365-gauge eval set) | 0.678 | 0.717 | 2365 | +| Best result: precip-disagg + L1 | 0.715 | 0.711 | 2365 | + +**Critical known behavior:** KGE does NOT beat the summed-Q baseline in any trained config as of 2026-07-05. NSE beats it (+0.037 with precip disagg). The KGE regression is structural: the L1 loss maximizes at simulated variance below observed (α < 1), rewarding over-attenuation. The whole KGE drop is in the `α = σ_sim/σ_obs` term. + +**Triage if NSE is far below baseline:** +1. Is the binary current? (T1) +2. Is the loss descending? Check `run.log` for epoch-mean L1. +3. Is CUDA graphs masking NaN? (T3 — disable and re-check) +4. Are gauge batch sizes reasonable? Too few gauges per batch → noisy gradient. +5. Is the data source correct? `streamflow resolution: Daily|Hourly` is logged at dataset open — verify it. + +**To improve KGE above baseline:** Switch to `experiment.loss.kind: nnse-kge`. The `(α-1)²` term in KGE provides the restoring gradient. This requires explicit config: +```yaml +experiment: + loss: + kind: nnse-kge + nnse_weight: 1.0 + kge_weight: 1.0 +``` + +--- + +### T10: Disaggregation no-op + +**Symptoms:** +- Hourly and daily runs produce byte-identical predictions. +- Head file size is ~103,459 B (no-disagg size) even when config has `disaggregation: ...`. + +**Root causes (in priority order):** + +1. **Stale binary** — most likely. See T1. The pre-disagg binary silently ignores the `disaggregation:` block. + +2. **Missing `aorc_precip` source.** The AORC precip zarr at `/mnt/ssd1/data/aorc/merit_unit_catchments.zarr` must be in `data_sources:`. Without it, `MeritGagesDataset::open` errors when `use_precip: true`. Check: + ```bash + ddrs sources list # '*' marks active group + # conus-hourly group includes aorc_precip + ddrs sources use conus-hourly + ``` + +3. **`use_precip: false` in config.** The `aorc_precip` source must be present AND `kan_head.disaggregation.use_precip: true` must be set. The source group (`conus-hourly`) splices in the source; the disagg block must be in the experiment config separately. + +**Verification after a fix:** +```bash +# Binary check: head file should be ~107,320 B (disagg + leakance) or ~107,178 B (disagg only) +ls -la .ddrs/runs//checkpoints/epoch_5_mb_9/head.mpk + +# Dataset log at run start: +grep "AORC precip store" .ddrs/runs//run.log +# Should show: "AORC precip store: 290878 catchments" + +# Forcing verification: eval predictions should differ between hourly and daily runs: +md5sum .ddrs/runs//eval/predictions.zarr/predictions/0.0 +md5sum .ddrs/runs//eval/predictions.zarr/predictions/0.0 +# These must NOT be identical +``` + +--- + +### T11: Checkpoint resume issues + +**Resume requires three files in a directory:** +``` +.ddrs/runs//checkpoints/epoch_E_mb_M/ + head.mpk # KAN weights (f16, CompactRecorder) + optim.mpk # Adam moments (f16) + state.json # epoch, next mini-batch, rng state, sampler permutation + cursor +``` + +**Resume trains zero batches:** `experiment.epochs` is at or below the checkpoint epoch. Fix: raise `experiment.epochs` past `E` in `ddrs.yaml`. + +**Resumed trajectory drifts from uninterrupted run:** Expected. Weights and moments are stored as f16 (`CompactRecorder = HalfPrecisionSettings`). The resumed trajectory is numerically valid but will not be bit-identical to an uninterrupted run. + +**`dump_parameters --checkpoint` path gotcha:** Pass the HEAD BASE, not the directory: +```bash +# CORRECT (head base — CompactRecorder appends .mpk): +target/release/dump_parameters --checkpoint .ddrs/runs//checkpoints/epoch_5_mb_9/head ... +# WRONG (directory — will fail to find head.mpk): +target/release/dump_parameters --checkpoint .ddrs/runs//checkpoints/epoch_5_mb_9 ... +``` + +--- + +### T12: Source lock drift + +**What happens.** `ddrs run --strict` exits with code 4 when the data-source fingerprints in `.ddrs/sources.lock` differ from the current `ddrs.yaml`. This preserves evidence; re-locking would overwrite it. + +**Fix (normal):** +```bash +ddrs plan # re-locks sources.lock to match current ddrs.yaml +ddrs run --workflow +``` + +**Fix (investigate first):** +```bash +cat .ddrs/sources.lock # shows last-locked fingerprints +# Compare against current data_sources: paths in ddrs.yaml +# If a path moved or a store was updated, decide whether to re-plan or roll back +``` + +--- + +### T13: Leakance identifiability status (as of 2026-07-05) + +**This section describes an active research limitation — not a bug to fix.** + +**Positive control experiment (2026-07-04, worktree):** A synthetic recoverability test was run to check whether the gradient path can recover a known planted zeta through gauged-only observations. The experiment FAILED: recovery ratio 0.009 vs the >=0.5 bar. Root cause: the windowed training objective has a hotstart-transient noise floor approximately 130× larger than the leakance signal. + +**Implication:** Leakance identifiability is NOT proven. The 2×2 GO-marginal verdict (leakance helps skill on the losing-stream subset under hourly forcing) stands, but the mechanism cannot be confirmed as genuine GW–SW exchange recovery until Phase B is complete. + +**Phase B objective (NOT YET MET as of 2026-07-05):** noise floor <= 0.25 mean L1 (i.e., <= 10% of a converged run's loss). Requires a state-cache hotstart to eliminate the transient. Do not make identifiability claims until Phase B passes. + +**Gradient probe results (2026-07-03, worktree):** +- P1 (gradient starvation to leakance params): REFUTED +- P3 (detectability — signal vs 5% obs band): NO-GO, signal 53× smaller than detectability threshold + +**Summary of leakance diagnosis verdicts (as of 2026-07-02):** + +| Hypothesis | Verdict | Key evidence | +|---|---|---| +| H1: K_D box clips zeta | REFUTED | Median utilization 3.4%; 71.5% of reaches CAN exceed 0.01 m³/s in-box | +| H2: Driving-head starvation | SUPPORTED | Median head 0.021 m; 47% of reaches gaining at eval-window mean | +| H3: KAN variance collapse | REFUTED | Max Spearman(param, attribute) = 0.71 (strong spatial structure) | +| H4: Gauge bias / gradient starvation | SUPPORTED | zeta–uparea ρ +0.76; gauged 11× ungauged median zeta; dry/wet ratio inverted | +| H5: Equifinality with routing params | SUPPORTED (daily only) | Daily Δn = +0.012 (0.59 IQR); hourly Δn nil | +| H6: Wrong yardstick (absolute bar) | REFUTED | Fractional loss agrees: 8.4% lose >1% of local flow | +| H7: Model-form error (d_gw bounds) | REFUTED | 0.0% of d_gw at bounds in any aridity tercile | + +**Do not run K_D widening.** The Phase-3 gate for K_D widening FAILED because H1 was REFUTED. The constraint is the signal, not the box. + +--- + +## Part 3 — Pre-flight checklist before any training run + +Use this before starting a new experiment to prevent the most common traps: + +- [ ] `stat ~/.cargo/bin/ddrs` — mtime is newer than your last `src/` change +- [ ] `ddrs sources list` — active group (`*`) matches intended dataset +- [ ] `ddrs plan` — no source drift warnings; `mode:` and `workflow:` agree +- [ ] Config leakance consistency: if `use_leakance: true`, confirm `use_cuda_graphs: false` and all three params in `kan_head.learnable_parameters` +- [ ] If hourly disagg: config has `aorc_precip:` source AND `kan_head.disaggregation.use_precip: true` +- [ ] If resuming: `experiment.epochs` is greater than the checkpoint epoch; checkpoint path ends at `head` base (not the directory) +- [ ] If touching `src/routing/`, `src/geometry.rs`, or `src/sparse/`: run `cargo run --release --example compare_ddr_sandbox` and confirm ABSOLUTE MATCH + +--- + +## Part 4 — Quick-reference test commands + +```bash +# V1 regression gate (routing core, geometry, sparse): +mkdir -p output && cargo run --release --example compare_ddr_sandbox + +# V1 on CUDA + graph-capture path: +DDRS_FORCE_GRAPHS=1 cargo run --release --example compare_ddr_sandbox + +# Leakance gradient correctness: +cargo test --test leakance_gradcheck +cargo test --test leakance_off_parity +cargo test --test zeta_accum + +# Sparse backward correctness: +cargo test --test sparse_gradcheck + +# KAN head parity vs DDR: +cargo test --features fixtures \ + --test kan_head_init_repro \ + --test kan_head_init_parity \ + --test kan_head_fixture_forward \ + --test kan_head_fixture_backward + +# Adjacency ordering and builder parity: +cargo test data_zarr_store::conus_adjacency_loads_real_merit_zarr +cargo test --test adjacency_parity + +# All lib unit tests: +cargo test --lib + +# Full test suite: +cargo test +``` + +--- + +## Provenance and maintenance + +Ground truth for this skill (re-read these files to verify facts remain current): + +```bash +# V1 / comparing-to-DDR reference: +cat /home/tbindas/projects/ddrs/.claude/references/ddrs-comparing-to-ddr.md + +# Stale-binary trap story + leakance 2x2 re-run: +cat /home/tbindas/projects/ddrs/docs/2026-07-01-leakance-hourly-experiment-handoff.md + +# 2x2 findings (all four arms, GO verdict): +cat /home/tbindas/projects/ddrs/docs/2026-07-01-leakance-hourly-findings.md + +# Low-zeta diagnosis (H1–H7 hypothesis verdicts): +cat /home/tbindas/projects/ddrs/docs/2026-07-02-leakance-diagnosis-findings.md + +# Config rules (invariants, leakance enable, CLI lifecycle): +cat /home/tbindas/projects/ddrs/CLAUDE.md + +# CUDA graphs mask NaN (memory note): +cat /home/tbindas/projects/ddrs/.claude/memories/cuda-graphs-mask-nan.md + +# Re-verify rskan version: +grep rskan /home/tbindas/projects/ddrs/Cargo.toml + +# Re-verify config leakance rejection: +grep -n "use_leakance.*use_cuda_graphs\|use_cuda_graphs.*use_leakance" \ + /home/tbindas/projects/ddrs/src/config.rs +``` diff --git a/.claude/skills/ddrs-dev/SKILL.md b/.claude/skills/ddrs-dev/SKILL.md new file mode 100644 index 0000000..f538a1e --- /dev/null +++ b/.claude/skills/ddrs-dev/SKILL.md @@ -0,0 +1,54 @@ +--- +name: ddrs-dev +description: Use at the start of any ddrs task to route to the right sub-skill. Load this first when the task type is unclear, when onboarding to the project, or when multiple skill areas may apply. +--- + +# ddrs skill index + +**Project:** Differentiable Muskingum-Cunge routing solver — BURN 0.21 Rust port of DDR (Python/PyTorch). Gradient-exact. Trains a KAN head to emit per-reach hydraulic parameters. + +**Live research goal (as of 2026-07-06):** Selective-equifinality paper — train on 4 NH inflow sources (daily-lstm, hourly-lstm, dHBV2.0-lumped, dHBV2.0-UH), compare convergence of geometry (identifiable) vs Manning's n (equifinal). Paper draft: `/home/tbindas/projects/ddr_equifinality/paper.tex`. + +--- + +## Route to the right skill + +| I need to… | Use skill | +|---|---| +| Understand what changes are safe to merge | `ddrs-change-control` | +| Debug a failed / wrong-result run | `ddrs-debugging-playbook` | +| Check whether a problem was already solved | `ddrs-failure-archaeology` | +| Understand a load-bearing design decision | `ddrs-architecture-contract` | +| Understand hydrology concepts (NSE/KGE, routing, equifinality) | `ddrs-hydrology-reference` | +| Add, change, or audit a YAML config key | `ddrs-config-and-flags` | +| Set up or rebuild the dev environment | `ddrs-build-and-env` | +| Install the CLI, run a train/eval job, resume a checkpoint | `ddrs-run-and-operate` | +| Measure a result, interpret zeta/gradient/metric diagnostics | `ddrs-diagnostics-and-tooling` | +| Decide whether a change is safe / what tests to run | `ddrs-validation-and-qa` | +| Write a findings doc, spec, plan, or paper section | `ddrs-docs-and-writing` | +| Make an external claim about novelty or results | `ddrs-external-positioning` | +| Plan or execute the selective-equifinality experiment | `ddrs-identifiability-campaign` | +| Verify gradient correctness / design a controlled ablation | `ddrs-proof-and-analysis-toolkit` | +| Identify the next high-value research direction | `ddrs-research-frontier` | +| Design an experiment / enforce evidence standards | `ddrs-research-methodology` | + +--- + +## Critical facts every agent must know + +- **Stale binary trap:** `cargo build` does NOT update `~/.cargo/bin/ddrs`. After any `src/` change: `cargo install --path .` +- **V1 gate:** `cargo run --release --example compare_ddr_sandbox` must print `ABSOLUTE MATCH` (max abs < 1e-3 m³/s) after any change to `src/routing/`, `src/geometry.rs`, or `src/sparse.rs`. +- **CUDA graphs mask NaN:** validate new forward paths with `use_cuda_graphs: false` before enabling graphs. +- **Leakance is CLOSED (NO-GO, 2026-07-06):** do not re-open without reading `docs/2026-07-06-leakance-nogo-scientific-summary.md`. +- **Best CONUS result (2026-06-23):** NSE 0.715 / KGE 0.711 (precip-driven disagg + L1, 2,365 gauges). KGE does NOT beat the summed-Q' baseline (0.7172) in any config as of 2026-07-06. + +--- + +## Provenance and maintenance + +Re-verify routing after any new master merge: +```bash +git log --oneline origin/master..HEAD # commits ahead +cargo run --release --example compare_ddr_sandbox # V1 gate +``` +Update this index when a new skill is added to `.claude/skills/`. diff --git a/.claude/skills/ddrs-diagnostics-and-tooling/SKILL.md b/.claude/skills/ddrs-diagnostics-and-tooling/SKILL.md new file mode 100644 index 0000000..8640b60 --- /dev/null +++ b/.claude/skills/ddrs-diagnostics-and-tooling/SKILL.md @@ -0,0 +1,582 @@ +--- +name: ddrs-diagnostics-and-tooling +description: > + Use when you need to MEASURE a ddrs result rather than eyeball it — diagnosing + a failed run, verifying gradient health, interpreting leakance GO/NO-GO gates, + checking the summed-Q' baseline, probing identifiability, or deciding whether + a hypothesis is SUPPORTED/REFUTED/INCONCLUSIVE. Triggers: "why is zeta small", + "how do I tell if training is working", "what does K_D ceiling mean", + "compare two runs", "is my gradient alive", "how do I reproduce the 2x2 + verdict". Do NOT use for architecture changes, config authoring, or writing + new training code — use ddrs-change-control or ddrs-architecture-contract + instead. +--- + +# ddrs Diagnostics and Tooling + +## Glossary (jargon defined once) + +| Term | Definition | +|---|---| +| **ddrs** | BURN-0.21 Rust port of the DDR differentiable Muskingum-Cunge solver | +| **BURN** | Rust deep-learning framework (analogous to PyTorch); autograd tapes differ | +| **KAN head** | `rskan::KanLayer` network (`Linear→KanLayer×N→Linear→Sigmoid`); maps catchment attributes to per-reach routing parameters | +| **Q'** (Q-prime) | Upstream-summed divide streamflow forcing from a pre-computed DHBv2 retrospective | +| **zeta (ζ)** | Per-reach GW–SW exchange flux (m³/s): `leakance_factor · area_z · K_D · (depth − d_gw)`. Positive = losing reach | +| **K_D** | Hydraulic exchange rate (1/s); log-space parameter in `[1e-8, 1e-6]` by default | +| **Muskingum-Cunge (MC)** | Linear flood-routing method; ddrs solves a CSR lower-triangular system per timestep | +| **eval network** | Gauge-subgraph union used during evaluation (64,892 reaches for CONUS) | +| **NSE / KGE** | Nash-Sutcliffe Efficiency / Kling-Gupta Efficiency; standard hydrology skill scores | +| **NNSE** | Normalized NSE: `NSE/(2-NSE)`, range [0,1], avoids -∞ floor | +| **summed-Q' baseline** | Upper bound with no routing: sum of upstream Q' at each gauge; median NSE 0.689 / KGE 0.723 (CONUS, as of 2026-07-05) | +| **2×2** | Leakance ON/OFF × hourly/daily forcing factorial experiment | +| **rho-window** | Training sub-sequence length (default 90 days); sampled from the full training period | +| **hotstart transient** | Initial-condition mismatch at window start; big rivers carry memory >> warmup days | +| **CsrSolveOp** | Hand-written BURN autograd backward for the sparse triangular solve (invariant 4) | +| **dump_parameters** | Binary/CLI command that exports full-CONUS KAN outputs to `kan_parameters.nc` | +| **run-id** | `-[-]` directory name under `.ddrs/runs/` | + +--- + +## When NOT to use this skill + +- **Changing `src/routing/`, `src/sparse.rs`, or `src/geometry.rs`** — use ddrs-change-control (blast-radius analysis required) +- **KAN head architecture changes** — use ddrs-architecture-contract (invariants 5-6) +- **Writing new Python analysis scripts** — use ddrs-proof-and-analysis-toolkit +- **Interpreting the research roadmap / phase gating** — use ddrs-identifiability-campaign or ddrs-research-frontier + +--- + +## Part 1: Non-negotiable regression gates + +Run these before and after ANY change to `src/routing/`, `src/geometry.rs`, or `src/sparse.rs`. + +### 1.1 DDR parity gate (invariant 1) + +```bash +cargo run --release --example compare_ddr_sandbox +``` + +**Pass:** prints `ABSOLUTE MATCH` — max abs diff < 1e-3 m³/s on the 5-reach RAPID sandbox. +**Fail:** any diff >= 1e-3 m³/s means the port broke. Do NOT merge. + +**Caveat (as of 2026-06-06):** the reference fixture requires the desktop's `~/projects/ddr` with the unpushed `geometry/trapezoidal.py` changes. A clean DDR clone will diverge ~1%. That is a wrong-reference failure, not a port failure. See `.claude/references/ddrs-comparing-to-ddr.md` §Regenerating fixtures before concluding a real regression. + +### 1.2 Leakance gradient-exactness gates + +Run whenever `src/routing/leakance.rs` or its backward op changes: + +```bash +cargo test --test leakance_gradcheck # analytical grad ≈ finite-difference (8 cases) +cargo test --test leakance_off_parity # byte-identical to no-leakance when off (3 cases) +cargo test --test zeta_accum # accumulated zeta == b_rhs delta (6 cases) +cargo run --release --example compare_ddr_sandbox # still ABSOLUTE MATCH +``` + +**Interpretation:** `leakance_gradcheck` failing means the analytical backward diverges from finite-diff; this breaks training correctness. `leakance_off_parity` failing means leakance bleeds into non-leakance paths. + +### 1.3 KAN head parity gates + +Run when `src/nn/kan_head.rs`, `Cargo.toml` rskan pin, or DDR's `nn/kan.py` changes: + +```bash +cargo test --features fixtures \ + --test kan_head_init_repro \ + --test kan_head_init_parity \ + --test kan_head_fixture_forward \ + --test kan_head_fixture_backward +``` + +### 1.4 Sparse gradcheck + +```bash +cargo test --test sparse_gradcheck +``` + +Verifies the CSR backward (invariant 4) is gradient-exact. + +--- + +## Part 2: Stale binary trap (the most common failure mode) + +`~/.cargo/bin/ddrs` is installed once by `cargo install`. `cargo build` does NOT update it. + +**Symptom:** run looks fine, metrics make no sense, or a new feature is silently missing. + +**Check whether you have the right binary:** + +```bash +# Current checkpoints are DIRECTORIES: +ls .ddrs/runs//checkpoints/ +# Should show: epoch_5_mb_9/ (a directory) +# If you see: epoch_5_mb_9.mpk (a flat file) → stale binary +``` + +**Fix:** + +```bash +cargo install --path . # canonical, ~2 min +# or faster: +cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs +# or bypass installed binary entirely: +cargo run --release --bin ddrs -- run --workflow train-and-test +``` + +The stale-binary trap caused the 2026-07-01 2×2 to produce byte-identical hourly and daily cells (the installed binary predated the disaggregation feature). + +--- + +## Part 3: CUDA graphs masking NaN + +**Symptom:** training loss is finite and slowly decreasing, but intermediate checks show NaN activations. + +**Cause:** `use_cuda_graphs: true` replays a captured graph; a NaN in a subsequent forward returns stale (pre-NaN) finite values. The loss looks healthy while the model is broken. + +**Diagnosis:** + +```bash +# In ddrs.yaml, temporarily set: +use_cuda_graphs: false +# Then re-run one mini-batch and inspect: +# If loss is NaN → confirmed NaN forward; debug with use_cuda_graphs: false +# If loss is fine → not a NaN issue +``` + +**Note:** `use_leakance: true` combined with `use_cuda_graphs: true` is rejected at config load time with a hard error. + +--- + +## Part 4: Run inspection and workspace navigation + +### 4.1 Check run status and disk + +```bash +ddrs status # workspace summary + disk usage by run +ddrs show # full manifest: config, sources, git SHA, metrics +``` + +### 4.2 Read a run's log + +```bash +cat .ddrs/runs//run.log # timestamped stdout+stderr (fd-level capture) +``` + +Useful patterns in the log to check: + +| What to grep | Meaning | +|---|---| +| `streamflow resolution: Daily\|Hourly` | Confirms whether icechunk store was read as daily or hourly | +| `warm start: loaded KAN head` | Checkpoint resume loaded correctly | +| `no …/optim.mpk` | Adam starts cold (expected for head-only warm-start) | +| `ABSOLUTE MATCH` | Sandbox regression passed during this run | +| `precip loading` | AORC precip store opened (needed for disaggregation) | + +### 4.3 Inspect a run's config + +```bash +cat .ddrs/runs//config.yaml # exact config that produced this run +``` + +### 4.4 Compare two runs' metrics + +```bash +ddrs show | grep -E "nse|kge|loss" +ddrs show | grep -E "nse|kge|loss" +``` + +--- + +## Part 5: Summed-Q' baseline + +**What it is:** per-gauge sum of upstream Q' with NO routing or learning. It is the ceiling that trained routing must beat. + +**Reference numbers (CONUS, as of 2026-07-05):** +- Median NSE: 0.689 +- Median KGE: 0.723 + +**Best trained result (precip-driven disaggregation + L1, 2365 gauges, as of 2026-06-23):** +- Median NSE: 0.715 (+0.037 vs baseline — beats it) +- Median KGE: 0.711 (-0.012 vs baseline — does NOT beat it) + +**Critical:** KGE does NOT beat the summed-Q' baseline in any config as of 2026-07-05. NSE beats it with precip disaggregation. This is a known open problem (over-attenuation; L1 and NSE reward low variance). + +**Reproduce baseline:** + +```bash +ddrs plan # computes and caches baseline automatically on first run +# or read the cached version: +cat .ddrs/baselines//manifest.json # shows metrics, provenance +``` + +**Interpretation:** if your trained run's median NSE does NOT beat 0.689, the routing term earns nothing. Debug training loss curves and KAN head gradient stats before touching the sparse solver. + +--- + +## Part 6: dump_parameters — export learned KAN outputs to NetCDF + +```bash +# Via the legacy eval binary (required for leakance zeta export): +cargo build --release --bin eval +target/release/eval \ + --config config/experiments/leakance_hourly_on.yaml \ + --checkpoint .ddrs/runs//checkpoints/epoch_5_mb_9 \ + --output /tmp/eval.zarr \ + --zeta-output .ddrs/runs//kan_parameters.nc + +# Or (no zeta, just KAN params): +cargo build --release --bin dump_parameters +target/release/dump_parameters \ + --config ddrs.yaml \ + --checkpoint .ddrs/runs//checkpoints/epoch_5_mb_9/head \ + --output .ddrs/runs//plot/kan_parameters.nc +``` + +**Output file layout (`kan_parameters.nc`):** + +| Variable | Dimension | Unit | Notes | +|---|---|---|---| +| `COMID` | `(COMID,)` | — | Reach IDs for full-CONUS params | +| `n` | `(COMID,)` | — | Manning's roughness, denormalized | +| `q_spatial` | `(COMID,)` | — | Channel geometry exponent | +| `x_storage` | `(COMID,)` | — | Muskingum X (storage weighting) | +| `K_D` | `(COMID,)` | 1/s | Hydraulic exchange rate (leakance only) | +| `d_gw` | `(COMID,)` | m | GW depth threshold (leakance only) | +| `leakance_factor` | `(COMID,)` | — | Scale factor (leakance only) | +| `COMID_eval` | `(COMID_eval,)` | — | Eval-network reach IDs (leakance only) | +| `zeta` | `(COMID_eval,)` | m³/s | Mean \|zeta\| over eval window | +| `zeta_net` | `(COMID_eval,)` | m³/s | Signed mean; positive = losing reach | +| `depth_mean` | `(COMID_eval,)` | m | Eval-window mean routed depth | +| `area_z_mean` | `(COMID_eval,)` | m² | Eval-window mean plan-view wetted area | +| `q_mean` | `(COMID_eval,)` | m³/s | Eval-window mean routed discharge | + +**Load in Python:** + +```python +import xarray as xr +ds = xr.open_dataset(".ddrs/runs//kan_parameters.nc") +# for a quick K_D ceiling check: +kd = ds["K_D"].values +import numpy as np +print(f"K_D: min={kd.min():.2e} median={np.median(kd):.2e} max={kd.max():.2e}") +print(f"fraction at ceiling (1e-6): {(kd > 9.9e-7).mean():.1%}") +``` + +--- + +## Part 7: Leakance GO/NO-GO evaluation + +### 7.1 The three gate criteria (per spec) + +| Gate | Threshold | Interpretation | +|---|---|---| +| 1 | ΔNSE or ΔKGE > 0 (median) on losing-stream subset, hourly arm | Leakance improves skill where physics predicts it should | +| 2 | Effect absent or weaker in daily arm | Rules out fudge-factor behavior | +| 3 | \|zeta\| > 0.01 m³/s on ≥ 10% of eval reaches | Learned exchange is non-trivially active | + +**Current status (as of 2026-07-01):** GO — but marginal (10.4% vs 10% threshold for gate 3, no headroom). + +### 7.2 Running the full verdict script + +```bash +cd ~/projects/ddr +uv run python ~/projects/ddrs/scripts/leakance_subset_analysis.py \ + --hourly-on 2026-07-01T13-43-32Z-train-and-test \ + --daily-on 2026-07-01T21-20-27Z-train-and-test \ + --hourly-off 2026-06-23T02-49-12Z-conus-hourly-train-and-test \ + --daily-off 2026-06-05T01-41-16Z-train-and-test \ + --ddrs-runs-dir /home/tbindas/projects/ddrs/.ddrs/runs +``` + +**Prerequisites:** both ON arms must have `kan_parameters.nc` with the `zeta`/`COMID_eval` variables (produced by `--zeta-output` or `train-and-test` Phase 2). + +**Output block to look for:** + +``` +VERDICT: GO + Leakance improves skill on the losing-stream subset under hourly forcing ... +``` + +or `VERDICT: NO-GO` with reasons, or `VERDICT: NEEDS_ZETA_EXPORT`. + +**Losing-stream subset definition:** gauges where the summed-Q' baseline mean(pred)/mean(obs) > 1 on the hourly-OFF run. CONUS result: 1883/2365 gauges (79.6%). + +### 7.3 Interpreting leakance parameter outputs + +| Observation | Interpretation | +|---|---| +| `K_D` 100% at ceiling (1e-6) | Optimizer wants MORE exchange; box is binding. NOT a model failure (H1 REFUTED, as of 2026-07-02) | +| `leakance_factor` interior (≈0.33) | Gate is open; reaches are actively exchanging | +| `d_gw` near mean depth | Driving head throttled; ~47% of reaches gaining at eval-window mean | +| zeta–uparea Spearman +0.76 | Exchange tracks river size, not aridity — gauge bias, not starvation | +| dry-tercile zeta < wet-tercile | Inverse of physics; training signal concentrates near large gauged rivers | + +--- + +## Part 8: Seven-hypothesis diagnosis battery (leakance low-zeta) + +**When to run:** after any leakance experiment returns small zeta (median |zeta| < 0.01 m³/s). + +**Prereqs:** the ON run's `kan_parameters.nc` must contain `depth_mean`, `area_z_mean`, `q_mean` on `COMID_eval`. Requires the re-eval pass with the current binary. + +```bash +cd ~/projects/ddr +uv run python ~/projects/ddrs/scripts/leakance_diagnosis.py +# uses hardcoded run IDs in ARM_IDS dict; edit if using different runs +``` + +**Hypothesis reference table (results as of 2026-07-02):** + +| # | Hypothesis | Verdict (2026-07-02) | Key number | +|---|---|---|---| +| H1 | K_D box clips zeta below detectability | REFUTED | 71.5% of reaches CAN exceed 0.01 m³/s in-box; median utilization 3.4% | +| H2 | d_gw near depth → driving head ≈ 0 | SUPPORTED | 57.6% of reaches < 0.1 m mean driving head; 47.0% ≤ 0 | +| H3 | KAN variance collapse (original hypothesis) | REFUTED | K_D–aridity ρ = +0.61; d_gw–meanP ρ = +0.71 — strong spatial structure | +| H4 | Gauge bias / gradient starvation | SUPPORTED | gauged median |zeta| 6.7e-3 vs ungauged 5.9e-4 (11×); dry/wet ratio 0.40 (inverse of physics) | +| H5 | Equifinality with n/x_storage | SUPPORTED (daily only) | daily Δn = +0.012 (0.59 IQR); hourly Δn nil (0.05 IQR) | +| H6 | Wrong yardstick (absolute 0.01 bar) | REFUTED | 8.4% >1% fractional loss agrees with absolute bar | +| H7 | d_gw boundary pinning (disconnected regime) | REFUTED | 0.0% of d_gw at bounds | + +**Diagnosis conclusion:** zeta is small because the optimizer throttles through the driving head (H2) and the gradient only reaches gauged large rivers (H4), not because the K_D box or KAN architecture fails. Widening K_D past 1e-6 is NOT recommended (supersedes the "top follow-up" in `docs/2026-07-01-leakance-hourly-findings.md`). + +--- + +## Part 9: Gradient probe (adjoint reachability + detectability) + +**Location:** `origin/worktree-zeta-sensitivity` branch. +**When to run:** when you want to know whether the leakance gradient is alive at a reach, or whether a real-magnitude loss would be detectable at a downstream gauge. + +### 9.1 Stage 1 — adjoint reachability map + +```bash +# Trained checkpoint (use worktree binary): +WT=/home/tbindas/projects/ddrs/.claude/worktrees/zeta-sensitivity +nice -n 10 $WT/target/release/probe_zeta_gradient \ + --config config/experiments/leakance_hourly_on.yaml \ + --checkpoint .ddrs/runs/2026-07-01T13-43-32Z-train-and-test/checkpoints/epoch_5_mb_9 \ + --windows 96 --seed 42 \ + --output output/zeta_probe/grad_trained.nc + +# Cold head (omit --checkpoint): +nice -n 10 $WT/target/release/probe_zeta_gradient \ + --config config/experiments/leakance_hourly_on.yaml \ + --windows 96 --seed 42 \ + --output output/zeta_probe/grad_cold.nc +``` + +**Output:** per-reach `|∂L/∂factor|`, `∂L/∂factor`, coverage count in NetCDF. + +**Interpretation thresholds:** + +| Ratio (gauged/ungauged |g|) | Interpretation | +|---|---| +| ≥ 10× at both trained and cold points | SUPPORTED starvation — auxiliary supervision fills genuine gap | +| < 10× | REFUTED starvation — gradient reaches everywhere | + +**Measured result (2026-07-03, as of 2026-07-05):** gauged/ungauged ratio = 1.5× (trained), 2.9× (cold). P1 starvation REFUTED. The gradient is alive everywhere. + +### 9.2 Stage 2 — planted-delta detectability + +```bash +# Plan sites first (ddrs-py venv): +cd ddrs-py && uv run python ../scripts/zeta_probe_sites.py + +# Perturb pass: +nice -n 10 $WT/target/release/probe_zeta_gradient \ + --config config/experiments/leakance_hourly_on.yaml \ + --checkpoint .ddrs/runs/2026-07-01T13-43-32Z-train-and-test/checkpoints/epoch_5_mb_9 \ + --mode perturb \ + --probe-plan output/zeta_probe/probe_plan.csv \ + --eval-days 1095 \ + --output output/zeta_probe/perturb +``` + +**Detectability criterion:** `|mean ΔQ| > 99th-pct noise floor AND > 5% of gauge's mean flow`. + +**Measured result (2026-07-03, as of 2026-07-05):** + +| Delta | Reference (Ref) gauges | Non-reference | +|---|---|---| +| 0.01 m³/s (literature-magnitude) | 4.2% detectable | 0.0% | +| 0.1 m³/s (upper-literature) | 16.7% detectable | 2.1% | + +P3 detectability: NO-GO. The planted loss arrives at gauges at ~95% fidelity (transmission fine) but is 53× smaller than the median Ref gauge's 5% discharge-uncertainty band. Detection fails on dilution, not transmission. No gauge-only objective can learn real-world leakance. + +--- + +## Part 10: Synthetic recoverability control (Phase B) + +**Location:** `origin/worktree-zeta-sensitivity` branch. +**Status (as of 2026-07-05):** FAILED — positive control not passed. + +**One-line result:** recovery ratio median 0.009 (bar: ≥ 0.5). Root cause: the windowed training objective (rho-90, warmup-5) has a ~130× hotstart-transient noise floor relative to the planted signal. The signal is invisible even with zero observation noise and warm-started weights. + +**Phase B objective:** reduce the noise floor to ≤ 0.25 mean L1 (≤ 10% of a converged run). This is NOT YET MET as of 2026-07-05. Required before any identifiability claim for leakance. + +**Key decomposition:** + +| Quantity | Value | +|---|---| +| Planted signal (continuous residual, teacher weights + teacher obs) | 0.0076 mean L1 | +| Step-0 windowed training loss (warm-started run A) | 1.017 mean L1 | +| Noise floor / signal ratio | ~130× | +| Run A continuous residual after 5 epochs of training | 0.4431 (58× worse than start) | + +**Implication for leakance identifiability:** gauge-loss training cannot reward reach-scale leakance even with: detectable gauge signal (constructed), zero obs noise, expressible head, and warm-start from the answer. Leakance identifiability is NOT proven. Phase B (state-cache hotstart, ≤ 0.25 mean L1 target) is required before any identifiability claim. + +**Verdicts from the control run:** + +| # | Metric | Measured | Verdict | +|---|---|---|---| +| R1 | Recovery ratio median | 0.009 | FAILED (bar: ≥ 0.5) | +| R2 | Non-planted \|zeta_net\| A/baseline | 1.11 | PRECISE — trivially, nothing moved | +| R3 | Final-epoch loss A vs B (42.2% gap) | A < B | CONFOUNDED — B's handicap accounts for gap | +| R4 | Manning's n shift (run B) | Δn = −0.019 (global, not localized) | H5 equifinality confirmed at global scale | +| R5 | Cold emergence ratio | 1.20 | SUPPRESSED (bar: > 3) | + +--- + +## Part 11: Leakance configuration checklist + +Three config changes are ALL required together to enable leakance. Missing any one causes silent failure or a config-load error. + +```yaml +# 1. Activate the term (also disables CUDA graphs): +params: + use_leakance: true + +# 2. Tell the KAN head to emit leakance parameters: +kan_head: + learnable_parameters: + - K_D + - d_gw + - leakance_factor + - n # keep existing routing params + - q_spatial + # x_storage # optional + +# 3. Set parameter ranges: +params: + parameter_ranges: + K_D: [1.0e-8, 1.0e-6] # log-space; 1/s + d_gw: [-2.0, 2.0] # m + leakance_factor: [0.0, 1.0] # dimensionless +``` + +**Important:** `use_leakance: true` AND `use_cuda_graphs: true` is rejected at config load. The combination is not supported without a separate capture path. + +--- + +## Part 12: Training monitoring checklist + +Use this list when a run produces unexpected metrics. + +- [ ] **Check the binary is current** — flat checkpoint files mean stale binary (Part 2) +- [ ] **Check `streamflow resolution` in `run.log`** — `Daily` vs `Hourly` must match your intent +- [ ] **Check precip loaded** — if `use_precip: true` in disagg block, grep `precip loading` in run.log +- [ ] **Disable CUDA graphs if loss is suspiciously smooth** — guards against NaN masking (Part 3) +- [ ] **Check `kan_parameters.nc` K_D ceiling fraction** — 100% at ceiling means the box is binding +- [ ] **Run `leakance_subset_analysis.py`** for GO/NO-GO after any leakance experiment (Part 7.2) +- [ ] **Compare against baseline** — trained NSE should exceed 0.689; KGE may not (Part 5) +- [ ] **Check `ddrs show `** for final metrics in the manifest +- [ ] **Verify checkpoint format** — directory `epoch_E_mb_M/` with `head.mpk`, `optim.mpk`, `state.json` + +--- + +## Part 13: Run ID and workspace layout quick reference + +``` +.ddrs/ + system.json # GPU/driver probe result + sources.lock # fingerprints of data_sources paths + adjacency// # managed CONUS + gauge adjacency (content-addressed) + baselines// # summed-Q' baseline cache + manifest.json # metrics + gage_ids + predictions.f32 # row-major [n_gauges, n_days] + observations.f32 + runs// + manifest.json # config + sources + git SHA + output metrics + config.yaml # exact config snapshot + run.log # timestamped stdout+stderr + checkpoints/ + epoch_E_mb_M/ # DIRECTORY (flat .mpk = stale binary) + head.mpk + optim.mpk + state.json + eval/ + predictions.zarr/ # zarr-v3 group + predictions/ # float64 [n_gauges, n_days] + observations/ + gage_ids/ # uint8 [n_gauges, 8] fixed-width ASCII + baseline/ # copy of .ddrs/baselines// + kan_parameters.nc # KAN outputs + zeta (leakance) or plot params + plot/ + kan_parameters.nc # full-CONUS dump_parameters output +``` + +--- + +## Part 14: Zeta accumulator — what it measures and how to verify + +The zeta accumulator is enabled automatically during eval when `use_leakance: true`. It recomputes per-step zeta from the SAME saved primitives the backward reads, then accumulates per-reach means over the eval window. + +**Correctness identity (from `tests/zeta_accum.rs`):** + +``` +q_no_leak[0] − q_leak[0] == zeta[0] (headwater reach; exact equality) +``` + +**Verify the identity is preserved:** + +```bash +cargo test --test zeta_accum +``` + +**Check zeta is non-trivial after a leakance run:** + +```python +import xarray as xr, numpy as np +ds = xr.open_dataset(".ddrs/runs//kan_parameters.nc") +z = np.abs(ds["zeta"].values) +print(f"median |zeta|: {np.median(z):.4e} m3/s") +print(f"|zeta| > 0.01 on {np.mean(z > 0.01):.1%} of eval reaches") +# CONUS target (as of 2026-07-01): 10.4% of 64,892 reaches +``` + +**Zeta dimensionality:** `COMID_eval` dimension = gauge-subgraph union = eval network (64,892 reaches for CONUS). NOT the full 346,321-reach CONUS. + +--- + +## Provenance and maintenance + +```bash +# Re-verify Part 1 regression gates: +cargo run --release --example compare_ddr_sandbox # ABSOLUTE MATCH +cargo test --test leakance_gradcheck # 8/8 +cargo test --test zeta_accum # 6/6 +cargo test --test sparse_gradcheck # pass + +# Re-verify leakance 2x2 results: +cd ~/projects/ddr && uv run python \ + ~/projects/ddrs/scripts/leakance_subset_analysis.py \ + --hourly-on 2026-07-01T13-43-32Z-train-and-test \ + --daily-on 2026-07-01T21-20-27Z-train-and-test \ + --hourly-off 2026-06-23T02-49-12Z-conus-hourly-train-and-test \ + --daily-off 2026-06-05T01-41-16Z-train-and-test \ + --ddrs-runs-dir /home/tbindas/projects/ddrs/.ddrs/runs + +# Re-verify 7-hypothesis diagnosis: +cd ~/projects/ddr && uv run python \ + ~/projects/ddrs/scripts/leakance_diagnosis.py + +# Source files for this skill: +# /home/tbindas/projects/ddrs/CLAUDE.md +# /home/tbindas/projects/ddrs/scripts/leakance_subset_analysis.py +# /home/tbindas/projects/ddrs/scripts/leakance_diagnosis.py +# /home/tbindas/projects/ddrs/docs/2026-07-01-leakance-hourly-findings.md +# /home/tbindas/projects/ddrs/docs/2026-07-02-leakance-diagnosis-findings.md +# origin/worktree-zeta-sensitivity:docs/2026-07-03-zeta-gradient-probe-findings.md +# origin/worktree-zeta-sensitivity:docs/2026-07-04-synthetic-recoverability-findings.md +# Skill last verified: 2026-07-05 +# Volatile facts: summed-Q baseline metrics, leakance GO/NO-GO verdict, +# recoverability Phase B status — re-verify after any new experiment run +``` diff --git a/.claude/skills/ddrs-docs-and-writing/SKILL.md b/.claude/skills/ddrs-docs-and-writing/SKILL.md new file mode 100644 index 0000000..d5df37a --- /dev/null +++ b/.claude/skills/ddrs-docs-and-writing/SKILL.md @@ -0,0 +1,366 @@ +--- +name: ddrs-docs-and-writing +description: "Use when writing, updating, or reviewing any ddrs project document: findings reports, session handoffs, experiment specs, implementation plans, or the paper at ddr_equifinality/paper.tex. Also use when asking what the correct doc type is for a given output, how to connect an experiment result to the paper narrative, or whether a hypothesis verdict belongs in a findings doc or a spec. Do NOT use for running experiments, editing Rust code, debugging builds, or plotting — use ddrs-run-and-operate, ddrs-debugging-playbook, or ddrs-eval-plots instead." +--- + +# ddrs docs and writing + +This skill covers the documentation conventions, doc-type taxonomy, template structures, house style, and paper-to-experiment-log connection for the ddrs project. + +## Glossary (defined once) + +| Term | Meaning | +|---|---| +| **ddrs** | BURN-0.21 Rust port of the Python DDR (Differentiable Discharge Routing) solver using Muskingum-Cunge | +| **DDR** | Python/PyTorch reference at `~/projects/ddr/`. ddrs must stay gradient-exact against it | +| **KAN head** | Kolmogorov-Arnold Network head (`rskan::KanLayer`) that maps catchment attributes → routing parameters | +| **summed-Q′** | No-routing baseline: per-gauge sum of upstream divide Qr. Any trained model must beat this to prove routing earns its keep | +| **CONUS** | Contiguous United States; 346,321 MERIT reaches, 338,814 edges | +| **eval network** | The gauge-subgraph union used at evaluation time (64,892 reaches in the 2×2 experiments) | +| **zeta** | Per-reach GW–SW exchange flux (m³/s); positive = losing stream | +| **leakance** | Experimental GW–SW water-loss term; off by default; controlled by `use_leakance: true` in config | +| **disagg head** | Daily→hourly disaggregation sub-head inside KanHead; driven by AORC hourly precip | +| **spec** | Design doc written BEFORE an experiment. Lives in `docs/superpowers/specs/` | +| **plan** | Implementation task list derived from a spec. Lives in `docs/superpowers/plans/` | +| **findings doc** | Post-experiment narrative with hypothesis table, results, and verdict. Lives in `docs/` | +| **handoff** | Session-boundary doc with the action list for the next session. Lives in `docs/` | +| **journal** | Multi-experiment chronological record (used for cross-cutting investigations). Lives in `docs/` | + +--- + +## Doc-type taxonomy + +Choose the correct doc type before writing anything. + +| Situation | Doc type | Location | Naming pattern | +|---|---|---|---| +| Planning a new experiment before any code runs | **spec** | `docs/superpowers/specs/` | `YYYY-MM-DD--design.md` | +| Breaking a spec into implementation tasks | **plan** | `docs/superpowers/plans/` | `YYYY-MM-DD-.md` (same slug) | +| Summarizing what an experiment found after it ran | **findings doc** | `docs/` | `YYYY-MM-DD--findings.md` | +| Handing state to the next session mid-experiment | **handoff** | `docs/` | `YYYY-MM-DD--handoff.md` | +| Documenting a cross-cutting investigation that spans multiple runs | **journal** | `docs/` | `_journal.md` or a named findings doc | +| Documenting a data source, contract, or external API | **reference** | `docs/reference/` or `docs/` | descriptive name, no date required | + +### When NOT to write a new doc + +- If a findings doc for that experiment already exists: update it in place (add a datestamped section) rather than creating a duplicate. +- Handoffs and journals are living docs — append, do not replace. +- The spec/plan pair is mandatory before running expensive GPU jobs. Do not skip the spec to save time: the pre-registered hypotheses and falsification criteria protect against HARKing (Hypothesizing After Results are Known). + +--- + +## Findings doc template + +Every experiment that runs on real data gets a findings doc. Use this structure exactly. + +```markdown +# — findings () + +Spec: `docs/superpowers/specs/.md` +Plan: `docs/superpowers/plans/.md` (if applicable) +Script: `scripts/