From b0f4b0372fa6e39f02a065dad9a9768d42fecf69 Mon Sep 17 00:00:00 2001 From: Tadd Bindas Date: Thu, 30 Jul 2026 21:54:59 -0400 Subject: [PATCH 1/4] docs(skills): audit all docs+skills+README, consolidate 18 skills into 2 Audited every `.claude/skills/` package (9,371 lines), all 32 `docs/` files, README.md, `.claude/references/`, and CLAUDE.md by verifying each concrete claim against source rather than assessing plausibility. Findings (full report: docs/2026-07-30-docs-and-skills-audit.md): - `kan_head.disaggregation.use_precip` does not exist. It is documented in CLAUDE.md, src/config.rs:113's comment, and config/sources/conus-hourly.yaml. Removed in 334f0fe; the head now always consumes precip. - `ddrs init` (exit 2) and `ddrs run --workflow eval` (`--from-run` unimplemented) are documented as working in README.md, docs/, and CLAUDE.md. Both hard-error. - The baseline `0.689 / 0.723` propagated into 13 of 18 skills as the CONUS bar. It is a global-MERIT 5,224-gauge number. The CONUS bar is 0.6781 / 0.7172 on 2,365 gauges, making the widely-quoted +0.026 delta a cross-population subtraction (correct: +0.037). - The 07-07 and 07-16 findings docs compare 3,211-gauge baselines against 2,365-gauge trained medians. Population-matched, hourly-lstm's "+0.022 NSE gain from routing" reverses to -0.051. - One superseding doc (2026-07-06 leakance NO-GO) invalidated the same "Phase B is required" paragraph in six skills; none were updated. Three still carried executable instructions for the closed campaign. - Test counts were wrong (leakance_gradcheck 16 not 8, zeta_accum 8 not 6), and two gate commands in CLAUDE.md/docs run zero tests and exit 0. - `.claude/references/` is a strict subset of `docs/`; its newer mtime is a 2026-06-10 back-port of corrections `docs/` received on 2026-06-08. Consolidation: 18 skills -> 2, 9,371 -> 2,359 lines (75% reduction). - `ddrs-dev` (build/code/run/debug): SKILL.md + references/{build-and-env, config, testing, traps, research-status}.md - `ddrs-eval-plots` (evaluate output): rewritten around `.ddrs/runs//`; the dead `output/saved_models_*/` layout is gone. Two dead-campaign references (H5 parameter_swap, H6 loss_landscape) removed since both plot instruments the v2 audit refuted; two orphaned parity_* files merged into parity.md and wired into the routing table. Editorial rules applied: do not duplicate CLAUDE.md (in context every turn); lead with the discriminating test, not the narrative; one authoritative numbers table with gauge-set definitions and a do-not-use list, replacing five copies of H1-H7; assert "all pass" instead of brittle test counts. Every rewritten eval-plots template was executed end-to-end against a real run, reproducing the committed parameter_convergence_stats.json values exactly. docs/ and README.md fixes are enumerated as prioritized follow-ups in the report, not applied here. Co-Authored-By: Claude --- .../ddrs-architecture-contract/SKILL.md | 428 ------------- .claude/skills/ddrs-build-and-env/SKILL.md | 439 ------------- .claude/skills/ddrs-change-control/SKILL.md | 413 ------------ .claude/skills/ddrs-config-and-flags/SKILL.md | 403 ------------ .../skills/ddrs-debugging-playbook/SKILL.md | 595 ------------------ .claude/skills/ddrs-dev/SKILL.md | 146 +++-- .../ddrs-dev/references/build-and-env.md | 80 +++ .claude/skills/ddrs-dev/references/config.md | 194 ++++++ .../ddrs-dev/references/research-status.md | 213 +++++++ .claude/skills/ddrs-dev/references/testing.md | 127 ++++ .claude/skills/ddrs-dev/references/traps.md | 212 +++++++ .../ddrs-diagnostics-and-tooling/SKILL.md | 582 ----------------- .claude/skills/ddrs-docs-and-writing/SKILL.md | 366 ----------- .claude/skills/ddrs-eval-plots/SKILL.md | 337 +++++----- .../skills/ddrs-eval-plots/evals/evals.json | 16 +- .../ddrs-eval-plots/references/hydrograph.md | 25 +- .../references/kan_interpretability.md | 103 --- .../references/loss_landscape_h6.md | 156 ----- .../ddrs-eval-plots/references/metrics.md | 94 ++- .../references/parameter_map.md | 162 ++++- .../references/parameter_swap.md | 243 ------- .../ddrs-eval-plots/references/parity.md | 235 +++++++ .../ddrs-eval-plots/references/parity_init.md | 85 --- .../references/parity_trained.md | 124 ---- .../scripts/load_ddrs_predictions.py | 66 +- .../skills/ddrs-external-positioning/SKILL.md | 386 ------------ .../skills/ddrs-failure-archaeology/SKILL.md | 485 -------------- .../skills/ddrs-hydrology-reference/SKILL.md | 398 ------------ .../ddrs-identifiability-campaign/SKILL.md | 527 ---------------- .../ddrs-proof-and-analysis-toolkit/SKILL.md | 569 ----------------- .../skills/ddrs-research-frontier/SKILL.md | 386 ------------ .../skills/ddrs-research-methodology/SKILL.md | 551 ---------------- .claude/skills/ddrs-run-and-operate/SKILL.md | 452 ------------- .../skills/ddrs-validation-and-qa/SKILL.md | 396 ------------ .claude/skills/regenerate-docs.md | 142 ----- docs/2026-07-30-docs-and-skills-audit.md | 275 ++++++++ 36 files changed, 1883 insertions(+), 8528 deletions(-) delete mode 100644 .claude/skills/ddrs-architecture-contract/SKILL.md delete mode 100644 .claude/skills/ddrs-build-and-env/SKILL.md delete mode 100644 .claude/skills/ddrs-change-control/SKILL.md delete mode 100644 .claude/skills/ddrs-config-and-flags/SKILL.md delete mode 100644 .claude/skills/ddrs-debugging-playbook/SKILL.md create mode 100644 .claude/skills/ddrs-dev/references/build-and-env.md create mode 100644 .claude/skills/ddrs-dev/references/config.md create mode 100644 .claude/skills/ddrs-dev/references/research-status.md create mode 100644 .claude/skills/ddrs-dev/references/testing.md create mode 100644 .claude/skills/ddrs-dev/references/traps.md delete mode 100644 .claude/skills/ddrs-diagnostics-and-tooling/SKILL.md delete mode 100644 .claude/skills/ddrs-docs-and-writing/SKILL.md delete mode 100644 .claude/skills/ddrs-eval-plots/references/kan_interpretability.md delete mode 100644 .claude/skills/ddrs-eval-plots/references/loss_landscape_h6.md delete mode 100644 .claude/skills/ddrs-eval-plots/references/parameter_swap.md create mode 100644 .claude/skills/ddrs-eval-plots/references/parity.md delete mode 100644 .claude/skills/ddrs-eval-plots/references/parity_init.md delete mode 100644 .claude/skills/ddrs-eval-plots/references/parity_trained.md delete mode 100644 .claude/skills/ddrs-external-positioning/SKILL.md delete mode 100644 .claude/skills/ddrs-failure-archaeology/SKILL.md delete mode 100644 .claude/skills/ddrs-hydrology-reference/SKILL.md delete mode 100644 .claude/skills/ddrs-identifiability-campaign/SKILL.md delete mode 100644 .claude/skills/ddrs-proof-and-analysis-toolkit/SKILL.md delete mode 100644 .claude/skills/ddrs-research-frontier/SKILL.md delete mode 100644 .claude/skills/ddrs-research-methodology/SKILL.md delete mode 100644 .claude/skills/ddrs-run-and-operate/SKILL.md delete mode 100644 .claude/skills/ddrs-validation-and-qa/SKILL.md delete mode 100644 .claude/skills/regenerate-docs.md create mode 100644 docs/2026-07-30-docs-and-skills-audit.md diff --git a/.claude/skills/ddrs-architecture-contract/SKILL.md b/.claude/skills/ddrs-architecture-contract/SKILL.md deleted file mode 100644 index 2b2493a..0000000 --- a/.claude/skills/ddrs-architecture-contract/SKILL.md +++ /dev/null @@ -1,428 +0,0 @@ ---- -name: ddrs-architecture-contract -description: "Use when you are about to touch src/routing/, src/sparse.rs, src/geometry.rs, src/nn/kan_head.rs, src/cuda_graph/, Cargo.toml's rskan pin, or any training/eval path; when you need to know which invariants are load-bearing and why; when a test fails and you need to triage against known weak points; when you are running an experiment and need to know the current performance baseline and which claims are proven vs open; or when setting up the binary install / CLI workflow." ---- - -# ddrs Architecture Contract - -## When NOT to use this skill - -- For CLI lifecycle / workflow orchestration details → read `docs/superpowers/specs/2026-05-30-ddrs-cli-lifecycle-design.md` -- For the BURN 0.21 autograd API recipe → `.claude/references/ddrs-burn-autograd.md` -- For data-source path details (zarr layout, icechunk sniffing) → `.claude/references/ddrs-reading-inputs.md` -- For eval output format / zeta netcdf schema → `.claude/references/ddrs-reading-outputs.md` - ---- - -## 1. What ddrs is (two sentences) - -`ddrs` is a **BURN-0.21 Rust port** of DDR, a differentiable Muskingum-Cunge routing solver originally in Python/PyTorch. The port must produce **gradient-exact** outputs against DDR's reference on the 5-reach RAPID sandbox; that guarantee is the V1 invariant and must hold after every commit. - -**BURN** = Rust deep-learning framework (analogous to PyTorch). **Muskingum-Cunge** (MC) = a linear reservoir routing scheme where each reach has coefficients c1–c4 derived from channel geometry and Manning's equation. **Gradient-exact** = `max(|ddrs_output - ddr_output|) < 1e-3 m³/s` on that sandbox. - ---- - -## 2. The seven invariants — break any of these and the port is meaningless - -| # | Invariant | File(s) | Test / guard | -|---|---|---|---| -| 1 | `examples/compare_ddr_sandbox` reports **ABSOLUTE MATCH** (max abs diff < 1e-3 m³/s) | `src/routing/`, `src/geometry.rs`, `src/sparse.rs` | `cargo run --release --example compare_ddr_sandbox` | -| 2 | **f32 throughout the routing core** — no f64 or bf16 casts inside the timestep chain | `src/routing/mmc_op.rs`, `src/sparse/mod.rs`, `src/geometry.rs` | V1 test; any precision drift breaks DDR parity at the f32 floor (~1e-7 rel diff per reach) | -| 3 | **Adjacency is topologically ordered, lower-triangular** (`rows[k] >= cols[k]`) | `src/sparse/`, `src/adjacency/build.rs` | `cargo test data_zarr_store::conus_adjacency_loads_real_merit_zarr` | -| 4 | **Do NOT replace the hand-written sparse backward** in `src/sparse/mod.rs` (`CsrSolveOp impl Backward`) | `src/sparse/mod.rs` | `cargo test --test sparse_gradcheck` | -| 5 | **Routing head is `rskan::KanLayer`** via `src/nn/kan_head.rs` — `Linear(F,H) → KanLayer(H,H)×N → Linear(H,P) → Sigmoid`, no inter-block ReLU | `src/nn/kan_head.rs` | `cargo test --test kan_head` | -| 6 | **rskan pinned to a tag** in `Cargo.toml` — bump tag, re-run KAN parity tests, validate before merging | `Cargo.toml` | `cargo test --features fixtures --test kan_head_init_repro --test kan_head_init_parity --test kan_head_fixture_forward --test kan_head_fixture_backward` | -| 7 | **leakance + `use_cuda_graphs: true` is a config error** — config load rejects this combination | `src/routing/leakance.rs`, `src/config.rs` | Config validation at `ddrs plan` / `ddrs run` | - -### Why invariant 4 matters (O(nnz) vs O(n²)) - -BURN's default autograd records one node per tensor operation. If you replace `CsrSolveOp`'s hand-written backward with plain tensor unrolling, the tape grows O(n²) per timestep (n = 346,321 CONUS reaches). The custom backward keeps it O(nnz) = O(338,814 edges). Same logic applies to `TimestepOp` in `src/routing/mmc_op.rs` — one node per timestep, not ~33. - -### Why invariant 3 matters - -Forward substitution on `A = I − c1·N` requires N to be strictly lower-triangular (every reach appears after all its upstream neighbors in the sorted order). If any `rows[k] < cols[k]` entry exists, the solver silently produces wrong answers with no error. - ---- - -## 3. Source tree in one screen - -``` -src/ -├── routing/ -│ ├── mmc.rs MuskingumCunge: setup_inputs, forward, route_timestep -│ ├── mmc_op.rs TimestepOp — single Backward per timestep; saves 23 intermediates -│ ├── leakance.rs GW–SW loss term; TimestepLeakanceOp: Backward (experimental) -│ └── utils.rs denormalize, hotstart, dense helpers -├── sparse/ -│ ├── mod.rs CsrPattern (Arc-shared), CsrSolveOp + hand-written Backward -│ ├── cusparse.rs cuSPARSE SpMV + SpSV FFI wrappers (SP-9) -│ └── dispatch.rs CPU forward-sub vs cuSPARSE SpSV selector -├── cuda_graph/ SP-10 CUDA Graph capture/replay (forward-only; backward not yet captured) -│ ├── capture.rs -│ ├── geometry_kernel.rs fused #[cube] kernels K1/K2/K3 -│ └── scratch.rs -├── geometry.rs Trapezoidal channel geometry (Leopold & Maddock) -├── config.rs YAML config, parameter ranges, log-space flags, SparseSolver enum -├── nn/kan_head.rs KAN head via rskan — matches DDR's kan.py exactly -├── adjacency/build.rs Managed adjacency builder (topological_sort matches petgraph DFS) -├── data/ Live zarr/netcdf/icechunk readers — no export step -│ ├── ids.rs Comid(i64), Staid(String) newtypes; IdIndex -│ ├── dates.rs TimeAxis + rho-window sampler -│ └── store/ zarr.rs, zarr_obs.rs, zarr_qprime.rs, obs_writer.rs -├── training/ -│ ├── loss.rs L1 (default) or nnse-kge; config-selectable -│ ├── forward.rs LeakanceOverride seam (eval path only) -│ ├── bootstrap.rs Checkpoint resume: weights + optim + RNG state -│ └── probe.rs lift_leaf, probe_forward, GradAccum (gradient probe instruments) -└── bin/ - ├── ddrs.rs Primary CLI: plan / run / show / status / gc / sources / import - ├── probe_zeta_gradient.rs gradient probe + synthetic teacher (--mode grad|perturb|teacher|floor|state-cache) - ├── train.rs Legacy (deprecated, removed in 0.4) - └── eval.rs Legacy; still used for --zeta-output on existing checkpoints -``` - ---- - -## 4. Per-timestep dataflow (the MC routing step) - -Everything below runs inside `forward_chain_inner` in `src/routing/mmc_op.rs` at the **inner-backend primitive level** — no autograd nodes are created inside this function. One `TimestepOp` node wraps the entire chain. - -``` -inputs: (n, q_spatial, p_spatial, q_t, q_prime_t) -fixed: (length, slope, x_storage, dt=3600 s) - -K1 — geometry + Muskingum coefficients (one fused #[cube] kernel on CUDA): - depth = ((Q·n·(q+1)) / (p·√slope))^(3/(3q+5)) - top_width = p · depth^q - side_slope = clamp(top_width·q / (2·depth), 0.5, 50) - bottom_width = clamp(top_width − 2·side_slope·depth, bw_lb) - hyd_radius = ((top_width+bottom_width)·depth/2) / (bottom_width + 2·depth·√(ss²+1)) - velocity = clamp((1/n)·R^(2/3)·√slope, v_lb, 15) - celerity = velocity · 5/3 - k_musk = length / celerity - denom = 2·k·(1−x) + dt - c1..c4 = Muskingum coefficients - -SpMV: i_t = N · q_t (cuSPARSE SpMV on GPU; scatter on CPU) - -K2 — RHS assembly: - b_rhs = c2·i_t + c3·q_t + c4·q_prime_t - -[optional leakance, when params.use_leakance: true] - area_z = (p · depth)^q_eps · length - zeta = leakance_factor · area_z · K_D · (depth − d_gw) - b_rhs = b_rhs − zeta - -A-values: a_values = assemble_primitive(c1) [CSR values of A = I − c1·N] - -SpSV: x_sol = triangular_csr_solve(a_values, b_rhs) [lower-triangular] - -K3 — clamp: - q_next = clamp_min(x_sol, discharge_lb) -``` - -On the CUDA path (SP-10), K1+K2+K3 are fused `#[cube]` kernels, and the captured per-step sequence is **K1 → SpMV → K2 → assemble → SpSV → K3** — six kernel launches replayed as one `cuGraphLaunch`. - -**Cold start (t = 0):** solves `(I − N)·Q_0 = q'_0`. On a linear chain this reduces to `Q_0[i] = Σ_{j ≤ i} q'_0[j]` (cumulative sum). - ---- - -## 5. KAN head architecture - -``` -Linear(F, H) - → KanLayer(H, H) × num_hidden_layers [ALL layers receive the SAME init seed — DDR kan.py :24-34 quirk] - → Linear(H, P) - → Sigmoid -→ output in [0, 1] (denormalized to physical units in setup_inputs via config.rs bounds) -``` - -- **F** = number of catchment attributes, **H** = hidden size, **P** = number of learnable routing parameters -- **No inter-block ReLU** — DDR's `kan.py` has none; adding one breaks parity -- `rskan` version as of 2026-07-05: **v0.1.3** (verify with `grep rskan Cargo.toml`) -- All `num_hidden_layers` KanLayers use the **same seed** (a DDR quirk preserved intentionally for parity — see `src/nn/kan_head.rs`) - ---- - -## 6. Operational traps and known weak points - -### STALE-BINARY TRAP (high severity — has caused silent wrong results) - -`cargo build` and `cargo run` do NOT update `~/.cargo/bin/ddrs`. If you type `ddrs run` after editing `src/`, you silently execute the old binary. The manifest's `git.sha` is stamped from `.git` at runtime, not from the binary, so the run log looks current. - -**After any `src/` change, do ONE of:** -```bash -cargo install --path . # canonical refresh -# or, faster if target/release is warm: -cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs -# or bypass the installed copy entirely: -cargo run --release --bin ddrs -- run --workflow … -``` - -**Self-check:** current checkpoints are **directories** (`.ddrs/runs//checkpoints/epoch_E_mb_M/head.mpk`). A stale pre-checkpoint-resume binary writes flat files (`epoch_E_mb_M.mpk`). Flat files = stale binary. - -This trap caused the 2026-07-01 leakance×hourly 2×2 first run: the hourly cell silently ran flat-repeat-24 because the installed binary predated the disaggregation feature. - -### CUDA Graphs mask NaN (high severity) - -`use_cuda_graphs: true` can return stale finite loss when the forward produces NaN. Always validate new forward-path changes with `use_cuda_graphs: false` before benchmarking with graphs on. - -`leakance + use_cuda_graphs: true` is rejected at config load time — this combination is a hard error, not a silent wrong result. - -### Checkpoint resume drifts slightly - -Checkpoints store weights/moments in **f16** (`CompactRecorder = HalfPrecisionSettings`). A resumed trajectory diverges slowly from an uninterrupted one. Exact state (epoch, mini-batch cursor, RNG permutation) is preserved, but weight precision is not. See `docs/2026-06-07-checkpoint-resume-handoff.md` follow-up #1. - -### Fixture regeneration caveat (as of 2026-07-05) - -The DDR reference state used to validate V1 lives only in the desktop's `~/projects/ddr` working tree (unpushed `geometry/trapezoidal.py` work). A fixture regenerated from a clean DDR clone diverges ~1% at every ddrs commit — that is a wrong reference, not a port bug. See `.claude/references/ddrs-comparing-to-ddr.md` §Regenerating fixtures. - -### Worktree binary path - -Fresh worktrees lack the gitignored `output/` and fixture directories. Relative `target/release` resolves to the main tree's stale binary in some shells. Always use absolute paths or `cargo run` in worktrees. See `.claude/memory/ddrs-worktree-gotchas.md`. - ---- - -## 7. Performance numbers (as of 2026-07-05) - -### CONUS network - -| Metric | Value | -|---|---| -| Reaches (CONUS MERIT) | 346,321 | -| Edges | 338,814 | -| Gauge training set | ~2,365 (CONUS) | - -### Summed-Q' baseline (no routing, no learned params) - -This is the sanity floor: per-gauge sum of upstream divide Q' over the eval window. - -| Metric | Value | -|---|---| -| Median NSE | 0.689 | -| Median KGE | 0.723 | - -**If a trained run does not beat NSE 0.689, routing is not earning its keep. Check training loss curves and KAN gradient stats first.** - -### Best trained result (as of 2026-07-05) - -Precip-driven disaggregation + L1 loss, 2,365 CONUS gauges, eval 2026-06-23: - -| Metric | Value | -|---|---| -| Median NSE | 0.715 (+0.037 vs baseline) | -| Median KGE | 0.711 (−0.012 vs baseline) | - -**KGE does NOT beat the summed-Q' baseline in any config as of 2026-07-05.** NSE does (+0.037 with precip disagg). The NSE gain is real; the KGE regression traces to over-attenuation of flood peaks (the L1 / NSE gradient rewards the MC solver for attenuating, reducing `α = σ_sim/σ_obs` below 1). - -The `nnse-kge` loss mode (`experiment.loss.kind: nnse-kge`) exists to restore the KGE gradient, but no validated CONUS result with this mode is available as of 2026-07-05. - ---- - -## 8. Leakance — experimental GW–SW water-loss term - -### What it is - -A losing-stream correction subtracted from the routing RHS `b` at each timestep: - -``` -zeta = leakance_factor · area_z · K_D · (depth − d_gw) -area_z = (p · depth)^q_eps · length (plan-view wetted area, m²) -b ← b − zeta positive zeta = losing reach -``` - -Implementation: `src/routing/leakance.rs`. Gradient is analytical via `TimestepLeakanceOp: Backward`. - -### How to enable (three required config changes) - -```yaml -params: - use_leakance: true # activates term; forces use_cuda_graphs: false - parameter_ranges: - K_D: [1.0e-8, 1.0e-5] # log-space; hydraulic exchange rate, 1/s - d_gw: [-2.0, 2.0] # groundwater depth offset, m - leakance_factor: [0.0, 1.0] # dimensionless scale - -kan_head: - learnable_parameters: [n, q_spatial, x_storage, K_D, d_gw, leakance_factor] -``` - -Note: original K_D range was `[1e-8, 1e-6]`. The recoverability experiment (2026-07-04) widened to `[1e-8, 1e-5]` to achieve 58/96 expressible sites (vs 23/96 at the original ceiling). Use `[1e-8, 1e-5]` for any future leakance work. - -### Gradient-exactness guard (run after any change to leakance.rs) - -```bash -cargo test --test leakance_gradcheck # analytical ≈ finite-difference (8/8) -cargo test --test leakance_off_parity # byte-identical to no-leakance when off (3/3) -cargo test --test zeta_accum # eval zeta == what was subtracted from b (6/6) -cargo run --release --example compare_ddr_sandbox # must still report ABSOLUTE MATCH -``` - -### Leakance status summary (as of 2026-07-05) - -**2×2 verdict (leakance × forcing, 2026-07-01): GO-marginal.** -- Leakance + hourly: ΔNSE +0.0005, ΔKGE +0.0018 on the losing-stream subset (55.5% of gauges improve) -- Leakance + daily: ΔNSE −0.0017, ΔKGE −0.0009 (35.6% improve — hurts) -- Zeta gate: |zeta| > 0.01 m³/s on 10.4% of 64,892 eval reaches (bar: ≥10%) - -**Low-zeta diagnosis (2026-07-02):** - -| Hypothesis | Verdict | Key evidence | -|---|---|---| -| H1 — K_D box clips flux | REFUTED | 71.5% of reaches CAN exceed 0.01 m³/s inside the current box; median utilization 3.4% | -| H2 — driving-head starvation | SUPPORTED | median head `(depth − d_gw)` = 0.02 m; 47% of reaches gaining at eval-window mean | -| H3 — KAN variance collapse | REFUTED | d_gw–meanP Spearman +0.71; K_D–aridity +0.61 — strong learned structure | -| H4 — gauge bias / gradient starvation | SUPPORTED | zeta–uparea ρ +0.76; gauged median |zeta| 6.7e-3 vs ungauged 5.9e-4; dry/wet ratio 0.40 (inverted from physics) | -| H5 — equifinality (n absorbs loss) | SUPPORTED (daily only) | daily Δn = +0.012 (0.59 IQR, ~20%); hourly Δn nil (0.05 IQR) | -| H6 — wrong yardstick | REFUTED | fractional loss agrees: 8.4% lose >1% of local flow | -| H7 — d_gw model-form error | REFUTED | 0.0% of d_gw at bounds, incl. dry tercile | - -**Implication of diagnosis: K_D widening alone is NOT recommended.** The binding constraint is the training signal (H2 + H4), not the parameter box. The pre-registered Phase-3 gate FAILED; the widened-K_D retrain was not run. This supersedes the "widen K_D — top follow-up" recommendation from the 2026-07-01 findings. - -**Gradient probe (2026-07-03, worktree `origin/worktree-zeta-sensitivity`):** - -| Hypothesis | Bar | Measured | Verdict | -|---|---|---|---| -| P1 — starvation (gradient dead off-gauge) | gauged/ungauged |g| ≥ 10× | 1.5× (trained), 2.9× (cold) | REFUTED | -| P2 — rejection (gradient pushes zeta down) | >67% dry-tercile push-down | 52.5% (≈ neutral) | REFUTED | -| P3 — detectability (real-magnitude loss visible at gauge) | ≥10% of Ref δ=0.01 probes detectable | 4.2% (4/96); delta is 53× smaller than median 5% discharge-uncertainty band | NO-GO | - -**Synthetic recoverability positive control (2026-07-04, worktree):** - -The positive control FAILED: median recovery ratio = 0.009 (bar: ≥0.5). Root cause: the windowed training objective has a ~130× hotstart-transient noise floor. Continuous residual with teacher weights on teacher obs = 0.0076 mean L1; step-0 windowed training loss = 1.017. The planted signal (0.8% of training loss) is invisible. After 5 epochs, Adam actively degrades the model (continuous residual grew from 0.0076 to 0.4431 — 58× worse than not training). - -**Leakance identifiability is NOT proven. The positive control must pass (Phase B objective: windowed training loss ≤ 0.25 mean L1, i.e. ≤10% of a converged run's loss) before any identifiability claim can be made.** - ---- - -## 9. Phase B objective and current state (as of 2026-07-05) - -**Phase B goal:** state-cache hotstart — inject continuous-run discharge state at each training window boundary to eliminate the hotstart-transient noise floor. - -**Target:** windowed training loss ≤ 0.25 mean L1 (≤10% of a converged run's 1.017 step-0 loss). - -**Status:** NOT YET MET as of 2026-07-05. The state-cache infrastructure (`experiment.state_cache`, `src/data/store/obs_writer.rs`, `src/training/forward.rs` injection seam, `--mode state-cache` in probe binary) is implemented in `origin/worktree-zeta-sensitivity` but the floor validation target has not been hit. - -**Until Phase B passes, do not claim leakance is learnable from gauge-only supervision.** - ---- - -## 10. CLI quick-reference - -### First-time setup - -```bash -cargo install --path . # installs ddrs to ~/.cargo/bin/ -ddrs plan # GPU probe + smoke test + writes ddrs.yaml (opens $EDITOR) -ddrs run --workflow train-and-test # train + eval + write manifest -``` - -### After any src/ change - -```bash -cargo install --path . # or: cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs -``` - -### Key commands - -```bash -ddrs sources use conus-hourly # switch to hourly AORC precip source group -ddrs sources list # show active group (* = match) -ddrs show # inspect run manifest -ddrs status # workspace summary + disk usage -ddrs gc --keep 5 --keep-successful # prune old runs - -# Hourly forcing requires BOTH: -# 1. ddrs sources use conus-hourly (adds aorc_precip path) -# 2. kan_head.disaggregation.use_precip: true in ddrs.yaml -# Without aorc_precip, config with use_precip: true is a hard error. - -# Resume from checkpoint: -# Set experiment.checkpoint: .ddrs/runs//checkpoints/epoch_E_mb_M in ddrs.yaml -# Then raise experiment.epochs above E. -``` - -### Regression tests (run after touching core routing) - -```bash -cargo run --release --example compare_ddr_sandbox # V1: must print ABSOLUTE MATCH -DDRS_FORCE_GRAPHS=1 cargo run --release --example compare_ddr_sandbox # V9: CUDA graphs bit-match -cargo test --test mmc # hotstart, coefficients, forward, autodiff -cargo test --test sparse_gradcheck # CsrSolveOp backward -cargo test --test sp8_gradcheck # fused TimestepOp backward -cargo test --test leakance_gradcheck # leakance backward (8/8) -cargo test --test leakance_off_parity # byte-identical to no-leakance (3/3) -cargo test --test zeta_accum # zeta diagnostic identity (6/6) -``` - ---- - -## 11. Workspace layout - -| Path | Purpose | -|---|---| -| `ddrs.yaml` | Workflow + experiment config (gitignored) | -| `.ddrs/system.json` | GPU/driver/smoke-test record | -| `.ddrs/sources.lock` | Fingerprints of data_sources paths | -| `.ddrs/adjacency//` | Cached CONUS + gauges adjacency zarr stores (content-addressed) | -| `.ddrs/baselines//` | Cached summed-Q' baseline (blake3 of data sources + time window) | -| `.ddrs/runs//manifest.json` | Per-run manifest (config + sources + git SHA + outputs) | -| `.ddrs/runs//config.yaml` | Snapshot of the config that produced this run | -| `.ddrs/runs//run.log` | Timestamped stdout+stderr (fd-level tee) | -| `.ddrs/runs//checkpoints/epoch_E_mb_M/` | Checkpoint directory: `head.mpk`, `optim.mpk`, `state.json` | -| `.ddrs/runs//kan_parameters.nc` | Eval-window per-reach zeta/zeta_net/depth_mean/area_z_mean/q_mean | - ---- - -## 12. Data sources summary - -| Source | Type | Path (as of 2026-07-05) | -|---|---|---| -| MERIT adjacency | managed zarr (built from fabric) | `.ddrs/adjacency//` | -| Streamflow Q' (CONUS) | icechunk | `/mnt/ssd1/data/icechunk/merit_dhbv2_UH_retrospective.ic` | -| USGS observations (CONUS) | icechunk | `/mnt/ssd1/data/icechunk/usgs_daily_observations` | -| AORC precip (hourly) | zarr-v3, catchment-major, mm/hr | `/mnt/ssd1/data/aorc/merit_unit_catchments.zarr` | -| Global streamflow Q' | zarr-v2 multi-zone (60 zones) | `/gpfs/hjj5218/data/dmc_forcing/streamflow/zarr/8km/merit_global_v2.7` | -| Global observations | zarr-v2, one array per `Provider__GageId` | `/gpfs/hjj5218/data/dmc_forcing/observation/dMC_global_v3.1` | - -Global Q' units: m³/s (confirmed empirically — no units attribute on the zarr). Time axis: CF `days since 1980-01-01`. ~42k fabric reaches lack predictions → 0.001 fill at read. - -Hourly AORC precip: zarr-v3, catchment-major (COMID-first), mm/hr, starts 1980-01-01 UTC. Experiment windows using hourly forcing must not reach into 1980 (hourly-lstm store starts 1981-01-01). - ---- - -## 13. Branches - -| Branch | Description | -|---|---| -| `master` | Main integration branch | -| `unit_catchments` | Current working branch (as of 2026-07-05) | -| `origin/worktree-zeta-sensitivity` | Most advanced — Phase B state-cache hotstart, gradient probe, recoverability control, unit-catchment attribute wiring | - ---- - -## Provenance and maintenance - -Re-verify commands (copy-pasteable, all from project root): - -```bash -# V1 invariant -cargo run --release --example compare_ddr_sandbox - -# Leakance gradient-exactness suite -cargo test --test leakance_gradcheck && cargo test --test leakance_off_parity && cargo test --test zeta_accum - -# KAN head parity -cargo test --features fixtures --test kan_head_init_repro --test kan_head_init_parity --test kan_head_fixture_forward --test kan_head_fixture_backward - -# Sparse backward -cargo test --test sparse_gradcheck && cargo test --test sp8_gradcheck - -# Check rskan version -grep rskan Cargo.toml - -# Check current binary is fresh (flat files = stale) -ls ~/.cargo/bin/ddrs -la && ls .ddrs/runs/ 2>/dev/null | tail -3 -``` - -Ground-truth sources read to produce this skill: `CLAUDE.md`, `.claude/ARCHITECTURE.md`, `.claude/references/ddrs-burn-autograd.md`, `.claude/references/ddrs-architecture.md`, `docs/2026-07-02-leakance-diagnosis-findings.md`, `origin/worktree-zeta-sensitivity:docs/2026-07-03-zeta-gradient-probe-findings.md`, `origin/worktree-zeta-sensitivity:docs/2026-07-04-synthetic-recoverability-findings.md`. Volatile facts dated 2026-07-05. Re-read those files when key numbers or experiment verdicts change. diff --git a/.claude/skills/ddrs-build-and-env/SKILL.md b/.claude/skills/ddrs-build-and-env/SKILL.md deleted file mode 100644 index 2fad55a..0000000 --- a/.claude/skills/ddrs-build-and-env/SKILL.md +++ /dev/null @@ -1,439 +0,0 @@ ---- -name: ddrs-build-and-env -description: Use when setting up a fresh ddrs checkout, diagnosing build failures, hitting the forked-dependency trap, missing fixture errors, static netcdf/HDF5 cmake issues, stale-binary symptoms, or CUDA graphs masking NaN. Also use when cargo build succeeds but runtime behaves as an old version, or when fixture regeneration is needed after DDR solver changes. ---- - -# ddrs Build and Environment Runbook - -## Glossary (read once; terms used throughout) - -| Term | Meaning | -|---|---| -| **BURN** | Rust deep-learning framework (like PyTorch but for Rust). Version 0.21 in ddrs. | -| **DDR** | The Python/PyTorch reference implementation at `~/projects/ddr`. ddrs is its Rust port. | -| **V1 gate** | The regression test that must always pass: `compare_ddr_sandbox` reports ABSOLUTE MATCH (max abs diff < 1e-3 m³/s). | -| **`[patch.crates-io]`** | Cargo mechanism to globally replace a dependency's source. ddrs uses this to swap crates.io `cubecl`/`burn` for forked GitHub branches. | -| **KAN head** | Kolmogorov-Arnold Network routing head (`rskan::KanLayer`). Replaces MLP. Must stay at tag v0.1.3. | -| **f32 invariant** | All routing-core tensors stay float32. No f64, bf16, or mixed precision. | -| **uv** | Python package manager (like pip + venv). DDR's venv is managed by uv; needed for fixture regeneration only. | -| **icechunk** | Transactional Zarr-over-filesystem store for streamflow + observations data. | -| **cuSPARSE** | NVIDIA sparse linear algebra library. Used for the GPU triangular solve in `src/sparse.rs`. | - ---- - -## When NOT to use this skill - -| If you need... | Use instead | -|---|---| -| MC routing algorithm math | `.claude/references/ddrs-algorithm.md` | -| Autograd / sparse backward internals | `.claude/references/ddrs-burn-autograd.md` | -| DDR parity / V1 failure debugging | `.claude/references/ddrs-comparing-to-ddr.md` | -| Training a run from scratch | `CLAUDE.md` §"ddrs CLI" or README §"Getting started" | -| Leakance experiment status | `docs/2026-07-01-leakance-hourly-findings.md` | -| Architecture diagram | `.claude/ARCHITECTURE.md` | - ---- - -## 1. Prerequisites Checklist - -Before `cargo build` can succeed, verify every item below. - -### 1a. Rust toolchain - -```bash -rustc --version # must be >= 1.80; tested on 1.94.0 as of 2026-07-05 -cargo --version -``` - -Install or update via rustup: - -```bash -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -rustup update stable -``` - -### 1b. cmake (required for static netcdf/HDF5) - -`Cargo.toml` declares `netcdf = { version = "0.12", features = ["static"] }`. The `static` feature compiles bundled HDF5 and netcdf-c from source during `cargo build`. This avoids depending on system-installed dev packages (HPC hosts often have only Intel-MPI-flavored shared libs that would break `ddrs` outside `module load`). - -**Consequence:** cmake must be available on your PATH before building. - -```bash -cmake --version # must exist; any reasonably recent version works -``` - -If missing: -```bash -# Debian/Ubuntu -sudo apt-get install cmake -# Arch -sudo pacman -S cmake -# HPC (check module system) -module load cmake -``` - -### 1c. CUDA Toolkit - -Required only for the GPU path (`sparse_solver: cuda`, `use_cuda_graphs: true`). The CPU/NdArray backend path builds and runs without CUDA. - -```bash -nvcc --version # want CUDA 12+ (13.2 validated as of 2026-07-05) -nvidia-smi # driver must support the CUDA version -``` - -Validated configurations: -- RTX 4080 SUPER, driver 610.43.02, CUDA 13.2 (desktop, as of 2026-07-05) -- 8× A100, driver 575.57.08, CUDA 12, sm_80 (HPC) - -For CPU-only machines: skip CUDA setup. Override the default GPU config at runtime (see §5 "CPU-only override"). - -### 1d. git (for fork resolution) - -The forked cubecl and burn are fetched via HTTPS git by Cargo automatically. No local clones of cubecl or burn are needed — `git clone ddrs && cargo build` is sufficient. - -```bash -git --version # any recent version -``` - ---- - -## 2. Build Steps - -```bash -# Clone ddrs -git clone git@github.com:taddyb/ddrs ~/projects/ddrs -cd ~/projects/ddrs - -# Release build (LTO=thin; matches what ddrs CLI uses) -cargo build --release - -# Sanity check: V1 gate -mkdir -p output -cargo run --release --example compare_ddr_sandbox -# Expected last line: "verdict: ABSOLUTE MATCH (max abs < 1e-3 m³/s)" -``` - -The first build fetches: -- `github.com/taddyb/cubecl` branch `ddrs-release` (all cubecl-* crates) -- `github.com/taddyb/burn` branch `ddrs-sp7-primitive-ctor` (all burn-* crates) -- `github.com/taddyb/rskan` tag `v0.1.3` - -This takes several minutes on first run; subsequent builds use the Cargo registry cache. - ---- - -## 3. The Forked-Dependency Trap - -### What is patched and why - -`Cargo.toml` contains a `[patch.crates-io]` block that replaces the published crates.io versions of cubecl and burn with fork branches on github.com/taddyb: - -``` -[patch.crates-io] -cubecl = { git = "https://github.com/taddyb/cubecl.git", branch = "ddrs-release" } -cubecl-cuda = { git = "..." } # ... and 8 more cubecl-* crates -burn-cubecl = { git = "https://github.com/taddyb/burn.git", branch = "ddrs-sp7-primitive-ctor" } -burn-autodiff = { git = "..." } # ... and 12 more burn-* crates -``` - -The patches add exactly two `pub` accessors needed by ddrs's cuSPARSE GPU solve (SP-7): - -| Crate | Added accessor | -|---|---| -| `cubecl-cuda` 0.10 | `pub fn CudaServer::stream() -> CUstream` | -| `burn-cubecl` 0.21 | `pub fn CubeTensor::from_handle(...) -> Self` | - -These were `pub(crate)` in the upstream releases. The plan is to upstream them as SP-8 and remove `[patch.crates-io]` once merged. - -### Rules when working with forks - -1. **All burn-* crates must come from the same fork branch.** If Cargo resolves any burn-* crate from crates.io while another resolves from the fork, you get duplicate `Device` trait objects and cryptic link errors. The `[patch.crates-io]` block covers all 13 burn sub-crates; do not add a direct dependency on a crates.io burn sub-crate that would escape the patch. - -2. **Same rule for cubecl-***: all 10 crates in the monorepo must come from `ddrs-release`. - -3. **To iterate on the fork locally**: push your changes to the fork branch, then pull into ddrs with `cargo update -p cubecl` (or the changed crate). Do NOT commit `path = "..."` overrides — they break the public build. - -4. **rskan is pinned to a tag**, not a branch: `rskan = { git = "https://github.com/taddyb/rskan.git", tag = "v0.1.3" }`. Bumping the tag requires re-running the KAN parity sweep (CLAUDE.md invariant 6-7). - -### Diagnosing fork resolution failures - -``` -error[E0277]: the trait `Device` is not implemented for ... -``` -→ burn crate split across crates.io and fork. Run `cargo tree -p burn-std` to find which sub-crate is from crates.io. Add it to `[patch.crates-io]`. - -``` -error: failed to resolve patches for ... -``` -→ The fork branch was renamed or deleted. Check `github.com/taddyb/cubecl` or `github.com/taddyb/burn` for current branch name. - ---- - -## 4. Static netcdf/HDF5 Build Details - -`netcdf = { version = "0.12", features = ["static"] }` causes Cargo's build script to: -1. Download HDF5 and netcdf-c sources. -2. Compile them via cmake during `cargo build`. -3. Link them statically into the final binary. - -This is intentional: the HPC hosts lack usable `libnetcdf-dev`/`libhdf5-dev` packages (the module-provided ones are Intel-MPI-flavored and break `ddrs` outside `module load`). - -**Build time impact**: first build takes several extra minutes for the cmake compile. Subsequent builds are cached by Cargo. - -**If cmake is not found**: -``` -error: failed to execute process `cmake`: No such file or directory -``` -Install cmake (see §1b) and retry `cargo build`. - -**If cmake finds system HDF5 but produces link errors**: the `static` feature should bypass system HDF5. Ensure you have not set `HDF5_DIR` in your environment pointing at an incompatible installation: -```bash -unset HDF5_DIR -unset NETCDF_DIR -cargo build --release -``` - ---- - -## 5. CPU-only Override - -On a machine without CUDA, the default config (`config/merit_training.yaml`) fails because it requests `sparse_solver: cuda`. Override it via a minimal config file: - -```yaml -# cpu_override.yaml (do not commit) -sparse_solver: cpu -use_cuda_graphs: false -``` - -```bash -ddrs --config cpu_override.yaml plan -ddrs --config cpu_override.yaml run --workflow train -# or for just the V1 gate (V1 always defaults to NdArray/CPU): -cargo run --release --example compare_ddr_sandbox # no override needed -``` - -The CPU NdArray backend is the default for `compare_ddr_sandbox`; V1 always passes on CPU. Only training-scale runs need the GPU config override. - ---- - -## 6. Gitignored Fixtures and Outputs - -Three categories of files are gitignored and must be recreated after a fresh clone: - -### 6a. Sandbox fixtures (`/fixtures/`, `examples/fixtures/`) - -These are the V1 gate inputs generated by DDR's Python solver. They are gitignored because they are derived artifacts. `tests/fixtures/` IS tracked; only root-level and examples-level fixtures are excluded. - -```bash -# Regenerate after DDR's solver changes, or on a fresh clone -cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/export_ddr_sandbox.py -``` - -CRITICAL CAVEAT (as of 2026-07-05): The valid V1 fixture lives ONLY in the desktop's `~/projects/ddr` working tree. That tree contains unpushed work — `src/ddr/geometry/trapezoidal.py` — that does not exist in any DDR public commit. Regenerating from a clean DDR clone produces a ~1%-divergent reference (max abs ≈ 0.55 m³/s) that would make V1 fail at every ddrs commit. This is a wrong-reference artifact, not a port bug. Until DDR's geometry work is pushed, only the desktop DDR tree produces a valid V1 fixture. - -### 6b. Output directory (`output/`) - -The `compare_ddr_sandbox` example writes to `output/ddrs_vs_ddr.{csv,png}` using `File::create`, which does NOT `mkdir -p`. A fresh clone will panic on the file create. - -```bash -mkdir -p output -cargo run --release --example compare_ddr_sandbox -``` - -### 6c. Workspace artifacts (`.ddrs/`) - -The entire `.ddrs/` directory is gitignored. This includes adjacency caches, baseline caches, run manifests, checkpoints, and run logs. They are rebuilt by `ddrs plan` on first run. - ---- - -## 7. Stale-Binary Trap - -This is one of the most common sources of silent wrong behavior. - -**The problem**: `ddrs` on your PATH is `~/.cargo/bin/ddrs`. `cargo build` and `cargo run` compile into `target/release/ddrs` but do NOT copy it to `~/.cargo/bin/`. If you edit `src/` and then type `ddrs run`, you silently execute the old binary. - -The manifest's `git.sha` field is stamped from `.git` at runtime, NOT from the binary. A run can look like current code in the manifest while a weeks-old binary actually executed. This caused the 2026-07-01 leakance×hourly 2×2 to produce byte-identical hourly cells — the installed binary predated the disaggregation feature. - -**Self-check**: current checkpoints are DIRECTORIES (`epoch_E_mb_M/head.mpk` etc.). If you see flat `.mpk` files at `epoch_E_mb_M.mpk`, you ran a pre-checkpoint-resume binary. - -**Fix after every `src/` change**: - -```bash -# Canonical (always correct): -cargo install --path . - -# Faster when target/release is already built: -cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs - -# Bypass the installed binary entirely (safest during development): -cargo run --release --bin ddrs -- run --workflow train -``` - ---- - -## 8. CUDA Graphs Masking NaN - -**The trap**: `use_cuda_graphs: true` records a graph on the first forward pass and replays it on subsequent steps. If the first forward passes with a finite loss but a later one would produce NaN (e.g., from a bad parameter initialization or data batch), the graph replay returns the stale finite loss from the captured pass. You see a training run that appears to converge smoothly but is actually computing nothing. - -**Affected config key**: `use_cuda_graphs: true` in `ddrs.yaml` or `config/merit_training.yaml`. - -**Config rejection rule**: `use_leakance: true` + `use_cuda_graphs: true` is REJECTED at config load time (no exception path). These cannot be used together; the leakance kernel has no separate CUDA graph capture path. - -**Diagnosis**: - -```bash -# 1. Disable graphs and rerun the suspicious config -# Add to ddrs.yaml: -# use_cuda_graphs: false - -# 2. Watch for NaN in the loss log — if it now appears, graphs were masking it - -# 3. Find the NaN source: typically a parameter initialized to zero or -# a data batch with all-NaN streamflow -``` - -**Rule**: always validate new configs and new parameter initializations with `use_cuda_graphs: false` first. Only re-enable graphs after confirming the forward is NaN-free. - ---- - -## 9. Critical Invariants (Do Not Break) - -Breaking any of these makes the ddrs port meaningless or incorrect. - -| # | Invariant | Test | -|---|---|---| -| 1 | `compare_ddr_sandbox` must report ABSOLUTE MATCH (max abs < 1e-3 m³/s) | `cargo run --release --example compare_ddr_sandbox` | -| 2 | f32 throughout routing core; no f64/bf16 casts in `src/routing/`, `src/geometry.rs`, `src/sparse.rs` | `grep -rn 'f64\|bf16\|cast\|to_dtype' src/routing/ src/geometry.rs src/sparse.rs` | -| 3 | Adjacency is topologically ordered, lower-triangular (`rows[k] >= cols[k]`) | `cargo test data_zarr_store::conus_adjacency_loads_real_merit_zarr` | -| 4 | Hand-written sparse backward in `src/sparse.rs` must NOT be replaced with tape unrolling | `cargo test --test sparse_gradcheck` | -| 5 | KAN head = `rskan::KanLayer` via `src/nn/kan_head.rs`; no MLP placeholder; no inter-block ReLU | `cargo test --test kan_head` | -| 6 | rskan pinned to tag `v0.1.3` in Cargo.toml | `grep 'rskan.*tag' Cargo.toml` | -| 7 | KAN parity vs DDR must pass on every PR touching `src/nn/`, rskan pin, or DDR's `nn/kan.py` | See §11 KAN parity command | - ---- - -## 10. Full Verification Command Set - -Run these in order after a fresh build or after touching `src/`: - -```bash -# V1 gate — must report ABSOLUTE MATCH -mkdir -p output -cargo run --release --example compare_ddr_sandbox - -# V1 gate on CUDA + graph-capture path (if GPU available) -DDRS_FORCE_GRAPHS=1 cargo run --release --example compare_ddr_sandbox - -# Sparse gradient correctness -cargo test --test sparse_gradcheck - -# Routing correctness (linear chain) -cargo test --test mmc mc_routes_linear_chain - -# Leakance gradient-exactness (if leakance was touched) -cargo test --test leakance_gradcheck -cargo test --test leakance_off_parity -cargo test --test zeta_accum -``` - ---- - -## 11. KAN Head Parity (after touching `src/nn/`, rskan pin, or DDR's `nn/kan.py`) - -```bash -cargo test --features fixtures \ - --test kan_head_init_repro \ - --test kan_head_init_parity \ - --test kan_head_fixture_forward \ - --test kan_head_fixture_backward -``` - -If DDR's `nn/kan.py` changed: regenerate fixtures first, then re-validate: -```bash -cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/dump_kan_fixture.py -cd ~/projects/ddrs && cargo test --features fixtures --test kan_head_fixture_forward --test kan_head_fixture_backward -``` - ---- - -## 12. DDR Reference Clone (for fixture regeneration) - -```bash -git clone git@github.com:mhpi/ddr ~/projects/ddr -cd ~/projects/ddr && uv sync --all-packages -``` - -uv creates a `.venv` automatically. Fixtures are generated by running scripts under this venv: - -```bash -cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/export_ddr_sandbox.py -``` - -The uv venv must remain at `~/projects/ddr/.venv` — ddrs scripts import from DDR's Python packages. - ---- - -## 13. Data File Paths - -`config/merit_training.yaml`'s `data_sources:` block references these paths. Edit the YAML to match your machine if they live elsewhere. - -| Source | Default path | -|---|---| -| Geospatial fabric | `riv_pfaf_7_MERIT_Hydro_v07_Basins_v01_bugfix1.shp` (+ sibling `.dbf`), or a `.gpkg` | -| MERIT adjacency | Managed — built by `ddrs plan` into `.ddrs/adjacency//` | -| Per-gauge subgraphs | Managed — same directory | -| Catchment attributes | `~/projects/ddr/data/merit_global_attributes_v2.nc` | -| Streamflow forcing | `/mnt/ssd1/data/icechunk/merit_dhbv2_UH_retrospective.ic` | -| USGS observations | `/mnt/ssd1/data/icechunk/usgs_daily_observations` | -| Gauges list | `~/projects/ddr/references/gage_info/gages_3000.csv` | - -To skip the managed adjacency build (e.g., you have pre-built zarr stores): -```yaml -# in ddrs.yaml, replace geospatial_fabric with: -conus_adjacency: /path/to/merit_conus_adjacency.zarr -gages_adjacency: /path/to/merit_gages_conus_adjacency.zarr -``` - ---- - -## 14. Common Failure Modes at a Glance - -| Symptom | Root cause | Fix | -|---|---|---| -| `cmake: No such file or directory` during `cargo build` | cmake not on PATH; needed for static netcdf | Install cmake (§1b) | -| `the trait Device is not implemented` link error | burn crate split across crates.io and fork | Check `cargo tree -p burn-std`; add missing crate to `[patch.crates-io]` | -| `failed to resolve patches` | Fork branch renamed or deleted | Check `github.com/taddyb/{cubecl,burn}` for current branch | -| `thread 'main' panicked at 'No such file or directory' (output/...)` | `output/` missing on fresh clone | `mkdir -p output` | -| V1 fails with max abs ≈ 0.55 m³/s | Fixtures regenerated from wrong DDR clone (no trapezoidal.py) | Use desktop's `~/projects/ddr` working tree | -| Training appears to converge but loss never moves | CUDA graphs masking NaN | Set `use_cuda_graphs: false` and check for NaN (§8) | -| `ddrs run` uses wrong/old feature after `cargo build` | Stale installed binary at `~/.cargo/bin/ddrs` | `cargo install --path .` (§7) | -| Checkpoint is a flat `.mpk` file, not a directory | Stale pre-checkpoint-resume binary executed | `cargo install --path .` and re-run | -| `use_leakance + use_cuda_graphs rejected at config load` | These two are mutually exclusive by design | Remove `use_cuda_graphs: true` from leakance configs | -| `fixtures/sandbox/` missing, V1 panics on CSV read | gitignored artifact not regenerated | `cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/export_ddr_sandbox.py` | - ---- - -## Provenance and maintenance - -Skill written 2026-07-05 from `Cargo.toml`, `CLAUDE.md`, `README.md`, `.claude/references/ddrs-setup.md`, `.claude/references/ddrs-comparing-to-ddr.md`, and `vendor/README.md`. - -Re-verification commands: -```bash -# Confirm fork branches still exist -git ls-remote https://github.com/taddyb/cubecl.git ddrs-release -git ls-remote https://github.com/taddyb/burn.git ddrs-sp7-primitive-ctor -git ls-remote https://github.com/taddyb/rskan.git refs/tags/v0.1.3 - -# Confirm static netcdf feature still declared -grep 'netcdf.*static' /home/tbindas/projects/ddrs/Cargo.toml - -# Confirm gitignore entries -grep -E 'fixtures|output|\.ddrs' /home/tbindas/projects/ddrs/.gitignore - -# Confirm Rust version meets minimum -rustc --version # must be >= 1.80 - -# V1 gate -mkdir -p /home/tbindas/projects/ddrs/output -cargo run --release --example compare_ddr_sandbox 2>&1 | grep -E 'verdict|ABSOLUTE|FAIL' -``` diff --git a/.claude/skills/ddrs-change-control/SKILL.md b/.claude/skills/ddrs-change-control/SKILL.md deleted file mode 100644 index 0c3464b..0000000 --- a/.claude/skills/ddrs-change-control/SKILL.md +++ /dev/null @@ -1,413 +0,0 @@ ---- -name: ddrs-change-control -description: "Use when reviewing, gating, or merging a change to ddrs source code, config files, or Cargo dependencies; when assessing whether a modification to src/routing/, src/sparse.rs, src/geometry.rs, src/nn/kan_head.rs, or Cargo.toml is safe; when a run produced unexpected results and binary staleness or an invariant violation may be the cause; or when designing an experiment that touches leakance, CUDA graphs, or the routing core." ---- - -# ddrs change-control runbook - -## Overview - -`ddrs` is a BURN-0.21 Rust port of the DDR differentiable Muskingum-Cunge routing model (Python/PyTorch reference at `~/projects/ddr/`). The port must remain **gradient-exact** against DDR. Breaking any of the seven invariants below makes the port meaningless; every PR that touches the affected files must clear its gate before merge. - -**Glossary for PyTorch engineers:** -- `BURN` — Rust deep-learning framework, analogous to PyTorch. BURN 0.21 is pinned. -- `Backward` — BURN's trait for a custom autograd function (analogous to `torch.autograd.Function`). `I` = backend (CPU/CUDA), `N` = number of saved tensors. -- `CsrPattern` — the sparsity structure of the river network adjacency matrix, stored as a Rust struct (row/col index arrays). Analogous to `torch.sparse_csr_tensor`. -- `KanLayer` — Kolmogorov-Arnold Network layer from the `rskan` crate (Rust equivalent of DDR's `kan.py`). -- `CompactRecorder` / `HalfPrecisionSettings` — BURN serializer for checkpoints. Saves weights as f16. -- `COMID` — unique 64-bit integer ID for each river reach in the MERIT-Hydro fabric. -- `Q'` (Qr) — lateral inflow forcing (divide-level runoff, m³/s) from the pre-trained DHBV2 model. Not observed discharge. -- `zeta` — leakance flux (m³/s), the GW–SW water-loss term. Subtracted from the routing RHS at every timestep. -- `rho-window` — a training mini-batch: a contiguous time slice of length `rho` (default 90 days) sampled from the training period. - ---- - -## When NOT to use this skill - -Do not use this skill for: -- **Plotting or analysis scripts only** (no `src/` change) — use `ddrs-eval-plots` instead. -- **Config tuning within documented safe ranges** (changing `experiment.epochs`, `learning_rate`, `batch_size`, loss weights) — no gate applies; these do not affect the routing core or port invariants. -- **Data source path changes only** — consult `CLAUDE.md §Data sources` directly. -- **CLI / workspace questions** — consult `CLAUDE.md §ddrs CLI`. - ---- - -## Change classification matrix - -Every change falls into one of four tiers. Look up the modified file(s) in the left column; the tier determines which gate checklist you must run. - -| Modified file(s) | Tier | Rationale | -|---|---|---| -| `src/routing/mmc.rs`, `src/routing/mmc_op.rs`, `src/routing/utils.rs` | **A — routing core** | Directly implements the Muskingum-Cunge timestep; must remain gradient-exact vs DDR | -| `src/routing/leakance.rs` | **A — routing core + leakance** | Custom `Backward` for the GW–SW term; both the forward kernel and analytical gradients must stay exact | -| `src/geometry.rs` | **A — routing core** | Trapezoidal geometry; changes cascade into every geometry-dependent variable | -| `src/sparse.rs` | **A — routing core** | Hand-written CSR triangular solve + custom `CsrSolveOp: Backward`; O(nnz) autograd tape invariant | -| `src/nn/kan_head.rs` | **B — KAN head** | Must match DDR `kan.py` exactly; rskan version pin governs this | -| `Cargo.toml` (rskan tag) | **B — KAN head** | rskan pin is the single authoritative version for KAN parity | -| `src/config.rs` | **C — config/ranges** | Parameter ranges and log-space flags affect denormalization; wrong range silently mis-scales gradients | -| `src/training/loss.rs` | **C — objective** | Autograd is unchanged (invariant 4 intact) but loss changes affect all metrics comparisons | -| `src/training/forward.rs` | **C — training path** | Disaggregation, leakance threading; changes can silently no-op features (see STALE-BINARY TRAP) | -| `config/experiments/*.yaml`, `config/sources/*.yaml` | **D — config only** | No Rust changes; validate with `ddrs plan` before running | -| Any other `src/` file | **C — default** | Run full test suite and DDR regression | - ---- - -## Tier A gate — routing core - -Run ALL of the following. A single failure is a merge blocker. - -```bash -# 1. DDR parity — THE non-negotiable regression -cargo run --release --example compare_ddr_sandbox -# Must print: "ABSOLUTE MATCH" with max abs diff < 1e-3 m³/s - -# 2. Core unit + integration tests -cargo test --lib -cargo test --test mmc -cargo test --test sparse_gradcheck - -# 3. Leakance gates (required even if you did not touch leakance.rs, -# because any routing change can disturb the leakance OFF parity) -cargo test --test leakance_gradcheck # 8/8 analytical ≈ finite-difference -cargo test --test leakance_off_parity # 3/3 byte-identical to no-leakance when off -cargo test --test zeta_accum # 6/6 accumulated zeta == headwater q difference -``` - -If you touched `src/routing/leakance.rs` specifically: -```bash -# Confirm gradient-exactness for all 8 leakance backward inputs -cargo test --test leakance_gradcheck -- --nocapture -``` - -**After passing all Tier A gates, run the binary self-check:** -```bash -# Stale-binary check: directory checkpoints = current binary -ls .ddrs/runs//checkpoints/ -# Must show directories like epoch_5_mb_9/, NOT flat files like epoch_5_mb_9.mpk -# Flat files = stale binary ran. Refresh: cargo install --path . -``` - ---- - -## Tier B gate — KAN head - -```bash -# Full KAN parity sweep (required on every PR touching src/nn/ or Cargo.toml rskan pin) -cargo test --features fixtures \ - --test kan_head_init_repro \ - --test kan_head_init_parity \ - --test kan_head_fixture_forward \ - --test kan_head_fixture_backward - -# Then run Tier A DDR parity to confirm the head change did not break routing -cargo run --release --example compare_ddr_sandbox -``` - -If a DDR-side change to `kan.py` broke the fixture: -```bash -# Regenerate under DDR's venv, then re-validate -cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/dump_kan_head.py -# Re-run the fixture tests above -``` - ---- - -## Tier C gate — config, training path, other src/ - -```bash -cargo test --lib -cargo test # full suite -cargo run --release --example compare_ddr_sandbox # DDR parity -``` - -For `src/training/forward.rs` changes that affect disaggregation or leakance threading, also run: -```bash -cargo test --test leakance_off_parity # leakance OFF must stay byte-identical -``` - ---- - -## Tier D gate — config files only - -```bash -# Validate the config parses and data sources resolve -ddrs plan --config config/experiments/.yaml \ - --workspace /home/tbindas/projects/ddrs/.ddrs -# Must exit 0 with no "drift" warnings -``` - ---- - -## The 7 non-negotiables with rationale and incidents - -### Invariant 1 — DDR sandbox ABSOLUTE MATCH - -**Rule.** `cargo run --release --example compare_ddr_sandbox` must print "ABSOLUTE MATCH" (max abs diff < 1e-3 m³/s on the 5-reach RAPID sandbox). Re-run after every change to `src/routing/`, `src/geometry.rs`, or `src/sparse.rs`. - -**Rationale.** The port exists to be gradient-exact against DDR. Any drift makes subsequent metric comparisons meaningless — you cannot tell whether a difference is a port bug or a genuine model improvement. - -**Caveat (as of 2026-06-06).** The reference DDR state lives only in the desktop's `~/projects/ddr` working tree (contains unpushed `geometry/trapezoidal.py` changes). A fixture regenerated from a clean DDR clone diverges ~1% per commit — that is a wrong reference, not a port bug. See `.claude/references/ddrs-comparing-to-ddr.md §Regenerating fixtures` before regenerating. - ---- - -### Invariant 2 — f32 throughout routing core - -**Rule.** No casts to f64 or bf16 inside `src/routing/`, `src/geometry.rs`, or `src/sparse.rs`. The DDR comparison sits at the f32 precision floor (~1e-7 relative difference per reach); any precision change breaks reproducibility. - -**Rationale.** Mixed precision introduces per-reach rounding that accumulates across the 346,321-reach CONUS network; the 1e-3 m³/s sandbox tolerance is calibrated for f32-only arithmetic. - ---- - -### Invariant 3 — lower-triangular adjacency - -**Rule.** The adjacency matrix must be topologically sorted and lower-triangular: `rows[k] >= cols[k]` for every non-zero entry. The forward-substitution solver (`triangular_solve_lower`) assumes no upstream values are uncomputed when it processes a reach. - -**Rationale.** Forward substitution over a topological order is the entire basis for the O(n) per-timestep solve. A non-lower-triangular entry means a downstream reach tries to read an upstream value before that upstream reach is solved — silent wrong output, no error. - -**Test.** `cargo test data_zarr_store::conus_adjacency_loads_real_merit_zarr` verifies the invariant on the real CONUS zarr store. - ---- - -### Invariant 4 — hand-written sparse backward - -**Rule.** Do NOT replace the hand-written `CsrSolveOp impl Backward` in `src/sparse.rs` with autograd-tape unrolling. - -**Rationale.** The entire point of the custom backward is O(nnz) tape entries per timestep. Tape unrolling would be O(n²) for a triangular solve — quadratic memory and time, infeasible at CONUS scale (346,321 reaches). The analytical backward is `∇A = -gradb[rows]·x[cols]`, exactly as in DDR's `torch.autograd.Function`. - -**Reference.** `.claude/references/ddrs-burn-autograd.md` has the full BURN-0.21 recipe. - ---- - -### Invariant 5 — KAN head architecture matches DDR - -**Rule.** The routing head is `rskan::KanLayer` via `src/nn/kan_head.rs`. The architecture is `Linear(F, H) → KanLayer(H, H) × num_hidden_layers → Linear(H, P) → Sigmoid`. No inter-block ReLU. All `num_hidden_layers` inner KanLayers receive the SAME seed (DDR's `kan.py` lines 24–34 quirk — preserved for parity). - -**Rationale.** DDR parity requires identical weight initialization. A ReLU between KAN blocks or different per-layer seeds changes the initialization and breaks the fixture tests. - -**What NOT to do.** Do not reintroduce the prior MLP placeholder. - ---- - -### Invariant 6 — rskan pinned to a tag - -**Rule.** `rskan` in `Cargo.toml` must be a git dependency pinned to a tag, currently `v0.1.3`. When updating, bump the tag, then re-run all Tier B tests and the Tier A DDR regression before merging. - -**Rationale.** An unpinned git dependency (`branch = "main"`) can change silently on `cargo update`, breaking KAN parity without any local code change. - -**Current pin (as of 2026-07-05):** -```toml -rskan = { git = "https://github.com/taddyb/rskan.git", tag = "v0.1.3" } -``` - ---- - -### Invariant 7 — KAN head parity on every relevant PR - -**Rule.** Any PR touching `src/nn/`, `Cargo.toml`'s rskan pin, or DDR's `nn/kan.py` must pass the full KAN parity suite (see Tier B gate above). - -**Rationale.** Invariant 5 is not self-enforcing at the compiler level. The fixture tests are the only automated check that the architecture, seed, and weight initialization actually match DDR. - ---- - -## The STALE-BINARY TRAP (historical incident: 2026-07-01) - -**Rule.** After touching ANY file under `src/`, refresh the installed binary before running experiments. - -```bash -# Canonical (always correct): -cargo install --path . - -# Faster if target/release is already built: -cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs - -# Bypass entirely (safest for one-off experiments): -cargo run --release --bin ddrs -- run --workflow train-and-test ... -``` - -**Why this matters.** `ddrs` on your PATH is `~/.cargo/bin/ddrs`. `cargo build` and `cargo run` do NOT update it. The manifest's `git.sha` is stamped from `.git` at runtime, so a run appears to have the correct SHA while silently executing a weeks-old binary. - -**What happened (2026-07-01).** The installed `ddrs` was dated 2026-06-03, before disaggregation (landed 2026-06-19) and leakance (landed 2026-06-29). The hourly-forcing cell silently ran flat repeat-24 with no leakance. Both hourly-ON and daily-ON cells produced byte-identical eval predictions (`52ec721`). The manifest showed `git.sha = 2cdd341` (correct HEAD), masking the stale binary completely. - -**Self-check.** Current checkpoints are DIRECTORIES: `.ddrs/runs//checkpoints/epoch_E_mb_M/head.mpk`. A stale pre-checkpoint-resume binary writes FLAT files: `epoch_E_mb_M.mpk`. Flat files = stale binary. - ---- - -## CUDA graphs + NaN masking (known gotcha) - -**Rule.** Validate model forwards with `use_cuda_graphs: false` when debugging NaN loss or unexpected constant loss. - -**Why.** `use_cuda_graphs: true` captures a kernel graph on the first forward pass and replays it on subsequent passes. If the first forward produces a NaN (e.g., during early training on bad data), the captured graph replays stale finite values rather than recomputing and propagating the NaN. The result is a constant finite loss that does not go to NaN even when the actual computation is invalid. See memory file `cuda-graphs-mask-nan.md` for the full diagnosis. - -**Hard constraint.** `params.use_leakance: true` combined with `use_cuda_graphs: true` is REJECTED at config load time. The leakance kernel is not captured in the current CUDA graph implementation; the rejection prevents silent wrong results. - ---- - -## Leakance-specific gates - -Leakance (`params.use_leakance: true`) is experimental and off by default. Any change that enables, modifies, or interacts with leakance must satisfy these gates in addition to the appropriate tier gates. - -### Enabling leakance requires three config changes together - -Missing any one causes either a config-load error or silent wrong behavior: - -| Config key | Required value | Why | -|---|---|---| -| `params.use_leakance` | `true` | Activates the leakance kernel in `route_timestep` | -| `kan_head.learnable_parameters` | Include `K_D`, `d_gw`, `leakance_factor` | Without these, the KAN head does not emit leakance params → all-zero zeta | -| `params.parameter_ranges.K_D` | `[1e-8, 1e-6]` (log-space) | Range gate; current recommendation is `[1e-8, 1e-5]` for recoverability experiments (see §Research status) | -| `params.parameter_ranges.d_gw` | `[-2, 2]` | Groundwater depth offset (m) | -| `params.parameter_ranges.leakance_factor` | `[0, 1]` | Dimensionless scale | -| `use_cuda_graphs` | `false` | Enforced by config load; leakance + graphs = rejected | - -### Leakance gradient-exactness gate - -```bash -cargo test --test leakance_gradcheck # 8/8 — all analytical grads match finite-diff -cargo test --test leakance_off_parity # 3/3 — OFF is byte-identical to no-leakance -cargo test --test zeta_accum # 6/6 — accumulated zeta == headwater q difference -cargo run --release --example compare_ddr_sandbox # must still say ABSOLUTE MATCH -``` - -### Leakance eval-time zeta diagnostic - -`dump_parameters` exports learned `K_D`/`d_gw`/`leakance_factor` per COMID but NOT the actual zeta flux (which depends on routed depth, only available during eval). The zeta diagnostic runs during `ddrs run --workflow train-and-test` Phase 2 automatically. For an existing checkpoint: - -```bash -cargo build --release --bin eval -target/release/eval \ - --config config/experiments/leakance_hourly_on.yaml \ - --checkpoint .ddrs/runs//checkpoints/epoch_5_mb_9 \ - --output /tmp/eval.zarr \ - --zeta-output .ddrs/runs//kan_parameters.nc -``` - -Output variables in `kan_parameters.nc` (dimension `COMID_eval`, 64,892 reaches on the CONUS eval network): -- `zeta` — mean |zeta| (m³/s) over eval window -- `zeta_net` — signed mean zeta (positive = losing reach) - -**GO/NO-GO bar.** |zeta| > 0.01 m³/s on at least 10% of eval reaches = zeta is physically active. - ---- - -## Research-status facts (as of 2026-07-05) - -These facts govern what claims can be made about leakance. Cite dates; do not generalize beyond what is measured. - -### Leakance 2×2 (as of 2026-07-01) — DONE - -Four valid arms, seed 42, eval window 1995/10/01–2010/09/30, 2,365 gauges: - -| arm | run id | NSE med | KGE med | -|---|---|---|---| -| hourly-OFF | `2026-06-23T02-49-12Z-conus-hourly-train-and-test` | 0.7153 | 0.7104 | -| hourly-ON | `2026-07-01T13-43-32Z-train-and-test` | 0.7145 | 0.7150 | -| daily-OFF | `2026-06-05T01-41-16Z-train-and-test` | 0.7004 | 0.7244 | -| daily-ON | `2026-07-01T21-20-27Z-train-and-test` | 0.6963 | 0.7250 | - -Losing-stream subset (1,883/2,365 gauges): hourly leakance ΔNSE +0.0005, ΔKGE +0.0018, 55.5% of gauges improve. Daily leakance ΔNSE −0.0017, ΔKGE −0.0009. Verdict: **GO — marginal** (3/3 gates met; zeta |>0.01| on 10.4% of 64,892 eval reaches). - -**Summed-Q' baseline (CONUS):** median NSE 0.689, KGE 0.723. Best trained result (precip-driven disagg + L1): median NSE 0.715, KGE 0.711 (2,365 gauges). NSE beats the baseline by +0.026; KGE does NOT beat the summed-Q' baseline in any config as of 2026-07-05. - -### Low-zeta diagnosis (as of 2026-07-02) - -| Hypothesis | Verdict | -|---|---| -| H1 — K_D ceiling clips zeta | REFUTED (71.5% of reaches CAN exceed 0.01 m³/s in-box; utilization median 3.4%) | -| H2 — driving-head starvation | SUPPORTED (median head 0.021 m; 47% of reaches gaining at eval-window mean) | -| H3 — KAN variance collapse | REFUTED (K_D–aridity ρ +0.61, d_gw–meanP ρ +0.71 — strong learned structure) | -| H4 — gauge bias / gradient starvation | SUPPORTED (gauged median |zeta| 11× ungauged; dry/wet zeta ratio 0.40, inverse of physics) | -| H5 — equifinality (daily only) | SUPPORTED (daily Δn +0.012, 0.59 IQR; hourly Δn nil) | -| H6 — wrong yardstick | REFUTED (fractional loss agrees: 8.4% of reaches lose >1% of local flow) | -| H7 — model-form error | REFUTED (0.0% of d_gw at bounds) | - -**Implication.** The K_D-widening follow-up (`[1e-8, 1e-5]`) recommended in the 2×2 findings is NOT recommended by the diagnosis. The diagnosis shows the K_D box is not the binding constraint. Widening K_D alone is expected to re-pin at the new ceiling with negligible zeta or skill change. - -### Gradient probe (as of 2026-07-03, worktree: zeta-sensitivity) - -| Probe | Verdict | Key number | -|---|---|---| -| P1 — gradient starvation | REFUTED | gauged/ungauged \|g\| ratio 1.5× trained, 2.9× cold (bar: ≥10×) | -| P2 — rejection at trained point | REFUTED | 52.5% of dry-tercile grads push zeta down (bar: >67%) | -| P3 — detectability | NO-GO | 4.2% of Ref probes detectable at δ=0.01 m³/s (bar: ≥10%); median 5%-band 0.531 m³/s vs planted signal 0.01 m³/s = **53× dilution** | - -P3 NO-GO means: gauge-only discharge supervision cannot distinguish real-world leakance magnitudes from measurement uncertainty. Transmission is fine (~95% fidelity); the problem is signal-to-noise at the sensor. - -### Synthetic recoverability positive control (as of 2026-07-04, worktree: zeta-sensitivity) - -| Metric | Measured | Verdict | -|---|---|---| -| R1 — recovery ratio median (n=58) | 0.009 | FAILED (bar: ≥0.5) | -| R2 — non-planted spatial precision | 1.11× baseline | PRECISE (trivial: model didn't move) | -| R3 — loss gap A vs B | A=1.339 vs B=2.317, +42% | A3) | - -**Root cause.** Windowed training objective (rho=90, warmup=5) has a ~130× hotstart-transient noise floor vs the planted signal. The continuous residual with teacher weights + teacher obs is 0.0076 mean L1; step-0 windowed training loss is 1.017. The optimizer chases irreducible initial-condition noise; after 5 epochs the continuous residual degrades from 0.0076 to 0.4431. - -**Implication.** Leakance identifiability is NOT proven. Phase B (state-cache hotstart, target: windowed loss ≤ 0.25 mean L1, ≤10% of a converged run) is required before any identifiability claim. Phase B is NOT yet complete as of 2026-07-05. - ---- - -## Binary management quick-reference - -| Goal | Command | -|---|---| -| Refresh installed binary after src/ change | `cargo install --path .` | -| Fast refresh (target/release already built) | `cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs` | -| Bypass installed binary for one run | `cargo run --release --bin ddrs -- run --workflow train-and-test ...` | -| Check if stale binary ran | Look for FLAT checkpoint files `epoch_E_mb_M.mpk`; current binary writes DIRECTORIES `epoch_E_mb_M/head.mpk` | - ---- - -## Workspace flag gotcha - -`--workspace` takes the path to the `.ddrs` DIRECTORY ITSELF, not its parent. Experiment configs in `config/experiments/` default to `config/experiments/.ddrs` (wrong). Always pass the root workspace explicitly: - -```bash -ddrs run --config config/experiments/leakance_hourly_on.yaml \ - --workspace /home/tbindas/projects/ddrs/.ddrs \ - --workflow train-and-test -``` - ---- - -## Provenance and maintenance - -Files read to write this skill (re-read to verify any fact): - -```bash -# Core invariants and CLI behavior -cat /home/tbindas/projects/ddrs/CLAUDE.md - -# Architecture and per-timestep dataflow -cat /home/tbindas/projects/ddrs/.claude/ARCHITECTURE.md - -# Stale-binary incident + 2×2 experiment -cat /home/tbindas/projects/ddrs/docs/2026-07-01-leakance-hourly-experiment-handoff.md - -# 2×2 final findings and GO-marginal verdict -cat /home/tbindas/projects/ddrs/docs/2026-07-01-leakance-hourly-findings.md - -# Low-zeta diagnosis (H1-H7 verdicts) -cat /home/tbindas/projects/ddrs/docs/2026-07-02-leakance-diagnosis-findings.md - -# Gradient probe (P1-P3 verdicts) — worktree -cat /home/tbindas/projects/ddrs/.claude/worktrees/zeta-sensitivity/docs/2026-07-03-zeta-gradient-probe-findings.md - -# Recoverability positive control failure — worktree -cat /home/tbindas/projects/ddrs/.claude/worktrees/zeta-sensitivity/docs/2026-07-04-synthetic-recoverability-findings.md - -# rskan version pin -grep rskan /home/tbindas/projects/ddrs/Cargo.toml -``` - -Re-verification commands: -```bash -# Confirm invariant 1 still holds on current HEAD -cargo run --release --example compare_ddr_sandbox - -# Confirm rskan pin -grep rskan /home/tbindas/projects/ddrs/Cargo.toml - -# Confirm leakance tests pass -cargo test --test leakance_gradcheck --test leakance_off_parity --test zeta_accum -``` diff --git a/.claude/skills/ddrs-config-and-flags/SKILL.md b/.claude/skills/ddrs-config-and-flags/SKILL.md deleted file mode 100644 index 49b0274..0000000 --- a/.claude/skills/ddrs-config-and-flags/SKILL.md +++ /dev/null @@ -1,403 +0,0 @@ ---- -name: ddrs-config-and-flags -description: "Use when you need to add, change, or audit any ddrs YAML configuration key; diagnose a config-load error; understand what a parameter controls; add a new routing parameter or training flag; or decide which experiment config to use as a starting point. Also use when modifying params.use_leakance, params.use_cuda_graphs, kan_head.disaggregation, or experiment.loss." ---- - -# ddrs Config and Flags Reference - -**Jargon primer (defined once):** -- **BURN** — Rust deep-learning framework (like PyTorch for Rust). Used instead of PyTorch here. -- **KAN head** — Kolmogorov-Arnold Network; maps per-reach catchment attributes to routing parameters. Replaces an MLP. -- **MC routing** — Muskingum-Cunge, a 1-D river routing solver. The `params:` block controls it. -- **CONUS** — Contiguous US; the default training domain (346,321 reaches). -- **Q'** — lateral inflow (m³/s) per reach; the forcing signal from dHBV2. -- **CSR** — Compressed Sparse Row; the sparse matrix format used for the triangular network solve. -- **CUDA graph** — GPU kernel sequence baked at compile time and replayed cheaply each timestep. -- **ddrs.yaml** — the live workspace config; `ddrs plan` generates it from a template. NEVER committed. -- **config/merit_training.yaml** — the canonical production template; committed and kept in sync with DDR-Python. - ---- - -## When NOT to use this skill - -| Situation | Use instead | -|---|---| -| Debugging NaN loss or gradient explosion | `ddrs-systematic-debugging` | -| Adding a new data source format (new zarr reader, new fabric) | `ddrs-data-sources` | -| Understanding the sparse backward / autograd tape | `.claude/references/ddrs-burn-autograd.md` | -| Per-timestep routing math | `.claude/ARCHITECTURE.md` | -| CLI lifecycle (`ddrs plan`, `ddrs run`, `ddrs gc`) | `CLAUDE.md` §"ddrs CLI" | - ---- - -## Overview: config file anatomy - -A ddrs config is a single YAML file with six top-level sections: - -``` -mode / workflow / geodataset / device / seed / np_seed ← top-level scalars -data_sources: ← where inputs live on disk -experiment: ← training-loop hyperparameters -kan_head: ← KAN head architecture + which parameters it predicts -params: ← routing engine settings -testing: ← overlay applied in eval mode (overrides experiment: keys) -``` - -The Rust struct is `src/config.rs::Config`. Deserialization uses -`Config::from_yaml_file_with_mode(path, ConfigMode::Training|Testing)`. - ---- - -## Top-level scalars - -| Key | Type | Default | Notes | -|---|---|---|---| -| `mode` | `"training"` \| `"testing"` | `"training"` | Must agree with `workflow:` (see guard below). | -| `workflow` | `train` \| `eval` \| `train-and-test` | absent (None) | `train-and-test` runs both phases and computes the baseline comparison. | -| `geodataset` | `"merit"` | `"merit"` | Only value supported as of 2026-07-05. | -| `device` | integer | `0` | CUDA device ordinal. On multi-GPU hosts, pick a non-display GPU. | -| `seed` | integer | `42` | Controls KAN weight initialization. | -| `np_seed` | integer | `42` | Controls per-epoch gauge shuffle order. | - -**Guard:** `mode: training` requires `workflow ∈ {train, train-and-test}`. `mode: testing` requires `workflow: eval`. A contradiction is rejected at load time with a message containing `"conflicting top-level keys"`. - ---- - -## `data_sources:` section - -All data is read in place — no export step. Every path is a `PathBuf`. - -| Key | Required | Notes | -|---|---|---| -| `attributes` | Yes | NetCDF catchment attributes. Columns must match `kan_head.input_var_names`. `attributes` can be a single path OR a list of paths — they are feature-concatenated on COMID (NaN-filled for missing COMIDs, inner join on COMID dimension). Example: `[merit_global_attributes_v2.nc, merit_channel_attributes_v1.nc]`. | -| `streamflow` | Yes | dHBV2 lateral inflow Q'. Icechunk (`.ic`) for CONUS; zarr-v2 for global. | -| `observations` | Yes | USGS (or global) daily observed discharge; training targets. | -| `gages` | Yes | CSV with STAID and COMID columns. | -| `geospatial_fabric` | Conditional | `.shp`/`.dbf`/`.gpkg`; triggers managed adjacency build into `.ddrs/adjacency//`. | -| `geospatial_fabric_layer` | Optional | Layer name inside a multi-layer `.gpkg`; invalid for `.shp`/`.dbf`. | -| `conus_adjacency` | Conditional | Pre-built zarr. Must be paired with `gages_adjacency`. | -| `gages_adjacency` | Conditional | Pre-built zarr. Must be paired with `conus_adjacency`. | -| `aorc_precip` | Optional | Hourly AORC precip zarr v3 (`merit_unit_catchments.zarr`). Required when `kan_head.disaggregation.use_precip: true`. | - -**Adjacency rule (enforced at load time):** provide EITHER both `conus_adjacency` + `gages_adjacency`, OR `geospatial_fabric` (managed build). Providing only one of the two adjacency zarrs is rejected. Providing none of the three is rejected. - -**Production path (CONUS workstation, as of 2026-07-05):** -```yaml -data_sources: - attributes: /home/tbindas/projects/ddr/data/merit_global_attributes_v2.nc - geospatial_fabric: /projects/mhpi/data/MERIT/raw/continent/riv_pfaf_7_MERIT_Hydro_v07_Basins_v01_bugfix1.shp - streamflow: /mnt/ssd1/data/icechunk/merit_dhbv2_UH_retrospective.ic - observations: /mnt/ssd1/data/icechunk/usgs_daily_observations - gages: /home/tbindas/projects/ddr/references/gage_info/gages_3000.csv -``` - ---- - -## `experiment:` section (training mode) - -| Key | Type | Default | Production value | Notes | -|---|---|---|---|---| -| `batch_size` | integer | none | `64` | **Gauges** per mini-batch during training. Meaning shifts in `testing:` — see below. | -| `start_time` | `"YYYY/MM/DD"` | none | `"1981/10/01"` | Training window start. | -| `end_time` | `"YYYY/MM/DD"` | none | `"1995/09/30"` | Training window end (water year 1995). | -| `epochs` | integer | none | `5` | Total training epochs. | -| `rho` | integer \| null | none | `90` | Sequence length in days per mini-batch. Set `null` in testing overlay. | -| `shuffle` | bool | `false` | `true` | Re-shuffle gauge order each epoch (seeded by `np_seed`). | -| `warmup` | integer | none | `5` | Days excluded from loss at sequence start (routing spin-up). | -| `learning_rate` | map epoch→f32 | `{}` | `{1: 0.001, 3: 0.0005}` | Step decay; applies from that epoch onward. | -| `grad_clip_max_norm` | float \| absent | absent | `1.0` | Global gradient-norm clip. Omit to disable. | -| `checkpoint` | path \| absent | absent | absent | Directory path to resume from (e.g. `.ddrs/runs//checkpoints/epoch_5_mb_9`). | -| `state_cache` | path \| absent | absent | absent | Path to a continuous-run state zarr store (from `probe_zeta_gradient --mode state-cache`). Injects window-start routing states (hotstarted from a continuous run rather than cold zero-flow start). Reduces the hotstart-transient noise floor. Optional; leave blank for standard cold-start training. | -| `loss` | block \| absent | L1 (see below) | absent | Training objective. Omit for historical L1. | - -### `experiment.loss:` sub-block - -Omit the entire `loss:` block to use the historical L1 objective. Including the block does NOT change behavior if `kind: l1`. - -| Key | Type | Default | Notes | -|---|---|---|---| -| `kind` | `l1` \| `nnse-kge` \| `kge` | `l1` | `l1` = mean absolute error. `nnse-kge` = composite NNSE + KGE. `kge` = component-weighted KGE (r, alpha, beta terms individually weighted). | -| `nnse_weight` | float | `1.0` | Weight on `1 - NNSE` term (all non-L1 kinds). | -| `kge_weight` | float | `1.0` | Weight on `1 - KGE` Euclidean term (`nnse-kge` only). | -| `r_weight` | float | `1.0` | Weight on `(r-1)²` correlation term (`kge` kind only). | -| `alpha_weight` | float | `1.0` | Weight on `(alpha-1)²` variance ratio (`kge` kind only). This is the restoring force against MC over-attenuation. | -| `beta_weight` | float | `1.0` | Weight on `(beta-1)²` mean ratio (`kge` kind only). | -| `kge_clamp` | float | `10.0` | Per-gauge upper bound on weighted KGE-component sum before averaging. Prevents near-constant gauges from hijacking the batch gradient. | -| `eps` | float | `0.1` | Stabilizes variance/mean denominators. Matches DDR `hydrograph_loss` default. | - -**Why L1 and NSE both fail KGE:** L1 and NSE are both maximized when simulated variance is below observed (NSE optimum is at `alpha = r < 1`). This rewards MC for over-attenuating flood peaks — the diagnosed cause of KGE regression vs the summed-Q' baseline in CONUS runs (median KGE 0.723→0.701 while NSE improved 0.639→0.684). The `(alpha-1)²` term in `nnse-kge` / `kge` supplies a restoring gradient. - ---- - -## `testing:` section (eval-mode overlay) - -These keys **replace** the matching `experiment:` keys when `mode: testing` is loaded. Absent keys inherit from `experiment:`. - -| Key | Default in testing | Notes | -|---|---|---| -| `start_time` | `"1995/10/01"` | Eval window start (water year 1996). | -| `end_time` | `"2010/09/30"` | Eval window end. | -| `batch_size` | `15` | **DAYS** per evaluation chunk — semantic shift from training's gauges-per-batch. | -| `rho` | `null` | Explicitly clears sequence sampling (null is distinct from absent). | - -**CAUTION:** `batch_size` changes meaning between modes. Training: gauges. Testing: days. This is not a typo — it's in the YAML comments. - ---- - -## `kan_head:` section - -The KAN head architecture: `Linear(F,H) → KanLayer(H,H) × num_hidden_layers → Linear(H,P) → Sigmoid`. - -| Key | Type | Code default | Production value | Notes | -|---|---|---|---|---| -| `hidden_size` | integer | none | `21` | Hidden dimension H. | -| `num_hidden_layers` | integer | none | `2` | Inner KanLayer repetitions. ALL receive the SAME seed (DDR `kan.py` quirk, preserved for parity). | -| `grid` | integer | `5` | `50` | B-spline grid intervals per KAN edge (`num` in pykan). Production overrides the code default. | -| `k` | integer | `3` | `2` | B-spline order. DDR overrides pykan's default of 3 to 2 in production; keep 2 for parity. | -| `input_var_names` | list of strings | none | 10 attributes (see below) | Column names in `attributes` NetCDF. | -| `learnable_parameters` | list of strings | none | `[n, q_spatial, p_spatial]` | Parameters the KAN head emits. Must have matching entries in `params.parameter_ranges`. | -| `disaggregation` | block \| absent | absent | absent | Enables the daily→hourly disaggregation head (see sub-block below). Absent = flat repeat-24. | - -**Production `input_var_names` (10 attributes):** -``` -SoilGrids1km_clay, aridity, meanelevation, meanP, NDVI, -meanslope, log10_uparea, SoilGrids1km_sand, ETPOT_Hargr, Porosity -``` - -### `kan_head.disaggregation:` sub-block - -Presence of this block enables the learnable daily→hourly disaggregation head (`src/nn/disagg_head.rs`). Absence = flat `repeat-24` (backward-compatible default). - -| Key | Type | Default | Notes | -|---|---|---|---| -| `hidden_size` | integer | `16` | Hidden dimension of the disagg MLP. | -| `use_attributes` | bool | `true` | Condition on catchment attributes. | -| `use_precip` | bool | `false` | Condition on 72-h AORC precip window `[d-1, d, d+1]`. Requires `data_sources.aorc_precip`. | -| `use_temp` | bool | `false` | Condition on 72-h AORC temperature window. Also requires `data_sources.aorc_precip`. | - -**If `use_precip: true` and `data_sources.aorc_precip` is absent:** `MeritGagesDataset::open` errors at runtime (not at config load time). The missing precip source cannot silently degrade. - ---- - -## `params:` section (routing engine) - -| Key | Type | Code default | Production value | Notes | -|---|---|---|---|---| -| `sparse_solver` | `cpu` \| `cuda` | `cpu` | `cuda` | `cuda` uses cuSPARSE for the triangular solve. Falls back to `cpu` on non-CUDA backends with a WARN log. | -| `use_cuda_graphs` | bool | `false` | `true` | Capture the routing forward as a CUDA graph; faster replay each timestep. **See guards below.** | -| `use_leakance` | bool | `false` | `false` | Enable the GW–SW water-loss term. **See guards below.** Experimental as of 2026-07-05. | -| `leakance_losing_only` | bool | `true` | `true` | When true, applies `max(0, depth − d_gw)` clamp so zeta is zero for gaining reaches. Config-gated; added in Phase C. No-op when `use_leakance: false`. | -| `leakance_impervious_threshold` | float | `0.7` | `0.7` | Hard-zero mask: reaches with `corridor_impervious` attribute ≥ this threshold get zeta = 0 regardless of other params. No-op when `corridor_impervious` is absent from attributes. | -| `tau` | integer | `3` | `3` (not set in YAML) | Muskingum routing sub-step count. Rarely changed. | -| `log_space_parameters` | list of strings | `["p_spatial"]` | `["p_spatial"]` | Parameters whose range spans decades; KAN output is exp-scaled before routing. | -| `defaults` | map str→f32 | `{p_spatial: 21.0}` | `{p_spatial: 21.0}` | Fixed values for parameters NOT in `learnable_parameters`. | - -### `params.parameter_ranges:` sub-block - -Physical `[min, max]` each sigmoid-normalized KAN output maps onto. All defaults are defined in `src/config.rs::ParameterRanges::default()`. - -| Key | Default range | Log-space | Notes | -|---|---|---|---| -| `n` | `[0.015, 0.25]` | No | Manning's roughness coefficient. | -| `q_spatial` | `[0.0, 1.0]` | No | Leopold & Maddock width–depth exponent (`top_width = p·depth^q`). | -| `p_spatial` | `[1.0, 200.0]` | Yes | Leopold & Maddock width coefficient. In log-space by default. | -| `x_storage` | `[0.0, 0.5]` | No | Muskingum storage weight X. Only consumed when listed in `learnable_parameters`; otherwise routing uses constant 0.3. | -| `K_D` | `[1e-8, 1e-6]` | Yes (add to `log_space_parameters`) | Hydraulic exchange rate (1/s). Leakance only. Note: uppercase in YAML. | -| `d_gw` | `[-2.0, 2.0]` | No | Groundwater depth offset (m). Leakance only. | -| `leakance_factor` | `[0.0, 1.0]` | No | Dimensionless leakance scale. Leakance only. | - -### `params.attribute_minimums:` sub-block - -Physical floor applied during routing for numerical stability. - -| Key | Default | Units | -|---|---|---| -| `discharge` | `1.0e-4` | m³/s | -| `slope` | `1.0e-3` | m/m | -| `velocity` | `0.01` | m/s | -| `depth` | `0.01` | m | -| `bottom_width` | `0.01` | m | - ---- - -## Guards enforced at `Config::from_yaml_file` (load-time errors) - -| Guard | Trigger | Error message substring | -|---|---|---| -| Mode/workflow conflict | `mode: training` + `workflow: eval`, or `mode: testing` + `workflow: train` | `"conflicting top-level keys"` | -| Partial adjacency | Only one of `conus_adjacency` / `gages_adjacency` set | `"gages_adjacency\` is missing"` or `"conus_adjacency\` is missing"` | -| No adjacency sources | Neither adjacency zarrs nor `geospatial_fabric` | `"adjacency sources are missing"` | -| Fabric layer on non-gpkg | `geospatial_fabric_layer` set with `.shp`/`.dbf` fabric | `"geospatial_fabric_layer"` and `".gpkg"` | -| Leakance + CUDA graphs | `use_leakance: true` and `use_cuda_graphs: true` | `"use_leakance"` and `"use_cuda_graphs"` | - ---- - -## Production configs vs experimental configs - -| Config file | Status | Key differences from production | -|---|---|---| -| `config/merit_training.yaml` | Production template | `grid:50`, `k:2`, `use_cuda_graphs:true`, managed adjacency via `geospatial_fabric`, no leakance, no disaggregation, L1 loss | -| `config/experiments/leakance_hourly_on.yaml` | Experimental (2026-07-01, GO-marginal) | `use_leakance:true`, `use_cuda_graphs:false`, `K_D`/`d_gw`/`leakance_factor` in head, precip disaggregation enabled, explicit zarr adjacency paths | -| `config/experiments/leakance_daily_on.yaml` | Experimental (2026-07-01) | Same as `leakance_hourly_on.yaml` but NO disaggregation block, NO `aorc_precip` | -| `config/sources/conus.yaml` | Source group | CONUS workstation paths without AORC precip | -| `config/sources/conus-hourly.yaml` | Source group | CONUS + `aorc_precip: /mnt/ssd1/data/aorc/merit_unit_catchments.zarr` | -| `config/sources/global.yaml` | Source group | GPFS global paths | - -**Source groups** are text-spliced into `data_sources:` by `ddrs sources use `. They do not set `kan_head` or `params`. - ---- - -## Critical runtime traps - -### STALE-BINARY TRAP -`cargo build` does NOT update `~/.cargo/bin/ddrs`. After ANY change to `src/`, run: -```bash -cargo install --path . -# or faster if target/release is current: -cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs -``` -Self-check: current checkpoints are **directories** (`.ddrs/runs//checkpoints/epoch_E_mb_M/head.mpk`). Flat `.mpk` files mean a stale binary ran. - -### CUDA graphs mask NaN -`use_cuda_graphs: true` captures a finite forward pass graph. If a NaN appears in a subsequent forward (different inputs), the graph replays stale finite values — you get a finite loss with no error. To validate forwards, test with `use_cuda_graphs: false`. This is why `use_leakance: true` + `use_cuda_graphs: true` is rejected at config load time. - -### Leakance status (as of 2026-07-05) -The leakance 2×2 (forcing × leakance) returned a **GO-marginal** verdict: -- `|zeta| > 0.01 m³/s` on 10.4% of 64,892 eval reaches (meets ≥10% bar) -- Leakance helps under hourly forcing on the losing-stream subset (ΔNSE +0.0005, ΔKGE +0.0018, 55.5% of gauges improve) -- Leakance hurts under daily forcing (ΔNSE −0.0017, ΔKGE −0.0009, 35.6%) -- `K_D` pinned at ceiling `1e-6` (binding constraint; widening NOT recommended as of 2026-07-05) - -Diagnosis hypotheses (2026-07-02): -- H2 (head throttling): SUPPORTED -- H4 (gauge bias): SUPPORTED -- H5 (equifinality with `n`): SUPPORTED -- H1 (K_D box too narrow): REFUTED -- H3 (KAN capacity): REFUTED -- H6, H7: REFUTED - -**Leakance identifiability is NOT proven.** The positive-control synthetic recovery experiment (2026-07-04) FAILED: recovery ratio 0.009 vs ≥0.5 bar. Root cause: windowed training objective has ~130x hotstart-transient noise floor. Phase B (state-cache hotstart, ≤0.25 mean L1 noise floor target) is required before any identifiability claim. - ---- - -## How to add a new routing parameter (checklist) - -A "routing parameter" is a per-reach scalar the KAN head predicts and the MC solver consumes. Example: adding a new parameter `my_param`. - -- [ ] **1. Add to `ParameterRanges` struct** (`src/config.rs`): - ```rust - pub my_param: [f32; 2], - ``` - Add a default in `ParameterRanges::default()`. - -- [ ] **2. Add YAML key parsing** in `From for Params` (`src/config.rs`): - ```rust - if let Some(v) = r.parameter_ranges.get("my_param") { - p.parameter_ranges.my_param = *v; - } - ``` - -- [ ] **3. Add to `config/merit_training.yaml`** under `params.parameter_ranges:` (if it has a production-relevant range). - -- [ ] **4. Wire into routing** (`src/routing/mmc.rs` or a new module): consume `Params.parameter_ranges.my_param` in `setup_inputs` or `route_timestep`. Follow the `denormalize` pattern in `src/routing/utils.rs`. - -- [ ] **5. Add to `kan_head.learnable_parameters:`** in any experiment config that uses it. - -- [ ] **6. Add to `params.log_space_parameters:`** if the range spans decades. - -- [ ] **7. If log-space:** add to `log_space_parameters` list in `params.log_space_parameters` in YAML and ensure `denormalize` handles it. - -- [ ] **8. Write a gradient-exactness test** if the new parameter enters a custom backward op. Run: - ```bash - cargo test --test - cargo run --release --example compare_ddr_sandbox - ``` - The sandbox must still report `ABSOLUTE MATCH`. - ---- - -## How to add a new training-mode boolean flag (checklist) - -Example: adding `use_my_feature: bool` under `params:`. - -- [ ] **1. Add field to `Params` struct** (`src/config.rs`): - ```rust - pub use_my_feature: bool, - ``` - Add `use_my_feature: false` to `Params::default()`. - -- [ ] **2. Add to `ParamsRaw`** and parse in `From for Params`: - ```rust - // in ParamsRaw: - use_my_feature: Option, - // in From impl: - if let Some(b) = r.use_my_feature { p.use_my_feature = b; } - ``` - -- [ ] **3. Add validation if needed** (`validate_*` functions in `src/config.rs`). Call from `from_yaml_file_with_mode`. Validation errors must include the YAML key name and the reason. - -- [ ] **4. Add a test** in the `#[cfg(test)]` block at the bottom of `src/config.rs` covering: flag defaults to false, flag parses true, any guard is rejected. - -- [ ] **5. Thread through call sites**: training bootstrap (`src/training/bootstrap.rs`), eval (`src/cli/eval.rs`), and any other entrypoints that construct `MuskingumCunge` or read `Params`. - -- [ ] **6. Document in `config/merit_training.yaml`** as a commented-out key with explanation if it has production relevance. - ---- - -## Quick reference: minimal leakance-on config diff - -Starting from `config/merit_training.yaml`, three changes activate leakance: - -```yaml -# 1. Under params: -params: - use_leakance: true - use_cuda_graphs: false # REQUIRED — leakance + cuda_graphs is rejected at load time - parameter_ranges: - K_D: [1.0e-8, 1.0e-6] - d_gw: [-2.0, 2.0] - leakance_factor: [0.0, 1.0] - log_space_parameters: - - p_spatial - - K_D # ADD — K_D range spans decades - -# 2. Under kan_head.learnable_parameters: - learnable_parameters: - - n - - q_spatial - - p_spatial - - K_D - - d_gw - - leakance_factor -``` - -See `config/experiments/leakance_hourly_on.yaml` for the full working example. - ---- - -## Provenance and maintenance - -Ground truth files read to produce this skill (verify before re-editing): -```bash -# Config struct (all fields, defaults, guards): -grep -n "pub use_" /home/tbindas/projects/ddrs/src/config.rs - -# Production defaults verified from: -grep -n "fn default" /home/tbindas/projects/ddrs/src/config.rs - -# Production YAML (single source of truth for hyperparameter values): -cat /home/tbindas/projects/ddrs/config/merit_training.yaml - -# Leakance experiment config: -cat /home/tbindas/projects/ddrs/config/experiments/leakance_hourly_on.yaml - -# Load-time guard tests (exhaustive): -grep -n "#\[test\]" /home/tbindas/projects/ddrs/src/config.rs | head -40 -``` - -Config struct location: `/home/tbindas/projects/ddrs/src/config.rs` -Production template: `/home/tbindas/projects/ddrs/config/merit_training.yaml` -Experiment configs: `/home/tbindas/projects/ddrs/config/experiments/` -Source group configs: `/home/tbindas/projects/ddrs/config/sources/` - -Volatile facts date-stamped: 2026-07-05. Re-verify leakance status, CONUS metric baselines, and K_D ceiling diagnosis before citing in new experiments. diff --git a/.claude/skills/ddrs-debugging-playbook/SKILL.md b/.claude/skills/ddrs-debugging-playbook/SKILL.md deleted file mode 100644 index 46b1e7e..0000000 --- a/.claude/skills/ddrs-debugging-playbook/SKILL.md +++ /dev/null @@ -1,595 +0,0 @@ ---- -name: ddrs-debugging-playbook -description: "Use when a ddrs run produces wrong results, silent failures, metric regressions, NaN loss, stale checkpoints, V1 mismatch, leakance anomalies, KAN head divergence, adjacency errors, data-source alignment issues, or any symptom that costs debug time. Also use before attributing a result to a code bug — many apparent bugs are operator error (stale binary, wrong fixture, config contradiction)." ---- - -# ddrs debugging playbook - -**Audience:** Sonnet-class AI or mid-level ML engineer who knows PyTorch but not Rust/BURN. -**Voice:** imperative runbook. Copy-paste every command. Verify before claiming. - ---- - -## Glossary (terms used throughout) - -| Term | Meaning | -|---|---| -| **ddrs** | BURN-based Rust port of DDR (Python/PyTorch Muskingum-Cunge routing solver) | -| **DDR** | Python reference: `~/projects/ddr/`. The gold standard for numerical parity | -| **BURN** | Rust deep-learning framework (version 0.21 in this project) | -| **MC solver** | Muskingum-Cunge routing: converts upstream + lateral inflow to routed discharge per reach per timestep | -| **V1 / ABSOLUTE MATCH** | Regression gate: `compare_ddr_sandbox` max abs diff < 1e-3 m³/s vs DDR | -| **KAN head** | The neural network head (`rskan::KanLayer` v0.1.3): maps catchment attributes → routing parameters | -| **f32 invariant** | All tensors in the routing core must stay float32; f64/bf16 casts break DDR parity | -| **lower-triangular adjacency** | The CSR sparse pattern has `rows[k] >= cols[k]`; the forward-sub solver requires this | -| **sparse backward** | Hand-written O(nnz) `CsrSolveOp: Backward` in `src/sparse.rs`; must not be replaced by autograd unrolling | -| **leakance** | Experimental GW–SW water-loss term (`src/routing/leakance.rs`); off by default | -| **zeta** | The per-reach per-timestep leakance flux (m³/s): `zeta = leakance_factor · area_z · K_D · (depth − d_gw)` | -| **summed-Q baseline** | No-routing reference: per-gauge sum of upstream divide Qr. CONUS: median NSE 0.689 / KGE 0.723 (as of 2026-07-05) | -| **CUDA Graphs** | CUDA kernel-replay optimization (`use_cuda_graphs: true`); incompatible with leakance; masks NaN loss | -| **Q'** | Lateral inflow (m³/s) from an upstream forcing model (DHBV, LSTM, etc.) | -| **worktree** | Git worktree at a branch tip — used for experimental campaigns without touching main tree | - ---- - -## When NOT to use this skill - -- You want to understand the routing math or architecture → read `.claude/ARCHITECTURE.md` and `.claude/references/ddrs-algorithm.md` -- You want to port or verify a new feature against DDR → use skill `ddrs-comparing-to-ddr` (`.claude/references/ddrs-comparing-to-ddr.md`) -- You want to set up a new experiment from scratch → read `CLAUDE.md` §"ddrs CLI" -- You are doing leakance identifiability research → see `docs/2026-07-02-leakance-diagnosis-findings.md` for the completed hypothesis battery - ---- - -## Part 1 — Symptom → triage table - -Scan this table first. Each row points to a Part 2 entry with the full story and fix. - -| Symptom | Most likely trap | Go to | -|---|---|---| -| Two runs that differ in config produce byte-identical predictions | Stale installed binary | [T1](#t1-stale-binary-trap) | -| Checkpoint files are flat `.mpk` (not a directory) | Stale binary | [T1](#t1-stale-binary-trap) | -| `manifest.json` shows current git SHA but behavior looks old | Stale binary (SHA stamps from `.git` at runtime, not the binary) | [T1](#t1-stale-binary-trap) | -| `ddrs run` silently ignores `disaggregation:` block | Stale binary (pre-disagg binary ignores unknown serde fields) | [T1](#t1-stale-binary-trap) | -| `compare_ddr_sandbox` reports diff > 1e-3 m³/s | V1 regression | [T2](#t2-v1-regression) | -| `compare_ddr_sandbox` fails after regenerating fixtures | Wrong DDR reference tree | [T2](#t2-v1-regression) | -| Loss goes NaN, but only with `use_cuda_graphs: true` | CUDA Graphs mask NaN | [T3](#t3-cuda-graphs-mask-nan) | -| Loss is finite but suspiciously constant across steps | CUDA Graphs returning stale capture | [T3](#t3-cuda-graphs-mask-nan) | -| Config parse error: "`use_leakance: true` requires `use_cuda_graphs: false`" | Config contradiction (intentional rejection) | [T4](#t4-leakance-config-contradictions) | -| Leakance run still uses CUDA graphs silently | Missing `use_cuda_graphs: false` in config | [T4](#t4-leakance-config-contradictions) | -| K_D pinned at ceiling (100% of reaches) | K_D box is binding — or head throttling (H2); see diagnosis | [T5](#t5-leakance-parameter-collapse-or-ceiling) | -| K_D collapsed to floor (sub-1e-8) | Replicates DDR's original revert failure; check forcing resolution | [T5](#t5-leakance-parameter-collapse-or-ceiling) | -| Gradient check fails on leakance op | Regression in `src/routing/leakance.rs` backward | [T6](#t6-leakance-gradient-correctness) | -| `zeta_accum` test fails | Accumulated zeta not matching headwater identity | [T6](#t6-leakance-gradient-correctness) | -| KAN head shape or init diverges from DDR | rskan version bump or inter-block ReLU accidentally re-added | [T7](#t7-kan-head-divergence) | -| KAN parity tests fail after `Cargo.toml` rskan bump | Fixture needs regeneration | [T7](#t7-kan-head-divergence) | -| Adjacency test fails or topological ordering wrong | `rows[k] < cols[k]` somewhere; lower-triangular invariant violated | [T8](#t8-adjacency-invariant) | -| `ddrs plan` hangs or errors on adjacency build | Bad fabric path or multi-layer gpkg needs `geospatial_fabric_layer` | [T8](#t8-adjacency-invariant) | -| Training NSE well below summed-Q baseline | Routing not earning its keep; loss or gradient issue | [T9](#t9-metric-regression-below-baseline) | -| KGE lower than baseline in every config | Expected — L1 loss penalizes variance; this is known behavior | [T9](#t9-metric-regression-below-baseline) | -| Baseline median NSE absurdly LOW; many gauges' baseline predictions exactly 0.0 | Pre-2026-07-29 binary: single-divide gauges summed an empty upstream set instead of being skipped | [T9](#t9-metric-regression-below-baseline) | -| Hourly run produces same predictions as daily run | Stale binary (pre-disagg) or `aorc_precip` source missing | [T1](#t1-stale-binary-trap), [T10](#t10-disaggregation-no-op) | -| `MeritGagesDataset::open` errors with `use_precip: true` | `aorc_precip` source not configured | [T10](#t10-disaggregation-no-op) | -| Checkpoint resume trains zero batches | `experiment.epochs` not raised above checkpoint epoch | [T11](#t11-checkpoint-resume-issues) | -| Resumed run drifts from uninterrupted run | Expected: weights stored as f16 (CompactRecorder) | [T11](#t11-checkpoint-resume-issues) | -| `ddrs run --strict` exits with code 4 | Source fingerprint drift vs `.ddrs/sources.lock` | [T12](#t12-source-lock-drift) | -| Recoverability / identifiability experiment fails | Hotstart transient noise floor issue (Phase B not yet met) | [T13](#t13-leakance-identifiability-status) | -| `probe_zeta_gradient --mode teacher` killed with no error in its own log | 365-day chunk peaks ~65 GB RSS on the 64,892-reach eval network; kernel OOM kill | [T14](#t14-teacher-mode-oom) | -| Workflow fails `icechunk read failed ... object not found` deep into eval; store probes clean | Transient icechunk read failure, not a data hole | [T15](#t15-transient-icechunk-read-failure) | - ---- - -## Part 2 — Trap stories and fixes - -### T1: Stale binary trap - -**Story (2026-07-01).** The leakance × hourly 2×2 experiment produced two runs that were byte-identical despite different configs (one with hourly disaggregation, one without). The `manifest.json` showed the current git SHA `2cdd341` — which made it look like a code bug. Root cause: `~/.cargo/bin/ddrs` had mtime 2026-06-03, predating both the disaggregation feature (June 19) and leakance (June 29). The installed binary silently ignored the `disaggregation:` config block (serde ignores unknown fields) and wrote flat `.mpk` checkpoints instead of the current directory format. - -**Discriminating test.** Check checkpoint format: -```bash -# Current binaries write DIRECTORIES: -ls .ddrs/runs//checkpoints/ -# → epoch_5_mb_9/ (directory = current binary) -# → epoch_5_mb_35.mpk (flat file = stale binary) -``` - -Check binary age: -```bash -stat ~/.cargo/bin/ddrs | grep Modify -# Should be >= your last src/ change date -``` - -**Fix (canonical):** -```bash -cargo install --path . -# OR faster if target/release/ is already built: -cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs -# OR bypass the installed binary entirely: -cargo run --release --bin ddrs -- run --workflow train-and-test -``` - -**Rule:** `cargo build` does NOT update `~/.cargo/bin/ddrs`. Re-install after every `src/` change before invoking `ddrs` by name. - -**Head size cross-check (CONUS, as of 2026-07-05):** -- No disagg, no leakance: ~103,459 B -- Disagg only: ~107,178 B -- Disagg + leakance (3 extra output cols): ~107,320 B - ---- - -### T2: V1 regression - -**What V1 is.** `examples/compare_ddr_sandbox` replays DDR's 5-reach RAPID2 sandbox through ddrs's MC solver. The threshold is `max abs diff < 1e-3 m³/s`. A passing run prints: -``` -verdict: ABSOLUTE MATCH (max abs < 1e-3 m³/s) -``` -Typical passing value is ~1.5e-5 m³/s — two orders of magnitude under the threshold. - -**Run it:** -```bash -mkdir -p output # required — the example does not mkdir -p -cargo run --release --example compare_ddr_sandbox -# Also test the CUDA + graph-capture path: -DDRS_FORCE_GRAPHS=1 cargo run --release --example compare_ddr_sandbox -``` - -**Triage checklist when V1 fails:** - -1. Inspect `output/ddrs_vs_ddr.csv` — which reaches are worst? Single bad reach = geometry/parameter bug. Global failure = solver or kernel issue. - -2. Check if fixtures are stale: - ```bash - cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/export_ddr_sandbox.py - cd ~/projects/ddrs && git diff fixtures/sandbox/ - ``` - **WARNING:** Only the local `~/projects/ddr` checkout is valid. The unpushed `geometry/trapezoidal.py` work is not in any public DDR commit. Regenerating from a clean DDR clone produces ~0.55 m³/s divergence at every ddrs commit — that is a wrong-reference artifact, NOT a port bug (as of 2026-06-06). - -3. Audit recent changes to the only paths that affect V1: - ```bash - git log -p -- src/routing/ src/geometry.rs src/sparse/ - ``` - -4. Look for precision leaks: - ```bash - grep -rn "f64\|bf16\|cast\|to_dtype" src/routing/ src/geometry.rs src/sparse/ - ``` - Any cast away from f32 in these paths breaks DDR parity. - -5. Cross-check with gradcheck: - ```bash - cargo test --test sparse_gradcheck - ``` - If gradcheck fails too, the algorithm changed. If only V1 fails, it's a kernel-ordering or arithmetic-fusion difference. - -**The threshold is non-negotiable.** Never relax `1e-3 m³/s`. Never declare "good enough." - ---- - -### T3: CUDA Graphs mask NaN - -**What happens.** When a forward pass produces NaN and `use_cuda_graphs: true`, the CUDA graph replays the stale pre-NaN capture instead of the live computation. The loss appears finite. You get silently wrong results with no error. - -**Confirmed behavior (as of 2026-07-05):** `use_cuda_graphs: true` returns stale finite loss on a NaN forward. - -**Discriminating test:** -```bash -# Reproduce the stale-loss symptom: -# Run once with cuda_graphs on vs off, inject a NaN input, compare loss values. -# If cuda_graphs=true gives finite loss and cuda_graphs=false gives NaN → confirmed. -``` - -**Fix:** Disable CUDA graphs when debugging any NaN or suspiciously-constant loss: -```yaml -# in ddrs.yaml or experiment config: -params: - use_cuda_graphs: false -``` - -**Rule:** Always debug loss anomalies with `use_cuda_graphs: false`. Re-enable only after confirming the forward is NaN-free. - ---- - -### T4: Leakance config contradictions - -**What happens.** Two config errors involving leakance are caught at load time. - -**Error 1 — leakance + CUDA graphs:** -``` -params: `use_leakance: true` requires `use_cuda_graphs: false` -``` -CUDA Graphs cannot capture the extra leakance kernel without a separate capture path. This is an intentional hard rejection in `src/config.rs:626`. - -**Fix:** -```yaml -params: - use_leakance: true - use_cuda_graphs: false # REQUIRED when use_leakance is true -``` - -**Error 2 — leakance parameters missing from KAN head.** -If `params.use_leakance: true` but `K_D`, `d_gw`, `leakance_factor` are not in `kan_head.learnable_parameters`, the head emits no leakance parameters and the routing silently has no exchange. No error is thrown — the leakance term gets a zero or garbage input. - -**Complete leakance config checklist:** -```yaml -params: - use_leakance: true - use_cuda_graphs: false - parameter_ranges: - K_D: [1.0e-8, 1.0e-6] # log-space; hydraulic exchange rate 1/s - d_gw: [-2.0, 2.0] # groundwater depth offset, m - leakance_factor: [0.0, 1.0] # dimensionless scale - -kan_head: - learnable_parameters: - - K_D - - d_gw - - leakance_factor - # ... plus your routing params (n, q_spatial, etc.) -``` - ---- - -### T5: Leakance parameter collapse or ceiling - -**Two failure modes:** - -**Mode A — K_D at ceiling (100% of reaches).** -Observed in both leakance-ON arms of the 2026-07-01 2×2 (hourly and daily). -- `K_D` median log10 = −5.999, IQR = 3.6e-4 (essentially a delta function at the `1e-6` upper bound). -- This is NOT the K_D box clipping the flux. Diagnosis (2026-07-02) showed median in-box utilization is only 3.4% — the optimizer maxes the rate constant and then throttles the product via the driving head (`d_gw` learned near typical depths, so `depth − d_gw ≈ 0`). -- **K_D widening is NOT recommended** — the Phase-3 gate failed because H1 (structural ceiling) was REFUTED. Root cause: H2 (head throttling) + H4 (gauge bias) + H5 (equifinality under daily forcing). - -**Mode B — K_D at floor (collapse to sub-1e-8).** -This replicates DDR's original revert failure (sub-0.01 m³/s exchange, physically negligible). If you see this, check: -- Is daily forcing being used? Under flat-daily forcing the depth dynamic range is too small for `zeta ∝ (depth − d_gw)` to be identifiable. -- Is hourly disaggregation actually running? Verify the binary is current (T1) and `aorc_precip` source is configured (T10). - -**Discriminating check after any leakance run:** -```bash -cargo build --release --bin dump_parameters -target/release/dump_parameters \ - --config \ - --checkpoint .ddrs/runs//checkpoints/epoch_E_mb_M/head \ - --output /tmp/kp.nc 2>&1 | grep -E "K_D|leakance_factor|d_gw|frac@" -``` -Expected for a non-collapsed run: `K_D` interior or at ceiling (not floor), `leakance_factor` interior (0.1–0.5), `d_gw` spatially varying. - ---- - -### T6: Leakance gradient correctness - -**Guard tests — run all four after any change to `src/routing/leakance.rs` or `mmc_op.rs`:** - -```bash -cargo test --test leakance_gradcheck # analytical ≈ finite-difference (8 params) -cargo test --test leakance_off_parity # byte-identical to no-leakance when off (3 tests) -cargo test --test zeta_accum # accumulated zeta == headwater identity -cargo run --release --example compare_ddr_sandbox # V1 must still pass -``` - -**If `leakance_gradcheck` fails:** The analytical backward in `TimestepLeakanceOp: Backward` is wrong. Compare against `src/routing/leakance.rs` math: `zeta = leakance_factor · area_z · K_D · (depth − d_gw)` where `area_z = (p · depth)^q_eps · length`. All partial derivatives are straightforward products/chains; check each of the 8 inputs. - -**If `zeta_accum` fails:** The accumulator in `evaluate` is not recomputing from the same primitives the backward used. The test verifies the headwater identity `q_no_leak[0] − q_leak[0] == zeta[0]`. - -**If `leakance_off_parity` fails:** The leakance gating is broken — the `None` path is not byte-identical to a run without leakance compiled in. - ---- - -### T7: KAN head divergence - -**Architecture (must not change without explicit intent):** -``` -Linear(F, H) → KanLayer(H, H) × num_hidden_layers → Linear(H, P) → Sigmoid -``` -- No inter-block ReLU. DDR's `kan.py` has none; adding one breaks parity. -- All `num_hidden_layers` inner KanLayers get the SAME initialization seed (DDR `kan.py:24-34` quirk — preserved for parity). -- rskan version: `v0.1.3` (as of 2026-07-05). Pinned in `Cargo.toml:27`. - -**Parity test suite:** -```bash -cargo test --features fixtures \ - --test kan_head_init_repro \ - --test kan_head_init_parity \ - --test kan_head_fixture_forward \ - --test kan_head_fixture_backward -``` - -**If tests fail after an rskan bump:** The fixtures need regeneration: -```bash -cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/dump_kan_weights.py -cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/dump_kan_forward.py -``` -Then re-run the test suite. If it still fails, the rskan API changed in a parity-breaking way — audit the diff and decide whether to update the DDR reference or roll back the bump. - -**If tests fail without an rskan bump:** Check for accidental re-introduction of inter-block ReLU in `src/nn/kan_head.rs`. - ---- - -### T8: Adjacency invariant - -**The invariant:** The CSR adjacency pattern must be lower-triangular: every nonzero at `(row, col)` must have `row >= col`. The forward-substitution solver reads rows in order and assumes all upstream contributions are already resolved. - -**Test:** -```bash -cargo test data_zarr_store::conus_adjacency_loads_real_merit_zarr -# Also: -cargo test --test adjacency_parity -``` - -**If adjacency build fails during `ddrs plan`:** -- Check the fabric path exists and is readable. -- For `.gpkg` files with multiple layers, set `geospatial_fabric_layer:` in config. -- The builder reads only the attribute table (`.dbf` or gpkg attributes) — never the geometry. Check `geospatial_fabric` points to the right file. -- On a fresh worktree, `.ddrs/adjacency/` does not exist — `ddrs plan` builds it on first run (~10 s for CONUS `.dbf`). - -**If topological ordering is wrong:** The managed builder replicates petgraph's deterministic DFS finish-time order. Check `src/adjacency/build.rs::topological_sort` against the engine's version. - ---- - -### T9: Metric regression below baseline - -**Baseline numbers (CONUS, as of 2026-07-05):** -| config | median NSE | median KGE | gauges | -|---|---|---|---| -| Summed-Q′ baseline, same-run (2,365-gauge eval set) | 0.678 | 0.717 | 2365 | -| Best result: precip-disagg + L1 | 0.715 | 0.711 | 2365 | - -**Critical known behavior:** KGE does NOT beat the summed-Q baseline in any trained config as of 2026-07-05. NSE beats it (+0.037 with precip disagg). The KGE regression is structural: the L1 loss maximizes at simulated variance below observed (α < 1), rewarding over-attenuation. The whole KGE drop is in the `α = σ_sim/σ_obs` term. - -**Triage if NSE is far below baseline:** -1. Is the binary current? (T1) -2. Is the loss descending? Check `run.log` for epoch-mean L1. -3. Is CUDA graphs masking NaN? (T3 — disable and re-check) -4. Are gauge batch sizes reasonable? Too few gauges per batch → noisy gradient. -5. Is the data source correct? `streamflow resolution: Daily|Hourly` is logged at dataset open — verify it. - -**The baseline itself can be the bug — phantom-zero single-divide gauges (fixed 2026-07-29).** -Gauges whose catchment is a single MERIT divide have ZERO edges in the gages -adjacency store (empty `indices_0`/`indices_1`, length-1 `order`). Before the fix, -the baseline included them but `GageSubgraph::upstream_comids` derives the upstream -set from edge endpoints only, so they summed an empty COMID set → the baseline -predicted exactly 0.0 for the whole window and scored it against real observations. -On the `daily_dhbv2_distributed_aorc2f` 3,211-gauge workspace this was 513 gauges -(16%) at median NSE −0.305, dragging the baseline median from 0.315 to 0.142. The -fix SKIPS zero-edge gauges in the baseline (`GageSubgraph::is_headwater`), matching -training's dataset filter ("dropped N headwater") so baseline and trained eval score -the same gauge population; the baseline cache key now includes `baseline-algo-v2` so -stale pre-fix caches recompute. Diagnostic: load -`/baselines//predictions.f32` and count rows that are all-zero — -any nonzero count on a post-fix binary is a new bug. Training was never affected: -`dataset.rs` drops zero-edge gauges as headwaters before batching. - -**To improve KGE above baseline:** Switch to `experiment.loss.kind: nnse-kge`. The `(α-1)²` term in KGE provides the restoring gradient. This requires explicit config: -```yaml -experiment: - loss: - kind: nnse-kge - nnse_weight: 1.0 - kge_weight: 1.0 -``` - ---- - -### T10: Disaggregation no-op - -**Symptoms:** -- Hourly and daily runs produce byte-identical predictions. -- Head file size is ~103,459 B (no-disagg size) even when config has `disaggregation: ...`. - -**Root causes (in priority order):** - -1. **Stale binary** — most likely. See T1. The pre-disagg binary silently ignores the `disaggregation:` block. - -2. **Missing `aorc_precip` source.** The AORC precip zarr at `/mnt/ssd1/data/aorc/merit_unit_catchments.zarr` must be in `data_sources:`. Without it, `MeritGagesDataset::open` errors when `use_precip: true`. Check: - ```bash - ddrs sources list # '*' marks active group - # conus-hourly group includes aorc_precip - ddrs sources use conus-hourly - ``` - -3. **`use_precip: false` in config.** The `aorc_precip` source must be present AND `kan_head.disaggregation.use_precip: true` must be set. The source group (`conus-hourly`) splices in the source; the disagg block must be in the experiment config separately. - -**Verification after a fix:** -```bash -# Binary check: head file should be ~107,320 B (disagg + leakance) or ~107,178 B (disagg only) -ls -la .ddrs/runs//checkpoints/epoch_5_mb_9/head.mpk - -# Dataset log at run start: -grep "AORC precip store" .ddrs/runs//run.log -# Should show: "AORC precip store: 290878 catchments" - -# Forcing verification: eval predictions should differ between hourly and daily runs: -md5sum .ddrs/runs//eval/predictions.zarr/predictions/0.0 -md5sum .ddrs/runs//eval/predictions.zarr/predictions/0.0 -# These must NOT be identical -``` - ---- - -### T11: Checkpoint resume issues - -**Resume requires three files in a directory:** -``` -.ddrs/runs//checkpoints/epoch_E_mb_M/ - head.mpk # KAN weights (f16, CompactRecorder) - optim.mpk # Adam moments (f16) - state.json # epoch, next mini-batch, rng state, sampler permutation + cursor -``` - -**Resume trains zero batches:** `experiment.epochs` is at or below the checkpoint epoch. Fix: raise `experiment.epochs` past `E` in `ddrs.yaml`. - -**Resumed trajectory drifts from uninterrupted run:** Expected. Weights and moments are stored as f16 (`CompactRecorder = HalfPrecisionSettings`). The resumed trajectory is numerically valid but will not be bit-identical to an uninterrupted run. - -**`dump_parameters --checkpoint` path gotcha:** Pass the HEAD BASE, not the directory: -```bash -# CORRECT (head base — CompactRecorder appends .mpk): -target/release/dump_parameters --checkpoint .ddrs/runs//checkpoints/epoch_5_mb_9/head ... -# WRONG (directory — will fail to find head.mpk): -target/release/dump_parameters --checkpoint .ddrs/runs//checkpoints/epoch_5_mb_9 ... -``` - ---- - -### T12: Source lock drift - -**What happens.** `ddrs run --strict` exits with code 4 when the data-source fingerprints in `.ddrs/sources.lock` differ from the current `ddrs.yaml`. This preserves evidence; re-locking would overwrite it. - -**Fix (normal):** -```bash -ddrs plan # re-locks sources.lock to match current ddrs.yaml -ddrs run --workflow -``` - -**Fix (investigate first):** -```bash -cat .ddrs/sources.lock # shows last-locked fingerprints -# Compare against current data_sources: paths in ddrs.yaml -# If a path moved or a store was updated, decide whether to re-plan or roll back -``` - ---- - -### T13: Leakance identifiability status (as of 2026-07-05) - -**This section describes an active research limitation — not a bug to fix.** - -**Positive control experiment (2026-07-04, worktree):** A synthetic recoverability test was run to check whether the gradient path can recover a known planted zeta through gauged-only observations. The experiment FAILED: recovery ratio 0.009 vs the >=0.5 bar. Root cause: the windowed training objective has a hotstart-transient noise floor approximately 130× larger than the leakance signal. - -**Implication:** Leakance identifiability is NOT proven. The 2×2 GO-marginal verdict (leakance helps skill on the losing-stream subset under hourly forcing) stands, but the mechanism cannot be confirmed as genuine GW–SW exchange recovery until Phase B is complete. - -**Phase B objective (NOT YET MET as of 2026-07-05):** noise floor <= 0.25 mean L1 (i.e., <= 10% of a converged run's loss). Requires a state-cache hotstart to eliminate the transient. Do not make identifiability claims until Phase B passes. - -**Gradient probe results (2026-07-03, worktree):** -- P1 (gradient starvation to leakance params): REFUTED -- P3 (detectability — signal vs 5% obs band): NO-GO, signal 53× smaller than detectability threshold - -**Summary of leakance diagnosis verdicts (as of 2026-07-02):** - -| Hypothesis | Verdict | Key evidence | -|---|---|---| -| H1: K_D box clips zeta | REFUTED | Median utilization 3.4%; 71.5% of reaches CAN exceed 0.01 m³/s in-box | -| H2: Driving-head starvation | SUPPORTED | Median head 0.021 m; 47% of reaches gaining at eval-window mean | -| H3: KAN variance collapse | REFUTED | Max Spearman(param, attribute) = 0.71 (strong spatial structure) | -| H4: Gauge bias / gradient starvation | SUPPORTED | zeta–uparea ρ +0.76; gauged 11× ungauged median zeta; dry/wet ratio inverted | -| H5: Equifinality with routing params | SUPPORTED (daily only) | Daily Δn = +0.012 (0.59 IQR); hourly Δn nil | -| H6: Wrong yardstick (absolute bar) | REFUTED | Fractional loss agrees: 8.4% lose >1% of local flow | -| H7: Model-form error (d_gw bounds) | REFUTED | 0.0% of d_gw at bounds in any aridity tercile | - -**Do not run K_D widening.** The Phase-3 gate for K_D widening FAILED because H1 was REFUTED. The constraint is the signal, not the box. - ---- - -### T14: Teacher mode OOM - -**Story (2026-07-23).** The synthetic-n teacher run (`probe_zeta_gradient --mode teacher`, full 29-year window, 64,892-reach eval network, CPU backend) died 11 minutes after launch with NOTHING in its own log past the setup lines — no panic, no error. The kernel OOM killer had taken it at 54 GB anon RSS (213 GB total-vm) during its FIRST 365-day chunk; desktop apps (Slack/Hyprland/browser) held ~24 GB of the 93 GB machine and the teacher had the highest oom_score. - -**Discriminating test.** Log ends abruptly after `teacher: N plants, ...` with no `chunk k/n` lines and the process is gone: -```bash -journalctl -k --since | grep -iE 'oom|killed process' -# → "Out of memory: Killed process (probe_zeta_grad)" -``` - -**Memory profile (measured 2026-07-28):** RSS climbs steadily WITHIN a chunk (transient per-timestep allocations) and collapses back to ~4 GB at every chunk boundary. Peak scales with chunk length: ~65 GB at 365 days, ~45 GB at 180 days. - -**Fix:** `--chunk-days 180` (added 2026-07-28; default 365 = old behavior). Cost: disagg boundary-artifact density doubles (0.55% → 1.1% of days over 29 years) — still negligible. State continuity across chunks is exact either way. The 180-day teacher completed the full 59-chunk window in ~14 h wall. - -**Caveat:** `run_state_cache` still hardcodes 365; if you ever pair a state cache with a teacher run, their chunk lengths must match (state boundaries align). - ---- - -### T15: Transient icechunk read failure - -**Story (2026-07-28).** The synthetic-n `distributed` student completed all 5 training epochs, then its eval died at chunk 364/366 with `icechunk read failed at .../daily_dhbv_aorc2f_merit_unit_catchments.ic: object not found` (empty context, no panic). Looked like the store ends before 2010-09-30 — it doesn't. - -**Why it can't be a data hole:** these icechunk Q' stores are divide-major — `Qr` chunk shape `(200 divides, ALL 14,976 days)`. Any time-slice read touches every divide-block chunk object, so eval chunks 1–363 had already read the exact objects chunk 364 "couldn't find". A direct Python probe of the same date range read clean immediately after. Verdict: transient. - -**Triage:** -1. Probe the store at the failing range (from `~/projects/ddr` venv): `icechunk.Repository.open(icechunk.local_filesystem_storage())` → read the failing day-slice. Clean read ⇒ transient. -2. Check `ddrs import --dry-run` — reports the declared time axis and sample-read health. -3. Only if BOTH fail is it a real store problem. - -**Recovery:** training and eval are separate phases — a post-training eval failure leaves the epoch-5 checkpoint valid. Dump parameters from it (`dump_parameters --checkpoint /head`) and re-run eval-only later (legacy `eval` binary) if the diagnostics matter. Drivers chaining multiple runs should treat "workflow exited nonzero but final checkpoint exists" as continue-with-warning, not abort (see `output/synthetic_n/run_students_sequential.sh`). - ---- - -## Part 3 — Pre-flight checklist before any training run - -Use this before starting a new experiment to prevent the most common traps: - -- [ ] `stat ~/.cargo/bin/ddrs` — mtime is newer than your last `src/` change -- [ ] `ddrs sources list` — active group (`*`) matches intended dataset -- [ ] `ddrs plan` — no source drift warnings; `mode:` and `workflow:` agree -- [ ] Config leakance consistency: if `use_leakance: true`, confirm `use_cuda_graphs: false` and all three params in `kan_head.learnable_parameters` -- [ ] If hourly disagg: config has `aorc_precip:` source AND `kan_head.disaggregation.use_precip: true` -- [ ] If resuming: `experiment.epochs` is greater than the checkpoint epoch; checkpoint path ends at `head` base (not the directory) -- [ ] If touching `src/routing/`, `src/geometry.rs`, or `src/sparse/`: run `cargo run --release --example compare_ddr_sandbox` and confirm ABSOLUTE MATCH - ---- - -## Part 4 — Quick-reference test commands - -```bash -# V1 regression gate (routing core, geometry, sparse): -mkdir -p output && cargo run --release --example compare_ddr_sandbox - -# V1 on CUDA + graph-capture path: -DDRS_FORCE_GRAPHS=1 cargo run --release --example compare_ddr_sandbox - -# Leakance gradient correctness: -cargo test --test leakance_gradcheck -cargo test --test leakance_off_parity -cargo test --test zeta_accum - -# Sparse backward correctness: -cargo test --test sparse_gradcheck - -# KAN head parity vs DDR: -cargo test --features fixtures \ - --test kan_head_init_repro \ - --test kan_head_init_parity \ - --test kan_head_fixture_forward \ - --test kan_head_fixture_backward - -# Adjacency ordering and builder parity: -cargo test data_zarr_store::conus_adjacency_loads_real_merit_zarr -cargo test --test adjacency_parity - -# All lib unit tests: -cargo test --lib - -# Full test suite: -cargo test -``` - ---- - -## Provenance and maintenance - -Ground truth for this skill (re-read these files to verify facts remain current): - -```bash -# V1 / comparing-to-DDR reference: -cat /home/tbindas/projects/ddrs/.claude/references/ddrs-comparing-to-ddr.md - -# Stale-binary trap story + leakance 2x2 re-run: -cat /home/tbindas/projects/ddrs/docs/2026-07-01-leakance-hourly-experiment-handoff.md - -# 2x2 findings (all four arms, GO verdict): -cat /home/tbindas/projects/ddrs/docs/2026-07-01-leakance-hourly-findings.md - -# Low-zeta diagnosis (H1–H7 hypothesis verdicts): -cat /home/tbindas/projects/ddrs/docs/2026-07-02-leakance-diagnosis-findings.md - -# Config rules (invariants, leakance enable, CLI lifecycle): -cat /home/tbindas/projects/ddrs/CLAUDE.md - -# CUDA graphs mask NaN (memory note): -cat /home/tbindas/projects/ddrs/.claude/memories/cuda-graphs-mask-nan.md - -# Re-verify rskan version: -grep rskan /home/tbindas/projects/ddrs/Cargo.toml - -# Re-verify config leakance rejection: -grep -n "use_leakance.*use_cuda_graphs\|use_cuda_graphs.*use_leakance" \ - /home/tbindas/projects/ddrs/src/config.rs -``` diff --git a/.claude/skills/ddrs-dev/SKILL.md b/.claude/skills/ddrs-dev/SKILL.md index f538a1e..36d1cb2 100644 --- a/.claude/skills/ddrs-dev/SKILL.md +++ b/.claude/skills/ddrs-dev/SKILL.md @@ -1,54 +1,126 @@ --- name: ddrs-dev -description: Use at the start of any ddrs task to route to the right sub-skill. Load this first when the task type is unclear, when onboarding to the project, or when multiple skill areas may apply. +description: Use for any ddrs building, coding, configuring, testing, running, or debugging task — build failures, config keys, which tests gate a change, launching train/eval jobs, resuming checkpoints, diagnosing a wrong or failed run, and research-status facts needed to avoid re-opening closed questions. Trigger on "build ddrs", "cargo test", "which tests do I run", "config key", "ddrs plan/run", "resume from checkpoint", "training is broken", "wrong result", "OOM", "is X still true". For plotting or interpreting eval output, use ddrs-eval-plots instead. --- -# ddrs skill index +# ddrs development -**Project:** Differentiable Muskingum-Cunge routing solver — BURN 0.21 Rust port of DDR (Python/PyTorch). Gradient-exact. Trains a KAN head to emit per-reach hydraulic parameters. +BURN-0.21 Rust port of DDR's differentiable Muskingum-Cunge routing. Gradient-exact +against the Python reference at `~/projects/ddr/`. A KAN head emits per-reach +hydraulic parameters; a sparse triangular solve routes discharge over the MERIT +network. -**Live research goal (as of 2026-07-06):** Selective-equifinality paper — train on 4 NH inflow sources (daily-lstm, hourly-lstm, dHBV2.0-lumped, dHBV2.0-UH), compare convergence of geometry (identifiable) vs Manning's n (equifinal). Paper draft: `/home/tbindas/projects/ddr_equifinality/paper.tex`. +**CLAUDE.md is already in your context and is the source of truth for**: the seven +invariants, CLI lifecycle, workspace layout, data sources, source groups, and +checkpoint resume. This skill does **not** repeat them. It carries what CLAUDE.md +does not: traps with their discriminating tests, the change→gate matrix, the full +config reference, test-authoring patterns, and the current research status. ---- - -## Route to the right skill +## Route -| I need to… | Use skill | +| I need to… | Read | |---|---| -| Understand what changes are safe to merge | `ddrs-change-control` | -| Debug a failed / wrong-result run | `ddrs-debugging-playbook` | -| Check whether a problem was already solved | `ddrs-failure-archaeology` | -| Understand a load-bearing design decision | `ddrs-architecture-contract` | -| Understand hydrology concepts (NSE/KGE, routing, equifinality) | `ddrs-hydrology-reference` | -| Add, change, or audit a YAML config key | `ddrs-config-and-flags` | -| Set up or rebuild the dev environment | `ddrs-build-and-env` | -| Install the CLI, run a train/eval job, resume a checkpoint | `ddrs-run-and-operate` | -| Measure a result, interpret zeta/gradient/metric diagnostics | `ddrs-diagnostics-and-tooling` | -| Decide whether a change is safe / what tests to run | `ddrs-validation-and-qa` | -| Write a findings doc, spec, plan, or paper section | `ddrs-docs-and-writing` | -| Make an external claim about novelty or results | `ddrs-external-positioning` | -| Plan or execute the selective-equifinality experiment | `ddrs-identifiability-campaign` | -| Verify gradient correctness / design a controlled ablation | `ddrs-proof-and-analysis-toolkit` | -| Identify the next high-value research direction | `ddrs-research-frontier` | -| Design an experiment / enforce evidence standards | `ddrs-research-methodology` | +| Fix a build failure (cmake, netcdf, fork resolution, fixtures) | `references/build-and-env.md` | +| Look up a config key, its default, or a load-time guard | `references/config.md` | +| Know which tests gate the change I just made | `references/testing.md` (tier matrix) | +| Write a new gradcheck / parity / fixture test | `references/testing.md` (authoring) | +| Diagnose a failed, hung, or wrong-result run | `references/traps.md` | +| Check whether a question is already settled | `references/research-status.md` | +| Plot or interpret eval output | skill `ddrs-eval-plots` | ---- +## The five facts that cause the most wasted time -## Critical facts every agent must know +1. **Stale binary.** `cargo build` / `cargo run` do **not** update `~/.cargo/bin/ddrs`. + The manifest stamps `git.sha` from `.git` at runtime, so a run *looks* current + while a weeks-old binary executed. After any `src/` change: `cargo install --path .`. + Self-check: checkpoints must be **directories** (`epoch_E_mb_M/head.mpk`); flat + `epoch_E_mb_M.mpk` files mean a stale binary ran. Full incident: `references/traps.md` T1. -- **Stale binary trap:** `cargo build` does NOT update `~/.cargo/bin/ddrs`. After any `src/` change: `cargo install --path .` -- **V1 gate:** `cargo run --release --example compare_ddr_sandbox` must print `ABSOLUTE MATCH` (max abs < 1e-3 m³/s) after any change to `src/routing/`, `src/geometry.rs`, or `src/sparse.rs`. -- **CUDA graphs mask NaN:** validate new forward paths with `use_cuda_graphs: false` before enabling graphs. -- **Leakance is CLOSED (NO-GO, 2026-07-06):** do not re-open without reading `docs/2026-07-06-leakance-nogo-scientific-summary.md`. -- **Best CONUS result (2026-06-23):** NSE 0.715 / KGE 0.711 (precip-driven disagg + L1, 2,365 gauges). KGE does NOT beat the summed-Q' baseline (0.7172) in any config as of 2026-07-06. +2. **`--workspace` is not optional when you pass `--config`.** The default is + `Workspace::beside(config)`, so `--config config/experiments/x.yaml` silently + creates `config/experiments/.ddrs/`. Always pass + `--workspace /home/tbindas/projects/ddrs/.ddrs`. ---- +3. **Use the right baseline number.** The CONUS summed-Q′ bar is + **0.6781 NSE / 0.7172 KGE on 2,365 gauges**. The widely-copied `0.689 / 0.723` + is a *global* MERIT number on a different 5,224-gauge network and must never be + used as a CONUS bar. See `references/research-status.md`. + +4. **CUDA graphs mask NaN.** `use_cuda_graphs: true` replays the captured graph and + returns a stale finite loss when the real computation went NaN. Validate every + new data path with `use_cuda_graphs: false` first. + +5. **A gauge observes a network sum.** Any per-reach quantity whose only supervision + is downstream discharge is structurally non-identifiable — a sum is not invertible + for its addends. This is why leakance is CLOSED (NO-GO, 2026-07-06) and it + generalizes to any new per-reach term you are tempted to add. -## Provenance and maintenance +## Change → gate, in one table + +Full commands and rationale in `references/testing.md`. + +| You touched | Tier | Must pass | +|---|---|---| +| `src/routing/`, `src/geometry.rs`, `src/sparse/` | **A** | `compare_ddr_sandbox` = ABSOLUTE MATCH, `cargo test --lib`, `--test mmc`, `--test sparse_gradcheck`, all three leakance tests | +| `src/nn/`, `Cargo.toml` rskan tag | **B** | 4-test KAN fixture sweep, then Tier A | +| `src/config.rs`, `src/training/`, other `src/` | **C** | `cargo test --lib`, `cargo test`, `compare_ddr_sandbox` | +| `config/**/*.yaml` only | **D** | `ddrs plan --config … --workspace …` exits 0, no drift | +| Plotting / analysis scripts only | — | no gate | +| `epochs`, `learning_rate`, `batch_size`, loss weights within documented ranges | — | no gate | + +`mkdir -p output` before `compare_ddr_sandbox` — the example calls +`File::create("output/…")` with no `create_dir_all` and panics on a fresh clone. + +## Commands you actually need -Re-verify routing after any new master merge: ```bash -git log --oneline origin/master..HEAD # commits ahead -cargo run --release --example compare_ddr_sandbox # V1 gate +cargo install --path . # refresh the PATH binary — after every src/ change +mkdir -p output && cargo run --release --example compare_ddr_sandbox # V1 gate +cargo test # full suite, 2–5 min +ddrs plan # GPU probe + smoke + baseline (cached) +ddrs run --workflow train-and-test --backend cpu # train then eval +ddrs run --workflow train --backend cpu --max-mini-batches 2 # mechanics smoke +ddrs show ; ddrs status; ddrs gc --keep 5 --keep-successful ``` -Update this index when a new skill is added to `.claude/skills/`. + +**`ddrs run --workflow eval` does not work** — it returns +`"standalone --workflow eval needs a --from-run flag"`, and `--from-run` +is unimplemented (`src/cli/run.rs:322`). Use `--workflow train-and-test`, or the +legacy `eval` binary against an existing checkpoint. **`ddrs init` is a dead stub** +(exits 2, `src/bin/ddrs.rs:167`); use `ddrs plan`. Both are still documented as +working in README.md and `docs/` — see `docs/2026-07-30-docs-and-skills-audit.md`. + +## Verifying a run did what you think + +Grep the run log — these strings are real (`src/data/dataset.rs`, `src/training/bootstrap.rs`): + +``` +streamflow resolution: Daily|Hourly +AORC precip store: N catchments, hourly … +gages_adjacency filter: kept X gauges (dropped Y missing, Z headwater) +warm start: loaded KAN head from … +warm start: no .mpk — Adam starts cold +``` + +There is no `"precip loading"` string — two retired skills told you to grep for it. + +## Maintenance + +This skill and `ddrs-eval-plots` are the only two skills in this repo. When a run, +eval, or export completes, update the relevant section here in the same session that +produced the knowledge — do not leave it only in a findings doc. If a rule here is +superseded, correct it in place with the new nuance rather than deleting it. + +**The mdBook under `docs/` is now the canonical prose documentation** — it is a +strict superset of the deleted `.claude/references/` copies. The old +`regenerate-docs` skill was removed: its input contract pointed at +`.claude/references/*.md` frontmatter that no longer exists, its +`.regenerate-state.json` was never created, and its dataflow diagram published an +"MLP head" node that violates invariant 5. If you rebuild that mechanism, keep its +one genuinely good rule: **never invent a function signature, file path, or API +detail — if a claim is not backed by a cited source, flag it rather than writing it +into a chapter.** This audit found that rule violated repeatedly. + +Every factual claim in this skill was verified against source on 2026-07-30. Claims +carrying a date are volatile; re-verify before citing externally. The audit that +produced this consolidation is `docs/2026-07-30-docs-and-skills-audit.md`. diff --git a/.claude/skills/ddrs-dev/references/build-and-env.md b/.claude/skills/ddrs-dev/references/build-and-env.md new file mode 100644 index 0000000..2a0f9ff --- /dev/null +++ b/.claude/skills/ddrs-dev/references/build-and-env.md @@ -0,0 +1,80 @@ +# Build and environment + +Verified against `Cargo.toml` and source on 2026-07-30. + +## Hard prerequisites + +- **cmake.** `netcdf = { version = "0.12", features = ["static"] }` compiles bundled + HDF5 + netcdf-c. Symptom without it: `error: failed to execute process 'cmake'`. + The static build is deliberate — HPC module-provided netcdf/hdf5 are + Intel-MPI-flavored shared libs that break `ddrs` outside `module load`. +- **A CUDA toolkit, even for CPU-only work.** `burn-cuda` and `cudarc` + (`cuda-version-from-build-system`) are **non-optional** dependencies, and + `use burn_cuda::Cuda;` is unconditional in `src/sparse/dispatch.rs`, + `src/routing/mmc.rs`, `src/cli/run.rs`, `src/training/driver.rs`, + `src/cli/system.rs`. CPU *execution* is supported via `--backend cpu`; CPU-only + *compilation* is not. (`docs/setup.md` claims otherwise — it is wrong.) + +If a system HDF5 leaks into the static build: `unset HDF5_DIR; unset NETCDF_DIR`. + +## Fork pins + +All **13** `burn-*` crates must resolve from `github.com/taddyb/burn` branch +`ddrs-sp7-primitive-ctor`; all **11** `cubecl*` crates from +`github.com/taddyb/cubecl` branch `ddrs-release`. `rskan` is pinned to tag +`v0.1.3`. + +The patch exists for exactly two accessors upstream keeps `pub(crate)`: +`cubecl-cuda`'s `CudaServer::stream() -> CUstream` and `burn-cubecl`'s +`CubeTensor::from_handle(...)`. The plan is to upstream them as SP-8 and drop +`[patch.crates-io]` entirely. + +| Symptom | Cause | Fix | +|---|---|---| +| `error[E0277]: the trait Device is not implemented` | Split resolution — some burn crates from crates.io, some from the fork | `cargo tree -p burn-std` and reconcile | +| `error: failed to resolve patches` | Fork branch renamed or deleted | Check the branch exists on the fork | + +Iterating on a fork: push to the fork branch, then `cargo update -p `. +**Never commit `path = "..."` overrides** — they break the public build. + +## Fixtures + +Gitignored (`/fixtures/`) and not auto-created. Two families: + +**V1 sandbox** — `fixtures/sandbox/`, read by `examples/compare_ddr_sandbox.rs`. +Regenerate with: +```bash +cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/export_ddr_sandbox.py +``` +> **Only the desktop's `~/projects/ddr` working tree is a valid reference.** It +> carries unpushed `geometry/trapezoidal.py` work. A fixture regenerated from a +> clean DDR clone diverges ~1% (max abs ≈ 0.55 m³/s) at *every* ddrs commit — +> that is a wrong reference, not a port bug. Details: +> `docs/reference/ddr-comparison.md` §Regenerating fixtures. + +**KAN parity** — `tests/fixtures/` (tracked, unlike `/fixtures/`), loaded behind +`#[cfg(feature = "fixtures")]`. Regenerate under DDR's venv with +`scripts/dump_kan_fixture.py` (and `scripts/dump_kan_init_stats.py`). +There is no `dump_kan_head.py` / `dump_kan_weights.py` / `dump_kan_forward.py` — +three retired skills named files that do not exist. + +## Gitignored artifacts + +`output/`, `/fixtures/`, `examples/fixtures/`, `.ddrs/`, `.ddrs-synthetic-n-*/`, +`ddrs.yaml`. Anything under `output/` is machine-local and not reproducible from a +clone — state that when citing data living there. + +## Cargo features + +`fixtures` (KAN parity tests) and `cuda`. `--features fixtures` is required for the +Tier B sweep. + +## Worktrees + +Worktrees live in `.claude/worktrees//` (gitignored). Two gotchas: +- A relative `target/release/...` path resolves against the **main tree**, so you + can silently run a stale main-tree binary from inside a worktree. Use absolute + paths or `cargo run`. +- Fresh worktrees lack gitignored `fixtures/` and `output/`. Copy `fixtures/` from + the main tree (watch for accidental nesting into `fixtures/fixtures/`) and + `mkdir -p output`. diff --git a/.claude/skills/ddrs-dev/references/config.md b/.claude/skills/ddrs-dev/references/config.md new file mode 100644 index 0000000..40b3a75 --- /dev/null +++ b/.claude/skills/ddrs-dev/references/config.md @@ -0,0 +1,194 @@ +# Config reference + +Struct: `src/config.rs::Config`. Loaded via +`Config::from_yaml_file_with_mode(path, ConfigMode::Training|Testing)`. +Six top-level sections. Verified against source 2026-07-30. + +**No `deny_unknown_fields` anywhere** — a typo'd key silently takes its default +instead of erroring. This is the single most common cause of "my config change did +nothing". + +## Top level + +| Key | Default | Notes | +|---|---|---| +| `mode` | `training` | `training` \| `testing` | +| `workflow` | none | Cross-validated against `mode`: `training` ↔ `{train, train-and-test}`, `testing` ↔ `eval` | +| `geodataset` | `merit` | | +| `device` | `0` | CUDA ordinal | +| `seed` / `np_seed` | `42` / `42` | | + +`kan_head:` accepts `mlp:` as a serde alias (backward compat with pre-KAN configs). + +## `data_sources:` — 8 path fields + +`attributes` is a `Vec` accepting a **single path or a list** +(feature-concatenated on COMID, NaN-filled, `deserialize_one_or_many_paths`). +An empty list is a hard error. + +Adjacency rule: provide **either** both `conus_adjacency` + `gages_adjacency`, +**or** `geospatial_fabric` (managed build into `.ddrs/adjacency//`). +Exactly one of the pair ⇒ error; neither source ⇒ `"adjacency sources are missing"`. +For multi-layer gpkg set `geospatial_fabric_layer` (participates in the cache key). + +`aorc_precip` is required whenever `kan_head.disaggregation:` is present — see below. + +## `experiment:` + +| Key | Production value | Notes | +|---|---|---| +| `batch_size` | 64 | **GAUGES** in training | +| `start_time` / `end_time` | 1981/10/01 – 1995/09/30 | | +| `epochs` | 5 | Resume requires raising this past the checkpoint's epoch or zero batches train | +| `rho` | 90 | Training window length in days | +| `warmup` | 5 | | +| `shuffle` | true | | +| `learning_rate` | `{1: 0.001, 3: 0.0005}` | Epoch→lr schedule. **Ignored under `optimizer: adadelta`** — AdaDelta derives its step from RMS[Δx]/RMS[g] | +| `grad_clip_max_norm` | 1.0 | | +| `checkpoint` | none | Directory `epoch_E_mb_M/` holding `head.mpk`, `optim.mpk`, `state.json` | +| `state_cache` | none | **netCDF** (not zarr) day-boundary discharge state, from `probe_zeta_gradient --mode state-cache` | +| `optimizer` | `adam` | `adam` \| `adadelta`. A checkpoint refuses to load across kinds rather than reinterpreting moment tensors | +| `use_grad_accum` | false | Master switch for optimizer micro-batching | +| `grad_accum_steps` | none | Micro-batches accumulated into one optimizer step, with exact valid-count weighting | + +> `optimizer` / `use_grad_accum` / `grad_accum_steps` / `loss.kind: nse-batch` land +> with the gradient-accumulation work (PR #31, branch `exp_train`). If you are on a +> commit without them, `nse-batch` fails config load with +> `unknown variant 'nse-batch'`. + +Under `use_grad_accum: true`, `--max-mini-batches` counts optimizer **STEPS**, not +mini-batches. + +### `experiment.loss:` + +`LossConfig::default()`: `kind: l1`, `nnse_weight 1.0`, `kge_weight 1.0`, +`r_weight 1.0`, `alpha_weight 1.0`, `beta_weight 1.0`, `kge_clamp 10.0`, `eps 0.1`. +Sub-keys are kebab-case in YAML. + +| `kind` | Objective | +|---|---| +| `l1` | `mean(\|p − o\|)`. Historical default | +| `nnse-kge` | Per-gauge `nnse_weight·(1−NNSE) + kge_weight·(1−KGE)` | +| `kge` | KGE term alone | +| `nse-batch` | dHBV `NSELossBatch`: mean over valid (day, gauge) of `(sim−obs)²/(σ_gauge+eps)²`, σ fixed over the training period | + +`kge_clamp` exists because a single near-constant gauge once drove batch loss to +~1e4. + +Why the loss menu exists: L1 and NSE are both maximized at a simulated variance +*below* observed (NSE's optimum is at `α = r < 1`), so they reward the router for +over-attenuating peaks. Note this did **not** turn out to be the binding constraint — +see `research-status.md`. + +## `testing:` overlay + +Replaces matching `experiment:` keys; absent keys inherit. Covers `start_time`, +`end_time`, `batch_size`, `rho`, `warmup`, `epochs`, `grad_clip_max_norm`, +`checkpoint`. + +- **`batch_size` semantically shifts**: GAUGES in training, **DAYS** in testing. +- `rho: null` is a *double-Option* (`deserialize_option_option`) — "present and null" + (disabled) is distinct from "absent" (inherit). + +## `kan_head:` + +Code defaults `grid: 5`, `k: 3`; production overrides to `grid: 50`, `k: 2` for DDR +parity. `hidden_size: 21`, `num_hidden_layers: 2`. + +Ten production `input_var_names`: `SoilGrids1km_clay`, `aridity`, `meanelevation`, +`meanP`, `NDVI`, `meanslope`, `log10_uparea`, `SoilGrids1km_sand`, `ETPOT_Hargr`, +`Porosity`. + +### `kan_head.disaggregation:` — the real fields + +> **There is no `use_precip`, `use_attributes`, or `use_temp` key.** They were +> removed in `334f0fe` ("rework disaggregation head to KAN + basin-normalized +> precip"). CLAUDE.md, `src/config.rs:113`'s comment, and +> `config/sources/conus-hourly.yaml:5` all still reference the phantom key. +> **Current contract:** presence of the `disaggregation:` block ⇒ the head always +> consumes precip ⇒ `data_sources.aorc_precip` is mandatory, else +> `MeritGagesDataset::open` errors. It cannot silently degrade to flat repeat-24. + +| Key | Default | Notes | +|---|---|---| +| `hidden_size` | 16 | | +| `num_hidden_layers` | 1 | | +| `grid` | 3 | | +| `k` | 3 | | +| `boundary_blend` | 0.0 | Day-boundary shape continuity λ; only used when `chunk_days <= 1` | +| `chunk_days` | 1 | `> 1` relaxes mass balance to the chunk aggregate, letting storms span day boundaries | +| `pretrained_checkpoint` | none | CompactRecorder `.mpk`. The five architecture fields above MUST match or `load_record` fails loudly | +| `freeze` | false | `Module::no_grad()`. Requires `pretrained_checkpoint` | + +## `params:` + +| Key | Default | Notes | +|---|---|---| +| `sparse_solver` | `cpu` | On a non-CUDA backend, `cuda` **silently WARN-falls-back** to cpu. An unrecognized value **panics** | +| `use_cuda_graphs` | **false** | `config/merit_training.yaml` *sets* true — that is a YAML value, not the code default | +| `use_leakance` | false | | +| `leakance_losing_only` | **true** | Clamps `head = max(0, depth − d_gw)`, so gaining reaches produce `zeta ≡ 0` | +| `leakance_impervious_threshold` | 0.7 | Masks reaches whose `corridor_impervious` is **`>`** this value (not `≥`) | +| `tau` | 3 | **Not** a routing sub-step count. It is the hourly→daily trim phase offset in `tau_trim_and_downsample`: DDR's slice `[13 + tau : -11 + tau]`, then area-pool to days (`src/training/loss.rs:17-45`). Nothing in `src/routing/` reads it | +| `log_space_parameters` | `["p_spatial"]` | | +| `defaults` | `{p_spatial: 21.0}` | Value used when a parameter is not in `learnable_parameters` | + +### `parameter_ranges` (7 keys parsed) + +`n [0.015, 0.25]`, `q_spatial [0, 1]`, `p_spatial [1, 200]`, `x_storage [0, 0.5]`, +`k_d [1e-8, 1e-6]`, `d_gw [-2, 2]`, `leakance_factor [0, 1]`. + +Case quirk: **`K_D` is uppercase in YAML, `k_d` in Rust.** `x_storage` is only +consumed when listed in `learnable_parameters`; otherwise routing uses a constant 0.3. +`p_spatial` is the Leopold-Maddock **coefficient**; `q_spatial` is the exponent +(`docs/algorithm.md` has this backwards). + +### `attribute_minimums` + +`discharge 1e-4`, `slope 1e-3`, `velocity 0.01`, `depth 0.01`, `bottom_width 0.01`. + +## Load-time guards + +Four validators run at `Config::from_yaml_file`, plus one at dataset open. + +| Guard | Rejects | Error substring | +|---|---|---| +| `validate_mode_workflow` | `mode`/`workflow` disagreement | `"conflicting top-level keys"` | +| `validate_data_sources` | one of the adjacency pair | `` "`gages_adjacency` is missing" `` | +| | neither adjacency nor fabric | `"adjacency sources are missing"` | +| | `geospatial_fabric_layer` on a non-gpkg | `"geospatial_fabric_layer"` + `".gpkg"` | +| `validate_leakance` | `use_leakance` + `use_cuda_graphs` | both key names | +| `validate_disagg_pretrained` | `freeze: true` without `pretrained_checkpoint` | `"freeze: true requires pretrained_checkpoint"` | +| `validate_grad_accum` | `grad_accum_steps: 0` | `"grad_accum_steps: 0"` | +| | `use_grad_accum: true` with steps < 2 | `"requires grad_accum_steps: N with N >= 2"` | +| `validate_disagg_vs_resolution` (runtime, `src/data/dataset.rs`) | `disaggregation:` + hourly-native streamflow store | hard error | + +## Adding a new routing parameter + +1. Add the range to `ParameterRanges` (`src/config.rs`) and to + `config/merit_training.yaml`'s `params.parameter_ranges`. +2. Add it to `kan_head.learnable_parameters` in the experiment config so the head + emits it. +3. Thread it through `denormalize` (`src/routing/utils.rs`) and + `SpatialParameters` (`src/routing/mmc.rs`). +4. If it enters the timestep, add it as a `Backward` parent and widen the op's `N`. +5. Add a gradcheck case (`references/testing.md`) and an OFF-parity test. +6. Update this file and `config/merit_training.yaml`'s comments. + +## Adding a new boolean flag + +1. Field on `Params` with `#[serde(default)]` and an explicit `Default` impl — + absent must be byte-identical to the old behavior. +2. If it is incompatible with another flag, add a validator and a test asserting the + error substring. +3. Add an OFF-parity test that asserts **bit-exact** equality with the pre-feature + expected array, and an ON test that asserts the output actually changes. Only the + second catches a silent no-op. + +## Leakance: enabling it + +Three changes are required together — `params.use_leakance: true` (which forces +`use_cuda_graphs: false`), `K_D`/`d_gw`/`leakance_factor` in +`kan_head.learnable_parameters`, and matching `parameter_ranges`. The term is +CLOSED (NO-GO) as a research direction but remains code-complete and gradient-exact; +see `research-status.md` before touching it. diff --git a/.claude/skills/ddrs-dev/references/research-status.md b/.claude/skills/ddrs-dev/references/research-status.md new file mode 100644 index 0000000..bc7eb70 --- /dev/null +++ b/.claude/skills/ddrs-dev/references/research-status.md @@ -0,0 +1,213 @@ +# Research status and authoritative numbers + +Re-verified 2026-07-30. This file replaces the leakance/equifinality status tables +that were duplicated across twelve retired skills — where the same superseded +paragraph appeared four to six times and none were updated when one findings doc +retired all of them. + +**Cite from this file, not from memory, and not from a skill that predates it.** + +## Gauge-set definitions — memorize these + +Most wrong numbers in this repo are population confusions, not arithmetic errors. + +| Set | N | Definition | +|---|---|---| +| Raw gage list | 3,211 | `gages_3000.csv`. **Pre-fix baseline population** — includes 513 phantom-zero single-divide gauges | +| DA_VALID | 2,859 | after the drainage-area validity filter | +| **Training / eval set** | **2,365** | after the `gages_adjacency` filter (dropped 494 headwater). **Every trained median is on this set** | +| Post-fix baseline population | 2,698 | 3,211 − 513 headwater. `ddrs plan` baselines from 2026-07-29 onward | +| Global matched set | 5,224 | a **different network** (global MERIT). Only in `6_19_26_journal.md` (repo root, not `docs/`) | + +## Benchmarks — CONUS, eval 1995-10-01 → 2010-09-30, 2,365 gauges + +| Quantity | NSE | KGE | Source | +|---|---|---|---| +| **Summed-Q′ baseline (dHBV2-UH store) — the CONUS bar** | **0.6781** | **0.7172** | `docs/2026-06-23-precip-disaggregation-findings.md` | +| **Best documented trained result** — precip-driven disagg + L1, run `2026-06-23T02-49-12Z-conus-hourly-train-and-test` | **0.7152** | **0.7106** | same | +| Δ vs baseline | **+0.037** | **−0.007** | same | +| Precip-disagg + `nnse-kge` (`2026-06-24T00-03-01Z`) | 0.710 | 0.710 | no dual win | +| Precip + temperature, L1 (`2026-06-24T02-10-49Z`) | 0.716 | 0.709 | temp does not earn its keep | +| daily-OFF flat repeat-24 (`2026-06-05T01-41-16Z`) | 0.700 | 0.724 | | + +Precip contribution decomposition (ON−OFF / OFF−base / ON−base): +NSE +0.020 / +0.018 / +0.037; KGE +0.018 / **−0.025** / −0.007. Reading: bare disagg +trades KGE for NSE; real precip timing rescues what bare disagg destroys. + +### The KGE claim — restate it carefully + +The long-standing claim was "KGE does not beat the summed-Q′ baseline in any +config". That was true through 2026-07-06 for the dHBV2-UH store. + +**As of 2026-07-30 it needs qualification.** Run +`2026-07-30T00-24-24Z-train-and-test` on the newer +`daily_dhbv2_distributed_aorc2f_merit_unit_catchments.ic` store scored median +NSE **0.6799** / KGE **0.7194** on 2,365 gauges against its own eval-subset baseline +of 0.6744 / 0.7082 — beating it on **both** metrics. But its absolute NSE is well +below the 0.7152 benchmark, and the store differs, so this is not a like-for-like +improvement. **No findings doc covers this run.** Treat it as unwritten evidence: +do not repeat the blanket claim undated, and do not upgrade it to a result either. + +## Closed campaigns — do not re-open + +### Leakance (GW–SW exchange): **CLOSED — NO-GO, 2026-07-06** + +Authority: `docs/2026-07-06-leakance-nogo-scientific-summary.md`. Read §3 before +proposing any retry. + +The term is code-complete and gradient-exact. **Do not remove it.** But it is not +promotable, and identifiability is **REFUTED — not "pending"**. + +| Finding | Value | +|---|---| +| Recovery ratio R1, clean objective, 58 planted reaches | **0.008** (bar ≥ 0.5) | +| Noise floor after the Phase B fix | 1.5 → **0.11 m³/s** — and recovery did **not** improve ⇒ the noise floor is REFUTED as the cause | +| Term usage | active on 78.2% of 64,892 eval reaches, Σ\|zeta\| = 1485 m³/s — not collapsed | +| Planted-reach share of that flux | **0.1%**; zero plants in the top-10 zeta reaches ⇒ the optimizer smears rather than localizes | +| Phase C Leg 1 (skill) | losing-subset ΔKGE +0.006 (bar ≥ +0.01) — FAIL | +| Phase C Leg 2 (equifinality) | Δn IQR 0.0143, ρ(Δn, zeta_net) +0.079 — PASS | +| Phase C Leg 3 (external) | ρ(\|zeta\|, bed-relative WTD) **−0.355** (bar > +0.3, and wrong sign) — FAIL | + +**The mechanism, which is the durable lesson:** a gauge observes Σ(flux) over its +entire upstream network, and a sum is not invertible for its addends. Any per-reach +term whose only supervision is downstream discharge is structurally +non-identifiable — invariant to optimizer, objective, input richness, and data +volume. Every rival explanation (gradient starvation, objective noise, uninformative +inputs, sign ambiguity) was individually REFUTED. + +**Superseded — do NOT cite as current:** R1 = 0.009 · "~130× noise floor" as the +live blocker · "Phase B is required before an identifiability claim" · the 2×2 +"GO — marginal" verdict as a promotion claim · "K_D widening is the top follow-up" +(it was widened to `[1e-8, 1e-5]` in Phase C and the field is still anti-physical) · +"K_D widening is NOT recommended" (moot). **Never write "leakance is identifiable" +in any form.** + +### Selective equifinality (H1–H6): **all INCONCLUSIVE or REFUTED-not-clean** + +Authority: `docs/2026-07-07-lstm-equifinality-v2-findings.md` and +`docs/2026-07-09-h5-h6-equifinality-v2-findings.md`. + +| # | Registered verdict | Key number | +|---|---|---| +| H1 geometry converges, n diverges | **REFUTED** | like-for-like n relative spread 0.4512 exceeds every geometry spread — the direction *reverses* | +| H2 n-divergence predicted by Q′ disagreement | **REFUTED** | ρ = −0.248 reach-scale, **−0.380** network-scale | +| H3 gradient alignment is selective | **REFUTED** | mean cosine n 0.656 > p 0.383 > q 0.361 — n aligns *more* | +| H4 gradient decays with gauge distance | **INCONCLUSIVE** | 3 S / 3 R / 3 I across 9 cells | +| H5 parameter-swap transfer | **INCONCLUSIVE** | the control (+0.1251) **exceeds** the primary (+0.0953) ⇒ source-disagreement effect ≈ 0; 10/2,340 gauges carry 82% of the penalty | +| H6 loss-landscape degeneracy | **INCONCLUSIVE** | anisotropy 44× (R1) / 65× (R3), but **n is the stiff axis and p the sloppy one — the inverse of the campaign's framing** | + +**Cite NEITHER direction.** The one audit-robust fact worth keeping is +geometry-gradient orthogonality across distinct Q′ stores (raw cosine 0.023–0.095 vs +ceilings 0.39–0.59). + +Do not cite v1's "isotropic bowl" or "neither swap moves the loss" framings — the v1 +statistics were invalidated by the v2 audit (v1 compared against *unpaired* window +std; the correct paired test has 15–40× smaller variance, and the 5%-sublevel +contour saturated, swallowing 100–105 of 121 grid points). + +### Q′-store waves (2026-07-16) — carry the population caveat + +Trained medians on 2,365 gauges: AORC2F distributed 0.3437/0.3256 · AORC2F lumped +0.5259/0.5175 · daily-lstm 0.5674/0.6169 · hourly-lstm 0.5543/0.4852. None beats the +0.7152/0.7106 benchmark. + +⚠️ **Every "Δ vs own baseline" figure in `docs/2026-07-16-*` and +`docs/2026-07-07-lstm-equifinality-findings.md` is population-inconsistent** — the +baseline column is the 3,211-gauge median (including phantom zeros) while the trained +column is 2,365 gauges. Recomputed population-matched, the hourly-lstm "+0.022 NSE +gain from routing" **reverses to −0.051**. Those docs need a correction note. + +### Synthetic-n recoverability — INTERIM, 1 of 4 arms + +`docs/2026-07-22-synthetic-n-recoverability-findings.md`. S1–S5 are **not yet +computable**. Arm-1 preview (explicitly not a verdict): n median abs err 0.0354, +corr(truth, recovered) 0.736, slope vs `log10_uparea` −0.0193 recovered vs −0.0421 +true (right sign, ~54% attenuated). Two caveats that must travel with any S3 result: +teacher and all four students share the same **frozen** disagg head (report as +"consistent under a shared, frozen disagg head"), and these four arms are **not** the +paper's R1–R5. + +## Do-not-use list + +| Number / claim | Why | +|---|---| +| `0.689 / 0.723` as a CONUS bar | Global MERIT, 5,224 gauges, different network. Use 0.6781 / 0.7172 | +| `+0.026` NSE improvement | Computed against the global baseline. The correct value is **+0.037** | +| Any "own baseline" NSE from the 07-07 / 07-16 docs | 3,211-gauge population including 513 phantom zeros. KGE is unaffected (phantom gauges are NaN-KGE and were dropped) | +| R1 = 0.009 / "130× noise floor" as the live blocker | Superseded by R1 = 0.008 on the fixed objective | +| "leakance is identifiable" (any phrasing) | Explicitly forbidden by the NO-GO summary §7 | +| H1–H6 in either direction | INCONCLUSIVE | +| "KGE has never beaten the baseline", undated | Needs the 2026-07-30 qualification above | + +## Structural constants (stable) + +CONUS 346,321 reaches / 338,814 edges · eval network 64,892 reaches for the +2,365-gauge set (**132,336** for the 3,211-gauge LSTM set — these are different +numbers and were conflated) · global fabric 2,939,408 reaches, 6,051 gauges · +BURN 0.21 · rskan tag `v0.1.3` · V1 gate < 1e-3 m³/s. + +The sparse backward lives in **`src/sparse/`** (`mod.rs`, `dispatch.rs`, +`cusparse.rs`) — four retired skills and CLAUDE.md cite a non-existent +`src/sparse.rs`. `TimestepLeakanceOp: Backward` is defined in +`src/routing/mmc_op.rs`, not in `src/routing/leakance.rs` (which exports +`zeta_forward` / `zeta_backward` / `ZetaGrads`). + +## Evidence standard + +The house rules that produced the results above, worth keeping: + +1. **Pre-register hypotheses in a spec before running anything.** The hypothesis + table must derive from the spec, not be reverse-engineered from results. +2. **Three verdict states only: SUPPORTED / REFUTED / INCONCLUSIVE.** Never + "confirmed", "partially supported", or "likely". +3. **Define the gate as a single boolean on a computable number, before the battery + runs.** Spend GPU only if it opens. When a gate fails, write "the gate FAILED", + not "we decided not to proceed". +4. **Order instruments cheapest-first**: adjoint reachability (no training) → + detectability bound (forward-only; detect if `|mean ΔQ| > 99th-pct rerun noise` + **and** `> 5% of the gauge's mean flow`) → synthetic recoverability (full + training). **If detectability is NO-GO, stop — no training objective can learn the + term.** Run the adjoint map at *both* cold and trained points to separate + "converged-flat" from "never-saw-signal". +5. **Every numeric claim carries a unit, a gauge count, and an eval window.** + "median NSE 0.715 (2,365 gauges, 1995/10–2010/09)", not "NSE was good". +6. **When a finding overturns a prior doc, name the prior doc and the exact item.** + The NO-GO summary §5 is the model implementation. +7. **A positive control needs a continuous baseline eval before training starts**, + and the recovery target must be the *flux field*, not a degenerate parameter + triple. +8. Watch the dam/lake regulation confound in any differential-gauging argument — + restrict detectability sites to GAGES-II Ref class. + +## Doc conventions + +| Doc type | Location | Naming | +|---|---|---| +| spec (before code runs) | `docs/superpowers/specs/` | `YYYY-MM-DD--design.md` | +| plan (tasks from a spec) | `docs/superpowers/plans/` | `YYYY-MM-DD-.md` | +| findings (after it ran) | `docs/` | `YYYY-MM-DD--findings.md` | +| handoff (mid-experiment) | `docs/` | `YYYY-MM-DD--handoff.md` | +| reference (data contract, API) | `docs/reference/` or `docs/` | descriptive, no date | + +A findings doc opens with the header block (spec / plan / script / prior finding), +then a **one-line verdict** before any section, then §1 pre-registered hypotheses, +§2 methods, §3 results with bold verdicts, §4 conclusions, §5 next steps (dropped +items labeled "Dropped — reason"), §6 raw output, §7 reproduce. If a findings doc for +that experiment exists, add a datestamped section rather than creating a duplicate. + +Always document the **binary provenance** in a methods section — the 2026-07-01 2×2 +was invalidated by a stale binary and the manifest did not reveal it. + +## Open, not closed + +- **Backward CUDA graphs (SP-11).** Forward capture landed (V7a 0.385, V10 29.2% + launch reduction); the backward pass is not captured. Path: profile → fuse backward + kernels → capture. Blocked for leakance configs (the leakance kernel has no capture + path). +- **Global scale-out.** `ddrs sources use global && ddrs plan && ddrs run`. Needs a + 1-epoch smoke and per-provider eval before any full run. +- **The paper** (`/home/tbindas/projects/ddr_equifinality/paper.tex`, "Beyond + Equifinality in Differentiable River Routing", Bindas & Shen). Five arms R1–R5, four + pre-registered hypotheses, leakance disabled (ζ=0) in every arm. Abstract, intro, + and methods are drafted; the Results section is still a `\tbd{}` skeleton. Only the + dHBV2 cross-family arms remain unrun. diff --git a/.claude/skills/ddrs-dev/references/testing.md b/.claude/skills/ddrs-dev/references/testing.md new file mode 100644 index 0000000..04f5307 --- /dev/null +++ b/.claude/skills/ddrs-dev/references/testing.md @@ -0,0 +1,127 @@ +# Testing: gates, inventory, authoring + +70 test files, 233 `#[test]` fns. Full suite runs in 2–5 minutes on the dev machine. +Each `tests/*.rs` compiles as its own crate; `mod common;` pulls `tests/common.rs`. + +**Do not hard-code test counts in docs.** Prior skills claimed "leakance_gradcheck +8/8" and "zeta_accum 6/6"; the real counts are 16 and 8, and they grow. Assert "all +pass" instead. + +## Tier gates + +### Tier A — routing core +`src/routing/`, `src/geometry.rs`, `src/sparse/` + +```bash +mkdir -p output && cargo run --release --example compare_ddr_sandbox # must print ABSOLUTE MATCH +cargo test --lib +cargo test --test mmc +cargo test --test sparse_gradcheck +cargo test --test leakance_gradcheck # run even if you did not touch leakance — +cargo test --test leakance_off_parity # any routing change can disturb OFF-parity +cargo test --test zeta_accum +``` + +### Tier B — KAN head +`src/nn/`, `Cargo.toml` rskan tag + +```bash +cargo test --features fixtures \ + --test kan_head_init_repro --test kan_head_init_parity \ + --test kan_head_fixture_forward --test kan_head_fixture_backward +``` +Then run Tier A to confirm the head change did not disturb routing. + +### Tier C — everything else in `src/` +```bash +cargo test --lib && cargo test && \ + mkdir -p output && cargo run --release --example compare_ddr_sandbox +``` +If you touched `src/training/forward.rs` (disagg / leakance threading), also run +`cargo test --test leakance_off_parity`. + +### Tier D — config YAML only +```bash +ddrs plan --config config/experiments/.yaml \ + --workspace /home/tbindas/projects/ddrs/.ddrs +``` +Exit 0, no drift warnings. + +## What covers what + +| Area | Tests | +|---|---| +| Routing | `mmc`, `routing_utils`, `geometry` | +| Sparse / autograd | `sparse_gradcheck`, `sp8_gradcheck` | +| KAN head | the 4 `kan_head_*` fixture tests (need `--features fixtures`) | +| Leakance | `leakance_gradcheck`, `leakance_off_parity`, `zeta_accum` | +| Adjacency | `adjacency_parity` (managed builder byte-identical to the petgraph engine on `order`/`indices_0`/`indices_1`), `adjacency_build`, `data_zarr_store::conus_adjacency_loads_real_merit_zarr` (invariant 3 on real CONUS data) | +| CLI / data | `data_dataset`, `data_static`, `cli_manifest`, `cli_lockfile`, `cli_json_contract` | +| Checkpointing | `checkpoint_resume` (**not** `cargo test --lib training::checkpoint` — that module has zero tests) | +| Eval robustness | `cargo test --lib training::eval::tests` | +| Disagg freeze | `disagg_freeze` | +| Grad accumulation | `grad_accum_equivalence`, `adadelta_nse_smoke` (on `exp_train`) | + +Two commands that appear in CLAUDE.md and `docs/` and **run zero tests**: +`cargo test --test mmc mc_routes_linear_chain` (no such test) and +`cargo test --test sp8_gradcheck -- --ignored` (nothing in that file is `#[ignore]`, +so it passes vacuously). Do not use either as a gate. + +## Acceptance thresholds + +| Check | Bar | +|---|---| +| DDR sandbox max abs diff | < 1e-3 m³/s | +| Typical per-reach rel diff | ~1e-7 (the f32 floor). Regression past 1e-6 warrants investigation | +| `zeta_accum` headwater identity | abs error < 1e-6 · max(\|zeta\|, 1.0) | +| KAN fixture forward/backward | bit-for-bit | +| Gradcheck | rel < 5e-3 **or** abs < 1e-4 | + +## Authoring patterns + +### Gradcheck +Follow `tests/leakance_gradcheck.rs`. Central difference +`(f(x+eps) − f(x−eps)) / (2·eps)` on the deterministic `NdArray` backend. + +**ε depends on the parent's nonlinearity — this is the non-obvious part:** +- Parents the output is **nonlinear** in (`n`, `q_spatial`, `p_spatial`): + `eps = max(1e-3·|base|, 1e-3)`. Smaller hits the f32 noise floor. +- Parents the output is **exactly linear** in (`K_D`, `d_gw`, `leakance_factor`): + use a **large** step — `K_D 4e-7` (base 5e-7), `d_gw 1.5` (base 0.0), + `factor 0.4` (base 0.5). Central differences have zero truncation error for + exactly-linear parents, while a tiny step sinks Δloss into the f32 round-off of + `q_next.sum()` (~O(500)). + +Tolerances `REL_TOL = 5e-3`, `ABS_TOL = 1e-4`; accept on rel **or** abs. The base +point must be **interior** to all ranges (no clamp saturation), and for leakance a +**losing** config (`depth > d_gw`). `x_storage` is a non-differentiated constant of +`TimestepLeakanceOp` — `.grad()` returns `None`; do not try to sweep it. + +### Parity +Follow `tests/leakance_off_parity.rs`. Capture the `EXPECTED` array **before** the +feature branch exists, on the deterministic CPU backend, and assert **bit-exact** +equality — not approximate. Test **both** directions: +1. feature off ⇒ byte-identical to the committed expectation, and +2. feature on ⇒ output demonstrably changes. + +Only the second catches a silent no-op, which is exactly what the 2026-07-01 stale +binary produced. + +### Fixtures +Generate under `~/projects/ddr/.venv`, write raw bytes to `tests/fixtures/` +(**tracked**, unlike the gitignored top-level `/fixtures/`), load behind +`#[cfg(feature = "fixtures")]`. + +## Why the zeta_accum headwater identity works + +Reach 0 is a headwater (no upstream), so the triangular solve gives +`x_sol[0] = b_rhs[0]`. Therefore `q_no_leak[0] − q_leak[0] == zeta[0]` exactly. +That identity is what proves the eval-time zeta accumulator reports precisely what +was subtracted from `b_rhs`, rather than a recomputation that might drift. + +## Checkpoint f16 drift is expected + +`CompactRecorder` = `HalfPrecisionSettings`, so weights and Adam moments are stored +as f16. A resumed trajectory drifts slowly from an uninterrupted one. This is not a +bug unless drift exceeds a few thousandths of NSE over many epochs. Resume *state* +(epoch, mini-batch cursor, rng, sampler permutation) is exact. diff --git a/.claude/skills/ddrs-dev/references/traps.md b/.claude/skills/ddrs-dev/references/traps.md new file mode 100644 index 0000000..0b31e94 --- /dev/null +++ b/.claude/skills/ddrs-dev/references/traps.md @@ -0,0 +1,212 @@ +# Traps + +Deduplicated from six retired skills. Each entry leads with the **discriminating +test** — the cheap check that tells you whether this is your problem — because that +is what a future session needs, not the narrative. + +## Symptom → trap + +| Symptom | Start at | +|---|---| +| Config change appears to do nothing; two arms byte-identical | T1, then "no `deny_unknown_fields`" in `config.md` | +| `compare_ddr_sandbox` reports a mismatch | T2 | +| Loss is finite but suspiciously constant; eval all-NaN | T3 | +| Baseline NSE is far worse than the trained model | T4 | +| Trained loss is flat across epochs | T5 | +| Eval phase hangs at ~900% CPU producing nothing | T6 | +| Long CPU forward dies with no error in its own log | T7 | +| Mid-eval `object not found` from icechunk | T8 | +| `.ddrs/` appeared somewhere unexpected | T9 | +| Resumed run trains zero batches | T10 | + +--- + +## T1 — Stale binary + +`~/.cargo/bin/ddrs` is not updated by `cargo build` or `cargo run`. The run manifest +stamps `git.sha` from `.git` at runtime, so the manifest looks correct while a +weeks-old binary executed. + +**Discriminating tests** +- Checkpoints must be **directories** (`epoch_E_mb_M/{head,optim,state}`). Flat + `epoch_E_mb_M.mpk` ⇒ pre-checkpoint-resume binary. +- `stat ~/.cargo/bin/ddrs` — compare mtime to your last `src/` edit. +- Head-file size, measured: ~103,459 B no-disagg · 107,178 B disagg · + 107,320 B disagg + leakance. + +**Fix**: `cargo install --path .` (canonical), or +`cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs`, or +bypass with `cargo run --release --bin ddrs -- run …`. + +**The incident that makes this stick (2026-07-01).** The installed `ddrs` was dated +2026-06-03 — before disaggregation (2026-06-19) and leakance (2026-06-29). The +hourly-forcing cell silently ran flat repeat-24 with no leakance. hourly-ON and +daily-ON produced **byte-identical** eval predictions. The manifest showed +`git.sha = 2cdd341` (the correct HEAD), masking it completely. A second occurrence +came from running `cargo install --path .` in the *main tree* while working in a +worktree, silently replacing the worktree build. + +## T2 — DDR sandbox mismatch + +**Triage order:** +1. `mkdir -p output && cargo run --release --example compare_ddr_sandbox`, then also + with `DDRS_FORCE_GRAPHS=1`. (Note: that env var only selects the CUDA *backend* — + it does **not** enable graph capture, which additionally requires + `use_cuda_graphs && sparse_solver == Cuda`, and the sandbox config sets neither.) +2. Inspect `output/ddrs_vs_ddr.csv`. **One bad reach ⇒ geometry/parameters. + Globally bad ⇒ solver/kernel.** +3. `grep -rn "f64\|bf16\|cast\|to_dtype" src/routing/ src/geometry.rs src/sparse/` + — invariant 2 says the core is f32 throughout. +4. `cargo test --test sparse_gradcheck`. **Also failing ⇒ the algorithm changed. + V1-only ⇒ kernel ordering or arithmetic fusion.** + +**Wrong-reference failure mode:** a fixture regenerated from a clean DDR clone +diverges ~1% (max abs ≈ 0.55 m³/s) at *every* ddrs commit. Only the desktop's +`~/projects/ddr` working tree is valid — see `build-and-env.md`. + +## T3 — CUDA graphs mask NaN + +`use_cuda_graphs: true` captures a kernel graph on the first forward and replays it. +If the first forward produced a NaN, the replay returns stale finite values instead +of propagating it — a constant, plausible-looking loss over a corrupt computation. + +**Discriminating test:** rerun with `use_cuda_graphs: false`. If the loss now goes +NaN, this was it. + +**Origin:** AORC precip carries ~14% genuine NaN (ocean / no coverage) → `log1p` → +NaN softmax → all 2,365 gauges NaN at eval while training loss looked healthy. Fixed +in `a5972d9` (zero-fill sanitization). + +**Rule:** validate every new data path with graphs off before enabling them. +`use_leakance: true` + `use_cuda_graphs: true` is rejected at config load. + +## T4 — Phantom-zero baseline (fixed 2026-07-28/29) + +Single-divide (zero-edge) gauges have no edge endpoints, so `upstream_comids()` +returned an empty set and the NaN-filtered sum produced **exactly 0.0** predictions +scored against real observations. 513 of 3,211 gauges (16%) at median NSE −0.305, +dragging the baseline median from 0.315 to 0.142. + +**Fix**: `valid_gauges()` skips them via `GageSubgraph::is_headwater()`, matching +training's own filter. The cache key now carries `BASELINE_ALGO_VERSION = +"baseline-algo-v2"`, hashed first. + +**Diagnostic**: count all-zero rows in `/baselines//predictions.f32`. +Any nonzero count on a post-fix binary is a new bug. + +**Generalized rule — the reusable part:** *the baseline must score the same gauge +population the model trains and evaluates on.* Skip filtered gauges; never impute. +This also means the "own baseline" columns in the 2026-07-07 and 2026-07-16 findings +docs are population-mismatched (3,211-gauge baseline vs 2,365-gauge trained medians) +— see `research-status.md`. + +## T5 — Flat training loss + +**When training loss is flat, fix the gradient path — do not try new loss +functions.** Two separate loss experiments (L1 vs NNSE-KGE; α-weighted KGE with a +learnable Muskingum X) both produced flat loss. Learnable X sat at its init (median +0.246, p10–p90 0.214–0.253, 0% at either bound despite α-weight 2), i.e. +∂loss/∂(routing params) ≈ 0. + +**Root cause:** daily `repeat-24` upsampling plus daily-mean aggregation puts +routing's within-day effect in the gradient **null space**. The disaggregation head +was the only change that ever made loss descend (1.224 → 1.02) and X move +(0.246 → 0.217). + +**Corollary:** fixing an optimization barrier can expose a structural ceiling. +Bare (precip-free) disagg unsticks the gradient but overfits (held-out KGE 0.624); +real precip timing is what rescues KGE. Once the gradient flows, `nnse-kge` does +*not* fix the KGE gap (0.7100 vs L1's 0.7106) and a temperature channel does not earn +its keep (0.7155/0.7088 vs 0.7152/0.7106). + +## T6 — GPU eval-phase OOM that never propagates + +`--backend cuda` `train-and-test` trains fine, then Phase 2 panics inside a +cubecl-cuda worker thread: `thread 'DSD-0-0' panicked … can't allocate buffer of +size: 4178264064`. **The panic does not propagate.** The main thread loops all +chunks at ~900% CPU writing a half-written GPU buffer. + +Happens **solo on the GPU** — Phase 1 alone leaves too little headroom for Phase 2's +~4.2 GB per-chunk buffer at `testing.batch_size: 15` on a 16 GB card. Confirmed +twice independently (2026-07-16). + +**Detection landed 2026-07-16** in `src/training/eval.rs`: two layered detectors +returning `DataError::CorruptedEvalChunk` — `corrupted_chunk_reason` (all-zero / +non-finite) and a process-global panic hook (`WORKER_PANICKED` / +`ensure_panic_hook_installed`) checked after every chunk. The first detector **alone +missed incident #2** — the stale buffer looked plausible. Gate: +`cargo test --lib training::eval::tests`. + +**Recovery without retraining:** +```bash +~/.cargo/bin/eval --config /config.yaml \ + --checkpoint /checkpoints/epoch_E_mb_M \ + --output /eval/predictions.zarr --backend cpu +``` + +**Operational rule:** do not co-schedule two `--backend cuda` trainings. Phase 1 +alone is ~9.9 GB on a 16 GB card; the second OOMs within seconds. Give each arm a +different backend from the start. + +## T7 — Silent kernel OOM on long CPU forwards + +`probe_zeta_gradient --mode teacher` at the default `--chunk-days 365` peaks ~65 GB +RSS on the 64,892-reach network and gets kernel-OOM-killed with **nothing in its own +log** — output just ends after `teacher: N plants, …`. + +**Discriminating test:** `journalctl -k | grep -i 'oom\|killed process'`. +RSS climbing *within* a chunk and collapsing to ~4 GB at boundaries is the signature. + +**Fix:** `--chunk-days 180` (~45 GB), at the cost of doubling disagg boundary +artifacts (0.55% → 1.1% of days). State continuity is exact either way. +**Caveat:** `run_state_cache` still hardcodes 365 — chunk lengths must match if you +pair a state cache with a teacher run. + +## T8 — Transient icechunk read, not a data hole + +A mid-eval `object not found` after hundreds of clean chunks is transient. These Q′ +stores are **divide-major** (`Qr` chunk = 200 divides × ALL 14,976 days), so any +time-slice read touches every chunk object — chunk 364 cannot be missing an object +that chunks 1–363 already read. + +**Confirm:** open the store directly in Python (`icechunk.Repository.open`) and run +`ddrs import --dry-run`. A clean read confirms transient. + +**Recovery:** training and eval are separate phases, so the final checkpoint stays +valid. Treat "nonzero exit but final checkpoint exists" as continue-with-warning. + +## T9 — `.ddrs/` beside the config + +`Workspace::beside(config)` is the default, so `--config config/experiments/x.yaml` +creates `config/experiments/.ddrs/`. `--workspace` takes the `.ddrs` **directory +itself**, not its parent. Always pass +`--workspace /home/tbindas/projects/ddrs/.ddrs`. + +## T10 — `--checkpoint` means different things per binary + +| Binary | Argument | +|---|---| +| `eval`, `probe_zeta_gradient` | the **directory** `epoch_E_mb_M` | +| `dump_parameters` | the **head base** `epoch_E_mb_M/head` (no `.mpk`) | + +Resume additionally requires `experiment.epochs > E` or the resumed run trains zero +batches. Stored weights are f16, so a resumed trajectory drifts slowly — expected, +not a bug. + +## Exit codes + +`src/cli/types.rs`: 0 Success · 1 Generic · 2 ConfigInvalid · 3 DataSourceMissing · +4 LockDrift · 5 RuntimeFailure · 6 WorkspaceNotInitialized. + +`ddrs run --strict` aborts with 4 *before* relocking, preserving the drift evidence; +plain `plan` reports drift then refreshes the lock. + +## Pre-flight before a long run + +1. `cargo install --path .` — refresh the binary. +2. `mkdir -p output && cargo run --release --example compare_ddr_sandbox` — V1 gate. +3. Confirm `--workspace` points at the repo-root `.ddrs`. +4. Confirm `--backend` matches `params.sparse_solver` intent. +5. Smoke it: `ddrs run --workflow train --max-mini-batches 2`. +6. Grep the smoke log for `streamflow resolution:` and the gauge-filter line. +7. Check no other CUDA job is resident if you are using `--backend cuda`. diff --git a/.claude/skills/ddrs-diagnostics-and-tooling/SKILL.md b/.claude/skills/ddrs-diagnostics-and-tooling/SKILL.md deleted file mode 100644 index 8640b60..0000000 --- a/.claude/skills/ddrs-diagnostics-and-tooling/SKILL.md +++ /dev/null @@ -1,582 +0,0 @@ ---- -name: ddrs-diagnostics-and-tooling -description: > - Use when you need to MEASURE a ddrs result rather than eyeball it — diagnosing - a failed run, verifying gradient health, interpreting leakance GO/NO-GO gates, - checking the summed-Q' baseline, probing identifiability, or deciding whether - a hypothesis is SUPPORTED/REFUTED/INCONCLUSIVE. Triggers: "why is zeta small", - "how do I tell if training is working", "what does K_D ceiling mean", - "compare two runs", "is my gradient alive", "how do I reproduce the 2x2 - verdict". Do NOT use for architecture changes, config authoring, or writing - new training code — use ddrs-change-control or ddrs-architecture-contract - instead. ---- - -# ddrs Diagnostics and Tooling - -## Glossary (jargon defined once) - -| Term | Definition | -|---|---| -| **ddrs** | BURN-0.21 Rust port of the DDR differentiable Muskingum-Cunge solver | -| **BURN** | Rust deep-learning framework (analogous to PyTorch); autograd tapes differ | -| **KAN head** | `rskan::KanLayer` network (`Linear→KanLayer×N→Linear→Sigmoid`); maps catchment attributes to per-reach routing parameters | -| **Q'** (Q-prime) | Upstream-summed divide streamflow forcing from a pre-computed DHBv2 retrospective | -| **zeta (ζ)** | Per-reach GW–SW exchange flux (m³/s): `leakance_factor · area_z · K_D · (depth − d_gw)`. Positive = losing reach | -| **K_D** | Hydraulic exchange rate (1/s); log-space parameter in `[1e-8, 1e-6]` by default | -| **Muskingum-Cunge (MC)** | Linear flood-routing method; ddrs solves a CSR lower-triangular system per timestep | -| **eval network** | Gauge-subgraph union used during evaluation (64,892 reaches for CONUS) | -| **NSE / KGE** | Nash-Sutcliffe Efficiency / Kling-Gupta Efficiency; standard hydrology skill scores | -| **NNSE** | Normalized NSE: `NSE/(2-NSE)`, range [0,1], avoids -∞ floor | -| **summed-Q' baseline** | Upper bound with no routing: sum of upstream Q' at each gauge; median NSE 0.689 / KGE 0.723 (CONUS, as of 2026-07-05) | -| **2×2** | Leakance ON/OFF × hourly/daily forcing factorial experiment | -| **rho-window** | Training sub-sequence length (default 90 days); sampled from the full training period | -| **hotstart transient** | Initial-condition mismatch at window start; big rivers carry memory >> warmup days | -| **CsrSolveOp** | Hand-written BURN autograd backward for the sparse triangular solve (invariant 4) | -| **dump_parameters** | Binary/CLI command that exports full-CONUS KAN outputs to `kan_parameters.nc` | -| **run-id** | `-[-]` directory name under `.ddrs/runs/` | - ---- - -## When NOT to use this skill - -- **Changing `src/routing/`, `src/sparse.rs`, or `src/geometry.rs`** — use ddrs-change-control (blast-radius analysis required) -- **KAN head architecture changes** — use ddrs-architecture-contract (invariants 5-6) -- **Writing new Python analysis scripts** — use ddrs-proof-and-analysis-toolkit -- **Interpreting the research roadmap / phase gating** — use ddrs-identifiability-campaign or ddrs-research-frontier - ---- - -## Part 1: Non-negotiable regression gates - -Run these before and after ANY change to `src/routing/`, `src/geometry.rs`, or `src/sparse.rs`. - -### 1.1 DDR parity gate (invariant 1) - -```bash -cargo run --release --example compare_ddr_sandbox -``` - -**Pass:** prints `ABSOLUTE MATCH` — max abs diff < 1e-3 m³/s on the 5-reach RAPID sandbox. -**Fail:** any diff >= 1e-3 m³/s means the port broke. Do NOT merge. - -**Caveat (as of 2026-06-06):** the reference fixture requires the desktop's `~/projects/ddr` with the unpushed `geometry/trapezoidal.py` changes. A clean DDR clone will diverge ~1%. That is a wrong-reference failure, not a port failure. See `.claude/references/ddrs-comparing-to-ddr.md` §Regenerating fixtures before concluding a real regression. - -### 1.2 Leakance gradient-exactness gates - -Run whenever `src/routing/leakance.rs` or its backward op changes: - -```bash -cargo test --test leakance_gradcheck # analytical grad ≈ finite-difference (8 cases) -cargo test --test leakance_off_parity # byte-identical to no-leakance when off (3 cases) -cargo test --test zeta_accum # accumulated zeta == b_rhs delta (6 cases) -cargo run --release --example compare_ddr_sandbox # still ABSOLUTE MATCH -``` - -**Interpretation:** `leakance_gradcheck` failing means the analytical backward diverges from finite-diff; this breaks training correctness. `leakance_off_parity` failing means leakance bleeds into non-leakance paths. - -### 1.3 KAN head parity gates - -Run when `src/nn/kan_head.rs`, `Cargo.toml` rskan pin, or DDR's `nn/kan.py` changes: - -```bash -cargo test --features fixtures \ - --test kan_head_init_repro \ - --test kan_head_init_parity \ - --test kan_head_fixture_forward \ - --test kan_head_fixture_backward -``` - -### 1.4 Sparse gradcheck - -```bash -cargo test --test sparse_gradcheck -``` - -Verifies the CSR backward (invariant 4) is gradient-exact. - ---- - -## Part 2: Stale binary trap (the most common failure mode) - -`~/.cargo/bin/ddrs` is installed once by `cargo install`. `cargo build` does NOT update it. - -**Symptom:** run looks fine, metrics make no sense, or a new feature is silently missing. - -**Check whether you have the right binary:** - -```bash -# Current checkpoints are DIRECTORIES: -ls .ddrs/runs//checkpoints/ -# Should show: epoch_5_mb_9/ (a directory) -# If you see: epoch_5_mb_9.mpk (a flat file) → stale binary -``` - -**Fix:** - -```bash -cargo install --path . # canonical, ~2 min -# or faster: -cargo build --release --bin ddrs && cp target/release/ddrs ~/.cargo/bin/ddrs -# or bypass installed binary entirely: -cargo run --release --bin ddrs -- run --workflow train-and-test -``` - -The stale-binary trap caused the 2026-07-01 2×2 to produce byte-identical hourly and daily cells (the installed binary predated the disaggregation feature). - ---- - -## Part 3: CUDA graphs masking NaN - -**Symptom:** training loss is finite and slowly decreasing, but intermediate checks show NaN activations. - -**Cause:** `use_cuda_graphs: true` replays a captured graph; a NaN in a subsequent forward returns stale (pre-NaN) finite values. The loss looks healthy while the model is broken. - -**Diagnosis:** - -```bash -# In ddrs.yaml, temporarily set: -use_cuda_graphs: false -# Then re-run one mini-batch and inspect: -# If loss is NaN → confirmed NaN forward; debug with use_cuda_graphs: false -# If loss is fine → not a NaN issue -``` - -**Note:** `use_leakance: true` combined with `use_cuda_graphs: true` is rejected at config load time with a hard error. - ---- - -## Part 4: Run inspection and workspace navigation - -### 4.1 Check run status and disk - -```bash -ddrs status # workspace summary + disk usage by run -ddrs show # full manifest: config, sources, git SHA, metrics -``` - -### 4.2 Read a run's log - -```bash -cat .ddrs/runs//run.log # timestamped stdout+stderr (fd-level capture) -``` - -Useful patterns in the log to check: - -| What to grep | Meaning | -|---|---| -| `streamflow resolution: Daily\|Hourly` | Confirms whether icechunk store was read as daily or hourly | -| `warm start: loaded KAN head` | Checkpoint resume loaded correctly | -| `no …/optim.mpk` | Adam starts cold (expected for head-only warm-start) | -| `ABSOLUTE MATCH` | Sandbox regression passed during this run | -| `precip loading` | AORC precip store opened (needed for disaggregation) | - -### 4.3 Inspect a run's config - -```bash -cat .ddrs/runs//config.yaml # exact config that produced this run -``` - -### 4.4 Compare two runs' metrics - -```bash -ddrs show | grep -E "nse|kge|loss" -ddrs show | grep -E "nse|kge|loss" -``` - ---- - -## Part 5: Summed-Q' baseline - -**What it is:** per-gauge sum of upstream Q' with NO routing or learning. It is the ceiling that trained routing must beat. - -**Reference numbers (CONUS, as of 2026-07-05):** -- Median NSE: 0.689 -- Median KGE: 0.723 - -**Best trained result (precip-driven disaggregation + L1, 2365 gauges, as of 2026-06-23):** -- Median NSE: 0.715 (+0.037 vs baseline — beats it) -- Median KGE: 0.711 (-0.012 vs baseline — does NOT beat it) - -**Critical:** KGE does NOT beat the summed-Q' baseline in any config as of 2026-07-05. NSE beats it with precip disaggregation. This is a known open problem (over-attenuation; L1 and NSE reward low variance). - -**Reproduce baseline:** - -```bash -ddrs plan # computes and caches baseline automatically on first run -# or read the cached version: -cat .ddrs/baselines//manifest.json # shows metrics, provenance -``` - -**Interpretation:** if your trained run's median NSE does NOT beat 0.689, the routing term earns nothing. Debug training loss curves and KAN head gradient stats before touching the sparse solver. - ---- - -## Part 6: dump_parameters — export learned KAN outputs to NetCDF - -```bash -# Via the legacy eval binary (required for leakance zeta export): -cargo build --release --bin eval -target/release/eval \ - --config config/experiments/leakance_hourly_on.yaml \ - --checkpoint .ddrs/runs//checkpoints/epoch_5_mb_9 \ - --output /tmp/eval.zarr \ - --zeta-output .ddrs/runs//kan_parameters.nc - -# Or (no zeta, just KAN params): -cargo build --release --bin dump_parameters -target/release/dump_parameters \ - --config ddrs.yaml \ - --checkpoint .ddrs/runs//checkpoints/epoch_5_mb_9/head \ - --output .ddrs/runs//plot/kan_parameters.nc -``` - -**Output file layout (`kan_parameters.nc`):** - -| Variable | Dimension | Unit | Notes | -|---|---|---|---| -| `COMID` | `(COMID,)` | — | Reach IDs for full-CONUS params | -| `n` | `(COMID,)` | — | Manning's roughness, denormalized | -| `q_spatial` | `(COMID,)` | — | Channel geometry exponent | -| `x_storage` | `(COMID,)` | — | Muskingum X (storage weighting) | -| `K_D` | `(COMID,)` | 1/s | Hydraulic exchange rate (leakance only) | -| `d_gw` | `(COMID,)` | m | GW depth threshold (leakance only) | -| `leakance_factor` | `(COMID,)` | — | Scale factor (leakance only) | -| `COMID_eval` | `(COMID_eval,)` | — | Eval-network reach IDs (leakance only) | -| `zeta` | `(COMID_eval,)` | m³/s | Mean \|zeta\| over eval window | -| `zeta_net` | `(COMID_eval,)` | m³/s | Signed mean; positive = losing reach | -| `depth_mean` | `(COMID_eval,)` | m | Eval-window mean routed depth | -| `area_z_mean` | `(COMID_eval,)` | m² | Eval-window mean plan-view wetted area | -| `q_mean` | `(COMID_eval,)` | m³/s | Eval-window mean routed discharge | - -**Load in Python:** - -```python -import xarray as xr -ds = xr.open_dataset(".ddrs/runs//kan_parameters.nc") -# for a quick K_D ceiling check: -kd = ds["K_D"].values -import numpy as np -print(f"K_D: min={kd.min():.2e} median={np.median(kd):.2e} max={kd.max():.2e}") -print(f"fraction at ceiling (1e-6): {(kd > 9.9e-7).mean():.1%}") -``` - ---- - -## Part 7: Leakance GO/NO-GO evaluation - -### 7.1 The three gate criteria (per spec) - -| Gate | Threshold | Interpretation | -|---|---|---| -| 1 | ΔNSE or ΔKGE > 0 (median) on losing-stream subset, hourly arm | Leakance improves skill where physics predicts it should | -| 2 | Effect absent or weaker in daily arm | Rules out fudge-factor behavior | -| 3 | \|zeta\| > 0.01 m³/s on ≥ 10% of eval reaches | Learned exchange is non-trivially active | - -**Current status (as of 2026-07-01):** GO — but marginal (10.4% vs 10% threshold for gate 3, no headroom). - -### 7.2 Running the full verdict script - -```bash -cd ~/projects/ddr -uv run python ~/projects/ddrs/scripts/leakance_subset_analysis.py \ - --hourly-on 2026-07-01T13-43-32Z-train-and-test \ - --daily-on 2026-07-01T21-20-27Z-train-and-test \ - --hourly-off 2026-06-23T02-49-12Z-conus-hourly-train-and-test \ - --daily-off 2026-06-05T01-41-16Z-train-and-test \ - --ddrs-runs-dir /home/tbindas/projects/ddrs/.ddrs/runs -``` - -**Prerequisites:** both ON arms must have `kan_parameters.nc` with the `zeta`/`COMID_eval` variables (produced by `--zeta-output` or `train-and-test` Phase 2). - -**Output block to look for:** - -``` -VERDICT: GO - Leakance improves skill on the losing-stream subset under hourly forcing ... -``` - -or `VERDICT: NO-GO` with reasons, or `VERDICT: NEEDS_ZETA_EXPORT`. - -**Losing-stream subset definition:** gauges where the summed-Q' baseline mean(pred)/mean(obs) > 1 on the hourly-OFF run. CONUS result: 1883/2365 gauges (79.6%). - -### 7.3 Interpreting leakance parameter outputs - -| Observation | Interpretation | -|---|---| -| `K_D` 100% at ceiling (1e-6) | Optimizer wants MORE exchange; box is binding. NOT a model failure (H1 REFUTED, as of 2026-07-02) | -| `leakance_factor` interior (≈0.33) | Gate is open; reaches are actively exchanging | -| `d_gw` near mean depth | Driving head throttled; ~47% of reaches gaining at eval-window mean | -| zeta–uparea Spearman +0.76 | Exchange tracks river size, not aridity — gauge bias, not starvation | -| dry-tercile zeta < wet-tercile | Inverse of physics; training signal concentrates near large gauged rivers | - ---- - -## Part 8: Seven-hypothesis diagnosis battery (leakance low-zeta) - -**When to run:** after any leakance experiment returns small zeta (median |zeta| < 0.01 m³/s). - -**Prereqs:** the ON run's `kan_parameters.nc` must contain `depth_mean`, `area_z_mean`, `q_mean` on `COMID_eval`. Requires the re-eval pass with the current binary. - -```bash -cd ~/projects/ddr -uv run python ~/projects/ddrs/scripts/leakance_diagnosis.py -# uses hardcoded run IDs in ARM_IDS dict; edit if using different runs -``` - -**Hypothesis reference table (results as of 2026-07-02):** - -| # | Hypothesis | Verdict (2026-07-02) | Key number | -|---|---|---|---| -| H1 | K_D box clips zeta below detectability | REFUTED | 71.5% of reaches CAN exceed 0.01 m³/s in-box; median utilization 3.4% | -| H2 | d_gw near depth → driving head ≈ 0 | SUPPORTED | 57.6% of reaches < 0.1 m mean driving head; 47.0% ≤ 0 | -| H3 | KAN variance collapse (original hypothesis) | REFUTED | K_D–aridity ρ = +0.61; d_gw–meanP ρ = +0.71 — strong spatial structure | -| H4 | Gauge bias / gradient starvation | SUPPORTED | gauged median |zeta| 6.7e-3 vs ungauged 5.9e-4 (11×); dry/wet ratio 0.40 (inverse of physics) | -| H5 | Equifinality with n/x_storage | SUPPORTED (daily only) | daily Δn = +0.012 (0.59 IQR); hourly Δn nil (0.05 IQR) | -| H6 | Wrong yardstick (absolute 0.01 bar) | REFUTED | 8.4% >1% fractional loss agrees with absolute bar | -| H7 | d_gw boundary pinning (disconnected regime) | REFUTED | 0.0% of d_gw at bounds | - -**Diagnosis conclusion:** zeta is small because the optimizer throttles through the driving head (H2) and the gradient only reaches gauged large rivers (H4), not because the K_D box or KAN architecture fails. Widening K_D past 1e-6 is NOT recommended (supersedes the "top follow-up" in `docs/2026-07-01-leakance-hourly-findings.md`). - ---- - -## Part 9: Gradient probe (adjoint reachability + detectability) - -**Location:** `origin/worktree-zeta-sensitivity` branch. -**When to run:** when you want to know whether the leakance gradient is alive at a reach, or whether a real-magnitude loss would be detectable at a downstream gauge. - -### 9.1 Stage 1 — adjoint reachability map - -```bash -# Trained checkpoint (use worktree binary): -WT=/home/tbindas/projects/ddrs/.claude/worktrees/zeta-sensitivity -nice -n 10 $WT/target/release/probe_zeta_gradient \ - --config config/experiments/leakance_hourly_on.yaml \ - --checkpoint .ddrs/runs/2026-07-01T13-43-32Z-train-and-test/checkpoints/epoch_5_mb_9 \ - --windows 96 --seed 42 \ - --output output/zeta_probe/grad_trained.nc - -# Cold head (omit --checkpoint): -nice -n 10 $WT/target/release/probe_zeta_gradient \ - --config config/experiments/leakance_hourly_on.yaml \ - --windows 96 --seed 42 \ - --output output/zeta_probe/grad_cold.nc -``` - -**Output:** per-reach `|∂L/∂factor|`, `∂L/∂factor`, coverage count in NetCDF. - -**Interpretation thresholds:** - -| Ratio (gauged/ungauged |g|) | Interpretation | -|---|---| -| ≥ 10× at both trained and cold points | SUPPORTED starvation — auxiliary supervision fills genuine gap | -| < 10× | REFUTED starvation — gradient reaches everywhere | - -**Measured result (2026-07-03, as of 2026-07-05):** gauged/ungauged ratio = 1.5× (trained), 2.9× (cold). P1 starvation REFUTED. The gradient is alive everywhere. - -### 9.2 Stage 2 — planted-delta detectability - -```bash -# Plan sites first (ddrs-py venv): -cd ddrs-py && uv run python ../scripts/zeta_probe_sites.py - -# Perturb pass: -nice -n 10 $WT/target/release/probe_zeta_gradient \ - --config config/experiments/leakance_hourly_on.yaml \ - --checkpoint .ddrs/runs/2026-07-01T13-43-32Z-train-and-test/checkpoints/epoch_5_mb_9 \ - --mode perturb \ - --probe-plan output/zeta_probe/probe_plan.csv \ - --eval-days 1095 \ - --output output/zeta_probe/perturb -``` - -**Detectability criterion:** `|mean ΔQ| > 99th-pct noise floor AND > 5% of gauge's mean flow`. - -**Measured result (2026-07-03, as of 2026-07-05):** - -| Delta | Reference (Ref) gauges | Non-reference | -|---|---|---| -| 0.01 m³/s (literature-magnitude) | 4.2% detectable | 0.0% | -| 0.1 m³/s (upper-literature) | 16.7% detectable | 2.1% | - -P3 detectability: NO-GO. The planted loss arrives at gauges at ~95% fidelity (transmission fine) but is 53× smaller than the median Ref gauge's 5% discharge-uncertainty band. Detection fails on dilution, not transmission. No gauge-only objective can learn real-world leakance. - ---- - -## Part 10: Synthetic recoverability control (Phase B) - -**Location:** `origin/worktree-zeta-sensitivity` branch. -**Status (as of 2026-07-05):** FAILED — positive control not passed. - -**One-line result:** recovery ratio median 0.009 (bar: ≥ 0.5). Root cause: the windowed training objective (rho-90, warmup-5) has a ~130× hotstart-transient noise floor relative to the planted signal. The signal is invisible even with zero observation noise and warm-started weights. - -**Phase B objective:** reduce the noise floor to ≤ 0.25 mean L1 (≤ 10% of a converged run). This is NOT YET MET as of 2026-07-05. Required before any identifiability claim for leakance. - -**Key decomposition:** - -| Quantity | Value | -|---|---| -| Planted signal (continuous residual, teacher weights + teacher obs) | 0.0076 mean L1 | -| Step-0 windowed training loss (warm-started run A) | 1.017 mean L1 | -| Noise floor / signal ratio | ~130× | -| Run A continuous residual after 5 epochs of training | 0.4431 (58× worse than start) | - -**Implication for leakance identifiability:** gauge-loss training cannot reward reach-scale leakance even with: detectable gauge signal (constructed), zero obs noise, expressible head, and warm-start from the answer. Leakance identifiability is NOT proven. Phase B (state-cache hotstart, ≤ 0.25 mean L1 target) is required before any identifiability claim. - -**Verdicts from the control run:** - -| # | Metric | Measured | Verdict | -|---|---|---|---| -| R1 | Recovery ratio median | 0.009 | FAILED (bar: ≥ 0.5) | -| R2 | Non-planted \|zeta_net\| A/baseline | 1.11 | PRECISE — trivially, nothing moved | -| R3 | Final-epoch loss A vs B (42.2% gap) | A < B | CONFOUNDED — B's handicap accounts for gap | -| R4 | Manning's n shift (run B) | Δn = −0.019 (global, not localized) | H5 equifinality confirmed at global scale | -| R5 | Cold emergence ratio | 1.20 | SUPPRESSED (bar: > 3) | - ---- - -## Part 11: Leakance configuration checklist - -Three config changes are ALL required together to enable leakance. Missing any one causes silent failure or a config-load error. - -```yaml -# 1. Activate the term (also disables CUDA graphs): -params: - use_leakance: true - -# 2. Tell the KAN head to emit leakance parameters: -kan_head: - learnable_parameters: - - K_D - - d_gw - - leakance_factor - - n # keep existing routing params - - q_spatial - # x_storage # optional - -# 3. Set parameter ranges: -params: - parameter_ranges: - K_D: [1.0e-8, 1.0e-6] # log-space; 1/s - d_gw: [-2.0, 2.0] # m - leakance_factor: [0.0, 1.0] # dimensionless -``` - -**Important:** `use_leakance: true` AND `use_cuda_graphs: true` is rejected at config load. The combination is not supported without a separate capture path. - ---- - -## Part 12: Training monitoring checklist - -Use this list when a run produces unexpected metrics. - -- [ ] **Check the binary is current** — flat checkpoint files mean stale binary (Part 2) -- [ ] **Check `streamflow resolution` in `run.log`** — `Daily` vs `Hourly` must match your intent -- [ ] **Check precip loaded** — if `use_precip: true` in disagg block, grep `precip loading` in run.log -- [ ] **Disable CUDA graphs if loss is suspiciously smooth** — guards against NaN masking (Part 3) -- [ ] **Check `kan_parameters.nc` K_D ceiling fraction** — 100% at ceiling means the box is binding -- [ ] **Run `leakance_subset_analysis.py`** for GO/NO-GO after any leakance experiment (Part 7.2) -- [ ] **Compare against baseline** — trained NSE should exceed 0.689; KGE may not (Part 5) -- [ ] **Check `ddrs show `** for final metrics in the manifest -- [ ] **Verify checkpoint format** — directory `epoch_E_mb_M/` with `head.mpk`, `optim.mpk`, `state.json` - ---- - -## Part 13: Run ID and workspace layout quick reference - -``` -.ddrs/ - system.json # GPU/driver probe result - sources.lock # fingerprints of data_sources paths - adjacency// # managed CONUS + gauge adjacency (content-addressed) - baselines// # summed-Q' baseline cache - manifest.json # metrics + gage_ids - predictions.f32 # row-major [n_gauges, n_days] - observations.f32 - runs// - manifest.json # config + sources + git SHA + output metrics - config.yaml # exact config snapshot - run.log # timestamped stdout+stderr - checkpoints/ - epoch_E_mb_M/ # DIRECTORY (flat .mpk = stale binary) - head.mpk - optim.mpk - state.json - eval/ - predictions.zarr/ # zarr-v3 group - predictions/ # float64 [n_gauges, n_days] - observations/ - gage_ids/ # uint8 [n_gauges, 8] fixed-width ASCII - baseline/ # copy of .ddrs/baselines// - kan_parameters.nc # KAN outputs + zeta (leakance) or plot params - plot/ - kan_parameters.nc # full-CONUS dump_parameters output -``` - ---- - -## Part 14: Zeta accumulator — what it measures and how to verify - -The zeta accumulator is enabled automatically during eval when `use_leakance: true`. It recomputes per-step zeta from the SAME saved primitives the backward reads, then accumulates per-reach means over the eval window. - -**Correctness identity (from `tests/zeta_accum.rs`):** - -``` -q_no_leak[0] − q_leak[0] == zeta[0] (headwater reach; exact equality) -``` - -**Verify the identity is preserved:** - -```bash -cargo test --test zeta_accum -``` - -**Check zeta is non-trivial after a leakance run:** - -```python -import xarray as xr, numpy as np -ds = xr.open_dataset(".ddrs/runs//kan_parameters.nc") -z = np.abs(ds["zeta"].values) -print(f"median |zeta|: {np.median(z):.4e} m3/s") -print(f"|zeta| > 0.01 on {np.mean(z > 0.01):.1%} of eval reaches") -# CONUS target (as of 2026-07-01): 10.4% of 64,892 reaches -``` - -**Zeta dimensionality:** `COMID_eval` dimension = gauge-subgraph union = eval network (64,892 reaches for CONUS). NOT the full 346,321-reach CONUS. - ---- - -## Provenance and maintenance - -```bash -# Re-verify Part 1 regression gates: -cargo run --release --example compare_ddr_sandbox # ABSOLUTE MATCH -cargo test --test leakance_gradcheck # 8/8 -cargo test --test zeta_accum # 6/6 -cargo test --test sparse_gradcheck # pass - -# Re-verify leakance 2x2 results: -cd ~/projects/ddr && uv run python \ - ~/projects/ddrs/scripts/leakance_subset_analysis.py \ - --hourly-on 2026-07-01T13-43-32Z-train-and-test \ - --daily-on 2026-07-01T21-20-27Z-train-and-test \ - --hourly-off 2026-06-23T02-49-12Z-conus-hourly-train-and-test \ - --daily-off 2026-06-05T01-41-16Z-train-and-test \ - --ddrs-runs-dir /home/tbindas/projects/ddrs/.ddrs/runs - -# Re-verify 7-hypothesis diagnosis: -cd ~/projects/ddr && uv run python \ - ~/projects/ddrs/scripts/leakance_diagnosis.py - -# Source files for this skill: -# /home/tbindas/projects/ddrs/CLAUDE.md -# /home/tbindas/projects/ddrs/scripts/leakance_subset_analysis.py -# /home/tbindas/projects/ddrs/scripts/leakance_diagnosis.py -# /home/tbindas/projects/ddrs/docs/2026-07-01-leakance-hourly-findings.md -# /home/tbindas/projects/ddrs/docs/2026-07-02-leakance-diagnosis-findings.md -# origin/worktree-zeta-sensitivity:docs/2026-07-03-zeta-gradient-probe-findings.md -# origin/worktree-zeta-sensitivity:docs/2026-07-04-synthetic-recoverability-findings.md -# Skill last verified: 2026-07-05 -# Volatile facts: summed-Q baseline metrics, leakance GO/NO-GO verdict, -# recoverability Phase B status — re-verify after any new experiment run -``` diff --git a/.claude/skills/ddrs-docs-and-writing/SKILL.md b/.claude/skills/ddrs-docs-and-writing/SKILL.md deleted file mode 100644 index d5df37a..0000000 --- a/.claude/skills/ddrs-docs-and-writing/SKILL.md +++ /dev/null @@ -1,366 +0,0 @@ ---- -name: ddrs-docs-and-writing -description: "Use when writing, updating, or reviewing any ddrs project document: findings reports, session handoffs, experiment specs, implementation plans, or the paper at ddr_equifinality/paper.tex. Also use when asking what the correct doc type is for a given output, how to connect an experiment result to the paper narrative, or whether a hypothesis verdict belongs in a findings doc or a spec. Do NOT use for running experiments, editing Rust code, debugging builds, or plotting — use ddrs-run-and-operate, ddrs-debugging-playbook, or ddrs-eval-plots instead." ---- - -# ddrs docs and writing - -This skill covers the documentation conventions, doc-type taxonomy, template structures, house style, and paper-to-experiment-log connection for the ddrs project. - -## Glossary (defined once) - -| Term | Meaning | -|---|---| -| **ddrs** | BURN-0.21 Rust port of the Python DDR (Differentiable Discharge Routing) solver using Muskingum-Cunge | -| **DDR** | Python/PyTorch reference at `~/projects/ddr/`. ddrs must stay gradient-exact against it | -| **KAN head** | Kolmogorov-Arnold Network head (`rskan::KanLayer`) that maps catchment attributes → routing parameters | -| **summed-Q′** | No-routing baseline: per-gauge sum of upstream divide Qr. Any trained model must beat this to prove routing earns its keep | -| **CONUS** | Contiguous United States; 346,321 MERIT reaches, 338,814 edges | -| **eval network** | The gauge-subgraph union used at evaluation time (64,892 reaches in the 2×2 experiments) | -| **zeta** | Per-reach GW–SW exchange flux (m³/s); positive = losing stream | -| **leakance** | Experimental GW–SW water-loss term; off by default; controlled by `use_leakance: true` in config | -| **disagg head** | Daily→hourly disaggregation sub-head inside KanHead; driven by AORC hourly precip | -| **spec** | Design doc written BEFORE an experiment. Lives in `docs/superpowers/specs/` | -| **plan** | Implementation task list derived from a spec. Lives in `docs/superpowers/plans/` | -| **findings doc** | Post-experiment narrative with hypothesis table, results, and verdict. Lives in `docs/` | -| **handoff** | Session-boundary doc with the action list for the next session. Lives in `docs/` | -| **journal** | Multi-experiment chronological record (used for cross-cutting investigations). Lives in `docs/` | - ---- - -## Doc-type taxonomy - -Choose the correct doc type before writing anything. - -| Situation | Doc type | Location | Naming pattern | -|---|---|---|---| -| Planning a new experiment before any code runs | **spec** | `docs/superpowers/specs/` | `YYYY-MM-DD--design.md` | -| Breaking a spec into implementation tasks | **plan** | `docs/superpowers/plans/` | `YYYY-MM-DD-.md` (same slug) | -| Summarizing what an experiment found after it ran | **findings doc** | `docs/` | `YYYY-MM-DD--findings.md` | -| Handing state to the next session mid-experiment | **handoff** | `docs/` | `YYYY-MM-DD--handoff.md` | -| Documenting a cross-cutting investigation that spans multiple runs | **journal** | `docs/` | `_journal.md` or a named findings doc | -| Documenting a data source, contract, or external API | **reference** | `docs/reference/` or `docs/` | descriptive name, no date required | - -### When NOT to write a new doc - -- If a findings doc for that experiment already exists: update it in place (add a datestamped section) rather than creating a duplicate. -- Handoffs and journals are living docs — append, do not replace. -- The spec/plan pair is mandatory before running expensive GPU jobs. Do not skip the spec to save time: the pre-registered hypotheses and falsification criteria protect against HARKing (Hypothesizing After Results are Known). - ---- - -## Findings doc template - -Every experiment that runs on real data gets a findings doc. Use this structure exactly. - -```markdown -# — findings () - -Spec: `docs/superpowers/specs/.md` -Plan: `docs/superpowers/plans/.md` (if applicable) -Script: `scripts/