Juniata single-catchment sample: self-contained bundle, train/test module, physics notebook - #193
Merged
Merged
Conversation
…sics notebook Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implement reindex_subgraph() to remap CONUS-positional indices into a compact 0..N-1 space, preserving topological ordering for CSR solve. Add synthetic-array tests validating edge remapping, physical array alignment, and lower-triangular structure preservation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the Juniata River (01567000) MERIT-Hydro subgraph bundle for self-contained training and testing without CONUS-scale data stores. Bundle: 213 reaches, 212 edges, gage_idx 212; icechunk stores start 1980-01-01; total size 8.9 MB. Adds bundle-contract test skeleton and updates .gitignore to commit the bundle directory while blocking any future statistics/ subdirectory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… baseline - examples/juniata/train_and_test.py: make_config / train / test / summed_qprime_baseline / CLI; mirrors scripts/train.py + test.py at batch_size=1 with no Hydra dependency. q_prime = flow(...) is the documented seam for a future differentiable runoff model. - tests/examples/test_juniata_bundle.py: appended TestSmoke (1-epoch train + 1-year test + baseline, all assertions green); fixed GeoDataset.get_dataset_class unbound-method bug in three TestBundleContract tests (mirror cfg.geodataset.get_dataset_class). - 30-epoch CPU run: routed NSE=0.784 KGE=0.877 vs baseline NSE=0.695 KGE=0.820 (test 1995/10/01–2010/09/30, gauge 01567000). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Guard against NaN observation days poisoning the gradient by masking with torch.isfinite() before l1_loss, rather than dropping whole gages as scripts/train.py does — safer for the single-gauge case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- make_notebook.py builder generates juniata_routing.ipynb (6 sections: basin structure, Muskingum-Cunge physics, network solve, differentiable chain, train/eval, road to end-to-end). Executes clean top-to-bottom via nbclient (epochs=1 fast mode). API correction: GeoDataset.get_dataset_class called as cfg.geodataset.get_dataset_class(cfg=cfg) (enum instance method); sys.path insert ensures examples package importable when CWD=examples/juniata. - README.md fills both reference numbers from 30-epoch CPU run: routed NSE 0.784 KGE 0.877; summed-q' baseline NSE 0.695 KGE 0.820. - pyproject.toml: extend *.ipynb per-file ruff ignores to include E702, D103, B905, I001 (semicolons, missing docstring, bare zip, import sort — all standard notebook style used by the brief's cells). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…axis, zarr attrs
- De-semicolonize cell sources in make_notebook.py so ruff format passes on the
generated notebook; remove E702 from *.ipynb per-file-ignores in pyproject.toml
- Add missing ## 1. The basin and ## 5. Train & evaluate section headings
- Fix obs hydrograph x-axis: use batch_daily_time_range instead of daily_time_range
- Set root attrs {format, shape, geodataset} on juniata_conus_adjacency.zarr so
Merit all-segments mode can read conus_adjacency.attrs["shape"] without KeyError;
update extract_bundle.py to write these attrs at extraction time
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same CI-vs-local drift as 3b18cfd: CI resolves ruff 0.16.3, which formats Python code fences inside markdown; the new spec and plan docs carried aligned inline comments it rewrites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Merged
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A self-contained sample under
examples/juniata/for teams learning how DDR works on a single catchment: the Juniata River at Newport, PA (USGS 01567000, 8,657 km², 213 MERIT reaches — the dMC-Juniata heritage basin, in the 5–10k km² band where routing physically acts). No HPC, S3, or external store access required.Spec:
docs/superpowers/specs/2026-08-18-juniata-single-catchment-sample-design.md· Plan:docs/superpowers/plans/2026-08-18-juniata-sample.mdWhat's included
examples/juniata/data/) — the 213-reach subgraph re-indexed from CONUS-positional space into a compact 0..212 space (tested pure function; topological/lower-triangular ordering preserved): icechunk Q' (dHBV2 UH retrospective) and USGS obs stores both starting 1980-01-01 (required by the readers' positional time origin), attributes NetCDF, binsparse adjacency zarrs, one-row gage CSV. The unmodified library (Merit,StreamflowReader,kan,dmc) runs on it — nothing undersrc/ddr/changed.extract_bundle.py— maintainer-run extraction script (regenerates the bundle from the CONUS stores).train_and_test.py— plain-Python, no Hydra:make_config/train/test/summed_qprime_baseline, runnable as a CLI or from the notebook. Theq_prime = flow(...)line is the documented seam for the next project: a differentiable runoff model returning the same gradient-capable hourly(T, N)m³/s tensor plugs in there for a full end-to-end gradient chain.juniata_routing.ipynb(+ committedmake_notebook.pygenerator) — six sections: the basin, Muskingum-Cunge physics (coefficients + mass identity, trapezoid geometry, trapezoid-exact celerity β, Cunge X diffusion matching), the lower-triangular network solve, a live backward pass into the KAN, train & evaluate, and the road to end-to-end. Executes clean top-to-bottom on CPU.Reference result (30 epochs, CPU, test 1995-10-01–2010-09-30)
Routing beats the no-routing baseline by +0.089 NSE on a single gauge with 30 optimizer steps — also a live single-catchment validation of the corrected physics from #192.
Test plan
uv run pytest -q— 650 passed, 1 skipped (includes 7 new tests: re-indexing semantics, bundle contract via real readers, 1-epoch train→test→baseline smoke)uv run ruff format . --checkanduv run ruff check .clean repo-wide (notebook included)🤖 Generated with Claude Code