Original default

.github/ISSUE_TEMPLATE/capability.md

@@ -0,0 +1,38 @@
+---
+name: JIRA Story/Capability
+about: The structure for outlining work being done on a JIRA story
+labels: JIRA Story
+---
+
+# Capability
+
+<!-- Please link below the JIRA Story for coupling this issue to the scheduled work -->
+
+## Task
+
+<!-- Provide a clear and concise description of the task / work to be done to the repo
+
+* This ticket is meant to do XYZ and to accomplish the following:
+- ...
+- ...
+ -->
+
+## Plan/Outline
+
+<!-- Provide a few sentences about how this issue / story will be completed
+
+* This work will be done by creating a BMI module in the XYZ dir
+ -->
+
+
+### TODOS
+
+<!-- Create a checklist of what needs to be accomplished in this work -->
+- [ ]
+
+### Additional components / Context
+
+<!-- Add any other context or screenshots about the story here. You can also include links to examples of other programs that have something similar to your request. For example:
+
+* Another project [...] solved this by [...]
+-->

.github/PULL_REQUEST_TEMPLATE.md

@@ -18,7 +18,6 @@
 
 ## Screenshots
 
-
 ## Notes
 
 -

.github/workflows/cicd.yaml

@@ -0,0 +1,56 @@
+name: CI/CD
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+
+jobs:
+  ruff:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Set up Python
+        uses: astral-sh/setup-uv@v5
+        with:
+          python-version: "3.13"
+
+      - name: Install dependencies
+        run: uv sync
+
+      - name: Run ruff check
+        run: |
+          uv run ruff check --config pyproject.toml --output-format=github
+
+      - name: Run ruff format check
+        run: |
+          uv run ruff format --config pyproject.toml
+
+  pytests:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ['3.12', '3.13']
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: uv sync --all-extras
+
+      - name: Run tests
+        run: uv run pytest tests

.pre-commit-config.yaml

@@ -0,0 +1,30 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+      - id: trailing-whitespace
+        exclude: LICENSE|\.csv$
+      - id: end-of-file-fixer
+        exclude: LICENSE|\.csv$
+      - id: check-yaml
+      - id: debug-statements
+
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.14.0
+    hooks:
+      - id: ruff-check
+        types_or: [ python, pyi ]
+        args: [ --fix ]
+      - id: ruff-format
+        types_or: [ python, pyi ]
+
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v1.17.0
+    hooks:
+      - id: mypy
+        additional_dependencies: ["types-PyYAML==6.0.12.20250516"]
+
+  - repo: https://github.com/kynan/nbstripout
+    rev: 0.8.1
+    hooks:
+      - id: nbstripout

LICENSE

@@ -201,11 +201,11 @@
    See the License for the specific language governing permissions and
    limitations under the License.
 
-- - - - - - - - - - - - - -
+- - - - - - - - - - - - - - -
 
-“Software code created by U.S. Government employees is not subject to copyright in the
+"Software code created by U.S. Government employees is not subject to copyright in the
 United States (17 U.S.C. §105). The United States/Department of Commerce reserves all
 rights to seek and obtain copyright protection in countries other than the United States
 for Software authored in its entirety by the Department of Commerce. To this end, the
 Department of Commerce hereby grants to Recipient a royalty-free, nonexclusive license
-to use, copy, and create derivative works of the Software outside of the United States.”
+to use, copy, and create derivative works of the Software outside of the United States."

NOTICE.txt

@@ -0,0 +1,7 @@
+--
+
+Inspiration and assistance in the creation, and storage of network graphs comes from the following repos. Credit to the authors:
+- https://github.com/DeepGroundwater/ddr/blob/ab4c3962c2c119e6a9182a77f2a9faceec19f2e0/engine/adjacency.py
+- https://github.com/CIROH-UA/NGIAB_data_preprocess/blob/main/modules/data_processing/graph_utils.py
+
+--

README.md

@@ -5,7 +5,9 @@
 3. Update the README, replacing the contents below as prescribed.
 4. Add any libraries, assets, or hard dependencies whose source code will be included
    in the project's repository to the _Exceptions_ section in the [TERMS](TERMS.md).
-  - If no exceptions are needed, remove that section from TERMS.
+
+- If no exceptions are needed, remove that section from TERMS.
+
 5. If working with an existing code base, answer the questions on the [open source checklist](opensource-checklist.md)
 6. Delete these instructions and everything up to the _Project Title_ from the README.
 7. Write some great software and tell people about it.
@@ -20,28 +22,26 @@ To install all of the template files, run the following script from the root of
 bash -c "$(curl -s https://raw.githubusercontent.com/NOAA-OWP/owp-open-source-project-template/open_source_template.sh)"
 ```
 
-----
+---
 
 # Project Title
 
-**Description**:  Put a meaningful, short, plain-language description of what
+**Description**: Put a meaningful, short, plain-language description of what
 this project is trying to accomplish and why it matters.
 Describe the problem(s) this project solves.
 Describe how this software can improve the lives of its audience.
 
 Other things to include:
 
-  - **Technology stack**: Indicate the technological nature of the software, including primary programming language(s) and whether the software is intended as standalone or as a module in a framework or other ecosystem.
-  - **Status**:  Alpha, Beta, 1.1, etc. It's OK to write a sentence, too. The goal is to let interested people know where this project is at. This is also a good place to link to the [CHANGELOG](CHANGELOG.md).
-  - **Links to production or demo instances**
-  - Describe what sets this apart from related-projects. Linking to another doc or page is OK if this can't be expressed in a sentence or two.
-
+- **Technology stack**: Indicate the technological nature of the software, including primary programming language(s) and whether the software is intended as standalone or as a module in a framework or other ecosystem.
+- **Status**: Alpha, Beta, 1.1, etc. It's OK to write a sentence, too. The goal is to let interested people know where this project is at. This is also a good place to link to the [CHANGELOG](CHANGELOG.md).
+- **Links to production or demo instances**
+- Describe what sets this apart from related-projects. Linking to another doc or page is OK if this can't be expressed in a sentence or two.
 
 **Screenshot**: If the software has visual components, place a screenshot after the description; e.g.,
 
 ![](https://raw.githubusercontent.com/NOAA-OWP/owp-open-source-project-template/master/doc/Screenshot.png)
 
-
 ## Dependencies
 
 Describe any dependencies that must be installed for this software to work.
@@ -88,8 +88,7 @@ important pieces, etc.
 
 General instructions on _how_ to contribute should be stated with a link to [CONTRIBUTING](CONTRIBUTING.md).
 
-
-----
+---
 
 ## Open source licensing info
 
@@ -99,8 +98,7 @@ as is, but remove this sentence):
 1. [TERMS](TERMS.md)
 2. [LICENSE](LICENSE)
 
-
-----
+---
 
 ## Credits and references
 

builds/build_reference.py

@@ -0,0 +1,86 @@
+"""An end-to-end build file that will take the NHD ReferenceConfig and turn it into a reference fabric"""
+
+import argparse
+import logging
+
+from pydantic import ValidationError
+
+from reference_builds.configs import BaseDataset, ReferenceConfig
+from reference_builds.local_runner import LocalRunner
+from reference_builds.pipeline import (
+    build_geoglows_graphs,
+    build_geoglows_reference,
+    build_nhd_graphs,
+    build_nhd_reference,
+    build_usgs_hf_graphs,
+    build_usgs_hf_reference,
+    download_geoglows_data,
+    download_nhd_data,
+    download_usgs_hf_data,
+    write_reference,
+)
+
+logger = logging.getLogger(__name__)
+
+
+def main() -> int:
+    """Main entry point for the hydrofabric-build pipeline CLI.
+
+    Returns
+    -------
+    int
+        Exit code: 0 for success, 1 for failure.
+    """
+    parser = argparse.ArgumentParser(description="A local runner for hydrofabric data processing")
+    parser.add_argument("--config", required=False, help="Config file")
+    args = parser.parse_args()
+
+    try:
+        config = ReferenceConfig.from_yaml(args.config)
+    except ValidationError as e:
+        print("Configuration validation failed:")
+        for error in e.errors():
+            print(f"  {error['loc']}: {error['msg']}")
+        return 1
+    except FileNotFoundError:
+        logger.error(f"Config file not found: {args.config}")
+        return 1
+    except TypeError as e:
+        logger.error("Config file not specified.")
+        raise TypeError("Config file not specified.") from e
+
+    with LocalRunner(config) as runner:
+        if config.base_dataset == BaseDataset.NHD:
+            runner.run_task(task_id="download", python_callable=download_nhd_data, op_kwargs={})
+            runner.run_task(task_id="build_nhd_graphs", python_callable=build_nhd_graphs, op_kwargs={})
+            runner.run_task(task_id="build_reference", python_callable=build_nhd_reference, op_kwargs={})
+            runner.run_task(task_id="write_reference", python_callable=write_reference, op_kwargs={})
+        elif config.base_dataset == BaseDataset.GEOGLOWS:
+            runner.run_task(task_id="download", python_callable=download_geoglows_data, op_kwargs={})
+            runner.run_task(
+                task_id="build_geoglows_graphs", python_callable=build_geoglows_graphs, op_kwargs={}
+            )
+            runner.run_task(task_id="build_reference", python_callable=build_geoglows_reference, op_kwargs={})
+            runner.run_task(task_id="write_reference", python_callable=write_reference, op_kwargs={})
+        elif config.base_dataset == BaseDataset.USGS_HF:
+            runner.run_task(task_id="download", python_callable=download_usgs_hf_data, op_kwargs={})
+            runner.run_task(
+                task_id="build_usgs_hf_graphs", python_callable=build_usgs_hf_graphs, op_kwargs={}
+            )
+            runner.run_task(task_id="build_reference", python_callable=build_usgs_hf_reference, op_kwargs={})
+            runner.run_task(task_id="write_reference", python_callable=write_reference, op_kwargs={})
+        else:
+            raise NotImplementedError("Base Dataset not implemented")
+
+        print("Pipeline completed")
+        print("=" * 60)
+        for task_id, info in runner.results.items():
+            status = "✓" if info["status"] == "success" else "✗"
+            print(f"  {status} {task_id}: {info['status']}")
+        print("=" * 60)
+
+    return 0
+
+
+if __name__ == "__main__":
+    exit(main())

config/example_ak.yaml

@@ -0,0 +1,5 @@
+domain: ak
+base_dataset: usgs-reference-hf
+input_file_regex: AK/reference_19.gpkg
+vpu_id: "19"
+crs: EPSG:3338

config/example_hi.yaml

@@ -0,0 +1,18 @@
+domain: hi
+input_file_regex: HI/NHDPLUS_H_20*_HU4_GPKG
+vpu_id: "20"
+crs: EPSG:32604
+permitted_fcodes:
+  - 'Canal/Ditch'
+  - 'Stream/River: Hydrographic Category = Perennial'
+  - 'Stream/River: Hydrographic Category = Intermittent'
+  - 'Artificial Path'
+  - 'Pipeline: Pipeline Type = Siphon'
+  - 'Pipeline: Pipeline Type = Aqueduct'
+  - 'Pipeline'
+  - 'Connector'
+  - 'Canal Ditch: Canal Ditch Type = Stormwater'
+  # These FCODES are currently disabled in HI due to incorrect NULL `toid` and `dn_hydroseq` values.
+  # - 'Pipeline: Pipeline Type = Aqueduct; Relationship to Surface = At or Near'
+  # - 'Pipeline: Pipeline Type = Aqueduct; Relationship to Surface = Underground'
+  # - 'Pipeline: Pipeline Type = Penstock; Relationship to Surface = At or Near'

config/example_prvi.yaml

@@ -0,0 +1,17 @@
+domain: prvi
+input_file_regex: PRVI/NHDPLUS_H_21*_HU4_GPKG
+vpu_id: "21"
+crs: EPSG:6566
+permitted_fcodes:
+  - 'Canal/Ditch'
+  - 'Stream/River: Hydrographic Category = Perennial'
+  - 'Stream/River: Hydrographic Category = Intermittent'
+  - 'Artificial Path'
+  - 'Pipeline: Pipeline Type = Aqueduct; Relationship to Surface = At or Near'
+  - 'Pipeline: Pipeline Type = Aqueduct; Relationship to Surface = Underground'
+  - 'Pipeline: Pipeline Type = Siphon'
+  - 'Pipeline: Pipeline Type = Aqueduct'
+  - 'Pipeline'
+  - 'Connector'
+  - 'Canal Ditch: Canal Ditch Type = Stormwater'
+  - 'Pipeline: Pipeline Type = Penstock; Relationship to Surface = At or Near'