Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 0 additions & 43 deletions .github/workflows/test.yml

This file was deleted.

24 changes: 0 additions & 24 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,27 +18,3 @@ repos:
- id: check-yaml
exclude: ^mkdocs\.yml$
- id: trailing-whitespace
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.14.1
hooks:
- id: mypy
files: src|tests
additional_dependencies:
# Package dependencies
- packaging
- donfig
- numcodecs[crc32c]
- numpy==2.1 # until https://github.com/numpy/numpy/issues/28034 is resolved
- typing_extensions
- universal-pathlib
# Tests
- pytest
- pint
- zarr
- xarray
- dask
- obstore
- repo: https://github.com/numpy/numpydoc
rev: v1.8.0
hooks:
- id: numpydoc-validation
65 changes: 18 additions & 47 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,41 +4,32 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

## What this is

Two things in one `uv` workspace:
A MkDocs Material documentation site (`docs/`) — the published guide at
[developmentseed.org/datacube-guide](https://developmentseed.org/datacube-guide/),
covering datacube "worst practices", a visualization tool catalog, and a glossary.

1. **A MkDocs Material documentation site** (`docs/`) — the published guide at
[developmentseed.org/datacube-guide](https://developmentseed.org/datacube-guide/),
covering datacube "worst practices", a visualization tool catalog, and a glossary.
2. **The `datacube_benchmark` Python package** (`packages/datacube-benchmark/`) — a
library for measuring Zarr read patterns and access costs, used both standalone and
to generate content in the docs.
The worst-practices notebooks depend on the
[`datacube-benchmark`](https://github.com/developmentseed/datacube-benchmark)
library, which is developed in its own repository and pulled in from PyPI like any
other dependency. **Don't edit benchmark library code from this repo** — file
changes against `developmentseed/datacube-benchmark` instead.

The workspace is managed by `uv`; run all tooling through `uv run`.
The project is managed by `uv`; run all tooling through `uv run`.

## Commands

```bash
uv sync # install workspace + dev deps
uv sync # install dev deps
uv run -- mkdocs serve --livereload # live-reload docs at localhost:8000
uv run -- mkdocs build --strict # what CI runs; --strict fails on any warning
uv run python main.py # smoke-test the benchmark library
uv run pytest --cov # run tests; enforces the 90% coverage gate
uv run mypy # type-check (files set in pyproject.toml)
uv run ruff check . && uv run ruff format .
prek run --all-files # ruff, codespell, mypy, numpydoc-validation
prek run --all-files # ruff, codespell
```

Two CI workflows gate merges:

- **`docs.yml`** runs `mkdocs build --strict` — broken links, missing `nav:` entries,
and autodoc failures all surface as warnings that fail the build. It deploys `site/`
to GitHub Pages on push to `main` (`mike` provides versioned docs).
- **`test.yml`** runs `pytest --cov`. The coverage gate (`--cov-fail-under` equivalent)
lives in `[tool.coverage.report] fail_under = 90` in `pyproject.toml`, enforced by
`pytest-cov`.

`mypy` and `numpydoc-validation` run via pre-commit (`prek`/`pre-commit`), not in CI —
keep them green locally.
`docs.yml` is the only CI workflow: it runs `mkdocs build --strict` — broken links,
missing `nav:` entries, and autodoc failures all surface as warnings that fail the
build. It deploys `site/` to GitHub Pages on push to `main` (`mike` provides
versioned docs).

## Docs architecture

Expand All @@ -52,35 +43,15 @@ keep them green locally.
`mkdocs-jupyter` from probing `.md` files (which otherwise breaks `mkdocstrings`
autodoc on API reference pages). Don't remove it.
- API reference pages (`docs/api-reference/`) use `mkdocstrings` to autodoc the
benchmark package. Docstrings are **numpy-style**; broken cross-references fail the
strict build.
`datacube-benchmark` package installed from PyPI. Docstrings are **numpy-style**;
broken cross-references fail the strict build. To pick up new symbols, bump the
pinned version in `pyproject.toml` and `uv sync`.
- `includes/abbreviations.md` is auto-appended to every page (pymdownx snippets) for
glossary tooltips.
- `docs/overrides/` holds the Material theme custom dir and CSS.

## Benchmark package

`packages/datacube-benchmark/src/datacube_benchmark/` — public API is re-exported from
`__init__.py` (`Config`, `create_*`, `benchmark_*`). Built on `zarr`, `obstore`,
`xarray`, `dask`, and `pint` (quantities carry units). Storage I/O goes through
`obstore` stores rather than fsspec/paths directly. Note `pint` doesn't know `KB` —
size strings use `MB`/`GB`/`bytes`.

Tests live in `packages/datacube-benchmark/tests/`:

- Pure chunk math (`chunks.py`) uses **Hypothesis** property tests — assert invariants
(thickness ≥ 1, a "pancake" never chunks the spatial dims, an "over" chunk exceeds
the target), not exact values — plus a few worked examples.
- Integration tests build real Zarr stores against an `obstore` `LocalStore` under
`tmp_path` (see `conftest.py` fixtures) with a tiny coarse grid — no cloud. The one
cloud-only branch (`credential_provider` in `create_or_open_zarr_store`) is
`# pragma: no cover`; `earthaccess`/`s3fs` are declared deps but unused in source.
- The RNG is seeded (autouse fixture) so the timing/fill sampling is deterministic.

## Conventions

- Numpy-style docstrings, enforced by `numpydoc-validation` (checks listed in
`pyproject.toml`); 80-char line length for signatures in autodoc.
- `ruff` for lint + format. `codespell` runs in pre-commit with a custom ignore list
(`fo,ihs,kake,te`) — don't "fix" those.
- `dev-docs/tasks/` holds working notes/TODOs, not published content.
41 changes: 13 additions & 28 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,32 @@
# Contributing to the Datacube Guide

Thanks for your interest in improving the guide! This repository is a [`uv`](https://docs.astral.sh/uv/)
workspace holding two things:
Thanks for your interest in improving the guide! This repository is the
**MkDocs Material documentation site** published at
[developmentseed.org/datacube-guide](https://developmentseed.org/datacube-guide/),
managed as a [`uv`](https://docs.astral.sh/uv/) project.

- the **MkDocs Material documentation site** (`docs/`), published at
[developmentseed.org/datacube-guide](https://developmentseed.org/datacube-guide/), and
- the **`datacube_benchmark` Python package** (`packages/datacube-benchmark/`), a library
for measuring Zarr read patterns and access costs.
The notebooks that generate the worst-practices figures depend on the
[`datacube-benchmark`](https://github.com/developmentseed/datacube-benchmark)
library, which lives in its own repository and is installed here from
PyPI like any other dependency. Library changes go in that repo; this
repo only consumes the released package.

See the [README](README.md) for a higher-level overview.

## Development setup

Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/), then install the
workspace and its dev dependencies:
project and its dev dependencies:

```bash
uv sync
```

Run all tooling through `uv run` so it uses the workspace environment.
Run all tooling through `uv run` so it uses the project environment.

```bash
uv run -- mkdocs serve # live-reload docs at http://localhost:8000
uv run -- mkdocs build --strict # what CI runs; fails on any warning
uv run python main.py # smoke-test the benchmark library
```

## The strict build is the gate
Expand Down Expand Up @@ -67,26 +69,11 @@ done
Some diagrams (e.g. the static-vs-dynamic comparison and the grid-topologies figure) are
hand-authored SVG with no `.dot` source — edit the `.svg` directly.

## Working on the benchmark package

The public API is re-exported from `packages/datacube-benchmark/src/datacube_benchmark/__init__.py`.
The library is built on `zarr`, `obstore`, `xarray`, `dask`, and `pint`, and routes storage I/O
through `obstore` stores rather than fsspec/paths directly.

```bash
uv run mypy # type-check (files configured in pyproject.toml)
uv run ruff check . && uv run ruff format .
```

Note: `uv run mypy` currently reports a handful of pre-existing errors (missing third-party
stubs and string-annotation lookups), so it does not start clean.

## Code quality

[`prek`](https://github.com/j178/prek) — a fast, drop-in replacement for `pre-commit` that
reads the same `.pre-commit-config.yaml` — runs `ruff`, `codespell`, `mypy`, and
`numpydoc-validation`. Install it (e.g. `uv tool install prek`), install the git hooks once,
then run the checks before pushing:
reads the same `.pre-commit-config.yaml` — runs `ruff` and `codespell`. Install it (e.g.
`uv tool install prek`), install the git hooks once, then run the checks before pushing:

```bash
prek install
Expand All @@ -96,8 +83,6 @@ prek run --all-files
- **`ruff`** handles lint and formatting.
- **`codespell`** has a custom ignore list in `.codespellrc` — add genuine false positives
there rather than disabling the check.
- **Docstrings are numpy-style**, enforced by `numpydoc-validation` (checks listed in
`pyproject.toml`); keep signatures within an 80-char line length for autodoc.

## Submitting changes

Expand Down
69 changes: 55 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,76 @@
# Datacube Guide

[![Docs](https://img.shields.io/badge/docs-developmentseed.org%2Fdatacube--guide-blue)](https://developmentseed.org/datacube-guide/)
[![PyPI](https://img.shields.io/pypi/v/datacube-benchmark.svg)](https://pypi.org/project/datacube-benchmark/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Practical guidance for producing, using, and visualizing datacubes.

Read the guide at [developmentseed.org/datacube-guide](https://developmentseed.org/datacube-guide/).

> Status: First draft, actively iterating.

## What's inside

- **Worst practices** — common pitfalls to avoid when producing and using multi-dimensional data products: chunking, metadata, datatypes, and default library configs (FSSpec, GDAL, Xarray).
- **Visualization** — a catalog and comparison of tools for visualizing Zarr-backed datacubes in the browser, covering both server-side dynamic tilers (TiTiler, Xpublish-tiles) and client-side rendering, with libraries for deck.gl, MapLibre/Mapbox, and Cesium, plus standalone viewer apps. Includes side-by-side comparisons and guidance on choosing an approach.
- **Benchmarking** — a small Python library (`datacube_benchmark`) for measuring read patterns and access costs.
The guide (`docs/`) is published at
[developmentseed.org/datacube-guide](https://developmentseed.org/datacube-guide/)
and covers:

- **Worst practices** — common pitfalls when producing and using
multi-dimensional data products: chunking, metadata, datatypes, and
default library configs (FSSpec, GDAL, Xarray).
- **Visualization** — a catalog and comparison of tools for visualizing
Zarr-backed datacubes in the browser, covering both server-side
dynamic tilers (TiTiler, Xpublish-tiles) and client-side rendering
with deck.gl, MapLibre/Mapbox, and Cesium, plus standalone viewer
apps.

The notebooks that generate the worst-practices figures use the
[`datacube-benchmark`](https://github.com/developmentseed/datacube-benchmark)
Python package — a library for measuring Zarr read patterns and access
costs — which lives in its own repository and is installable from PyPI:

```bash
pip install datacube-benchmark
```

## Installation
## Working on the guide locally

```bash
git clone https://github.com/developmentseed/datacube-guide.git
cd datacube-guide
# Serve the documentation
uv run -- mkdocs serve
# Try out the `datacube_benchmark` library in Python
uv run python
# Try out the `datacube_benchmark` library in JupyterLab
uv run ipython kernel install --user --env VIRTUAL_ENV $(pwd)/.venv --name=project
uv sync # install dev deps
uv run -- mkdocs serve --livereload # docs at http://localhost:8000
```

To experiment in a notebook:

```bash
uv run ipython kernel install --user \
--env VIRTUAL_ENV "$(pwd)/.venv" --name=datacube-guide
uv run --with jupyter jupyter lab
```

For the full development workflow — strict docs build, regenerating
diagrams — see [CONTRIBUTING.md](CONTRIBUTING.md).

## Acknowledgements

The Datacube Guide was initiated in partnership with the Microsoft Planetary Computer team. We recommend checking out the wonderful work going on as part of the [Microsoft Planetary Computer Pro service](https://learn.microsoft.com/en-us/azure/planetary-computer/) as well as the [Open Planetary Computer Data Catalog](https://planetarycomputer.microsoft.com/). We greatly appreciate Microsoft's dedication to supporting open resources and building impactful geospatial services.
The Datacube Guide was initiated in partnership with the Microsoft
Planetary Computer team. We recommend checking out the wonderful work
going on as part of the
[Microsoft Planetary Computer Pro service](https://learn.microsoft.com/en-us/azure/planetary-computer/)
as well as the [Open Planetary Computer Data Catalog](https://planetarycomputer.microsoft.com/).
We greatly appreciate Microsoft's dedication to supporting open
resources and building impactful geospatial services.

The latest updates to this guide were supported by NASA's Office of
Data Science and Informatics (ODSI) as part of the Data Systems
Evolution team. The Data Systems Evolution team at NASA Marshall Space
Flight Center's Office of Data Science and Informatics enables
scientific exploration and discovery through innovative data
visualization techniques and analysis capabilities that lower the
barrier to entry for cloud-hosted data.

## License

`datacube-guide` is distributed under the terms of the [MIT](https://spdx.org/licenses/MIT.html) license.
`datacube-guide` is distributed under the terms of the
[MIT](https://spdx.org/licenses/MIT.html) license.
2 changes: 2 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,5 @@ How to render Zarr and other cloud-optimized datacubes dynamically on the web
## Acknowledgements

The Datacube Guide was initiated in partnership with the Microsoft Planetary Computer team. We recommend checking out the wonderful work going on as part of the [Microsoft Planetary Computer Pro service](https://learn.microsoft.com/en-us/azure/planetary-computer/) as well as the [Open Planetary Computer Data Catalog](https://planetarycomputer.microsoft.com/). We greatly appreciate Microsoft's dedication to supporting open resources and building impactful geospatial services.

The latest updates to this guide were supported by NASA's Office of Data Science and Informatics (ODSI) as part of the Data Systems Evolution team. The Data Systems Evolution team at NASA Marshall Space Flight Center's Office of Data Science and Informatics enables scientific exploration and discovery through innovative data visualization techniques and analysis capabilities that lower the barrier to entry for cloud-hosted data.
18 changes: 0 additions & 18 deletions main.py

This file was deleted.

2 changes: 0 additions & 2 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,6 @@ hooks:
- hooks/mkdocs_jupyter_md_filter.py

watch:
- packages
- docs
- mkdocs.yml
- pyproject.toml
Expand Down Expand Up @@ -126,7 +125,6 @@ plugins:
enable_inventory: true
handlers:
python:
paths: [virtualizarr]
options:
allow_inspection: false
docstring_section_style: list
Expand Down
1 change: 0 additions & 1 deletion packages/datacube-benchmark/README.md

This file was deleted.

Loading