Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
059e065
docs: design for Sentinel-3 OLCI L1 EFR -> GeoZarr export
d-v-b Jun 21, 2026
ee43eba
docs: implementation plan for Sentinel-3 OLCI export
d-v-b Jun 21, 2026
c613e24
chore: regenerate uv.lock for released zarr-cm 0.4.1
d-v-b Jun 21, 2026
c49ab06
feat(s3-olci): add OLCI band mapping
d-v-b Jun 22, 2026
cbe586f
feat(s3-olci): add Sentinel3OlciRoot data-api model
d-v-b Jun 22, 2026
8debe8d
fix(s3-olci): remove Any from data-api model (use precise opaque-grou…
d-v-b Jun 22, 2026
5874f26
feat(s3-olci): add swath /2 decimation for overviews
d-v-b Jun 22, 2026
01abf31
fix(s3-olci): move xarray import to TYPE_CHECKING; strengthen decimat…
d-v-b Jun 22, 2026
f7998de
feat(s3-olci): add OLCI product detection
d-v-b Jun 22, 2026
c05a060
feat(s3-olci): swath spatial-convention attrs (no transform)
d-v-b Jun 22, 2026
67e86d4
feat(s3-olci): add convert_olci_optimized entry point
d-v-b Jun 22, 2026
26edc80
fix(s3-olci): correct overview min_dimension guard; document returned…
d-v-b Jun 22, 2026
b672759
feat(s3-olci): CLI auto-detect + convert-s3-olci-optimized command
d-v-b Jun 22, 2026
cec1e38
test(s3-olci): real OLCI product fixture + round-trip detection
d-v-b Jun 22, 2026
0b45392
test(s3-olci): golden-file snapshot of converted OLCI structure
d-v-b Jun 22, 2026
c673287
fix(s3-olci): strip inherited v2 encoding before v3 write; fix fixtur…
d-v-b Jun 22, 2026
ade00de
docs(s3-olci): document Sentinel-3 OLCI export support
d-v-b Jun 22, 2026
73a47b9
fix(s3-olci): fill-aware block-avg overviews, multiscales CMO, orphan…
d-v-b Jun 22, 2026
3849b24
fix(s3-olci): restore faithful 'columns' dim name in OLCI fixture (re…
d-v-b Jun 22, 2026
43d4bf0
fix(s3-olci): open with mask_and_scale=False; sanitize stale attrs; r…
d-v-b Jun 22, 2026
a0cf790
fix: revert shared sanitize_array_attrs to pre-OLCI semantics; add OL…
d-v-b Jun 22, 2026
a112179
fix(s3-olci): align coordinate decimation with coarsen trim on odd di…
d-v-b Jun 23, 2026
56b2b2e
docs(s3-olci): add Sentinel-3 OLCI GeoZarr demonstration notebook
d-v-b Jun 23, 2026
7a8361a
bump zarr conventions; migrate type checking to pyright (#199)
d-v-b Jul 6, 2026
da5cf57
Merge remote-tracking branch 'upstream/main' into feat/sentinel3-export
d-v-b Jul 6, 2026
c774b85
fix(s3-olci): open raw input in convert auto-detect path; guard overv…
d-v-b Jul 6, 2026
1db0929
fix(s3-olci): forward --min-dimension in auto-detect path; avoid fill…
d-v-b Jul 6, 2026
5925c64
fix(s3-olci): add --no-s3-olci-optimized opt-out; clarify divergent s…
d-v-b Jul 6, 2026
1d4ef57
fix(s3-olci): clamp fill-collision nudge to dtype range; document str…
d-v-b Jul 6, 2026
55f110c
chore: remove superpowers planning docs from version control
d-v-b Jul 6, 2026
cc203e0
fix(s3-olci): warn on unwired converter options; NaN-aware fill masking
d-v-b Jul 6, 2026
1ad264f
fix(s3-olci): defensive return-tree assembly; test fill-collision nudge
d-v-b Jul 6, 2026
10db2dd
perf(s3-olci): materialize coarsened mean once; never raise from dete…
d-v-b Jul 6, 2026
31ca390
fix(s3-olci): clear error for missing swath dims; finer return-tree f…
d-v-b Jul 6, 2026
4e2f7f8
fix(s3-olci): explicit skipna in fill-aware mean; depth-sort tree fal…
d-v-b Jul 6, 2026
b2606b2
fix(s3-olci): truncate store on re-run; handle transposed swath dims
d-v-b Jul 6, 2026
c2b4fc0
docs(s3-olci): plot each pyramid level on its own native grid in note…
d-v-b Jul 7, 2026
55b950a
feat(s3-olci): overview lat/lon are geodesic block centroids, not dec…
d-v-b Jul 7, 2026
964c047
fix(s3-olci): raw return tree; block-average altitude in overviews
d-v-b Jul 7, 2026
cd847b9
perf(cli): check --no-s3-olci-optimized before running OLCI detection
d-v-b Jul 7, 2026
36b6b07
docs(s3-olci): draw swath quadrants from different pyramid levels ove…
d-v-b Jul 9, 2026
36fb851
fix(conversion): open sources with native-chunk-aligned reads and an …
lhoupert Jul 17, 2026
cc87664
chore: release 0.10.2 (#187)
github-actions[bot] Jul 17, 2026
acbce35
Merge remote-tracking branch 'origin/main' into merge-main-into-senti…
lhoupert Jul 17, 2026
a4a6973
chore: re-lock uv.lock after merging main
lhoupert Jul 17, 2026
5ea5662
fix(s3-olci): open OLCI sources with native-chunk-aligned reads
lhoupert Jul 17, 2026
8828401
Merge pull request #3 from EOPF-Explorer/merge-main-into-sentinel3-ex…
d-v-b Jul 17, 2026
d69bd45
Merge pull request #4 from EOPF-Explorer/port-open-source-to-olci
d-v-b Jul 17, 2026
4121f94
fix(s3-olci): write native resolution to measurements/r0 so the store…
d-v-b Jul 27, 2026
73cc511
docs: design for OLCI reprojection to regular grid with CRS
d-v-b Jul 27, 2026
88808d9
docs: implementation plan for OLCI reprojection and cross-product CRS…
d-v-b Jul 27, 2026
d21fd98
feat(s3-olci): add swath-to-grid reprojection via rasterio geolocatio…
d-v-b Jul 27, 2026
ccd72ee
fix(s3-olci): guarantee _FillValue on all warped variables; test dims…
d-v-b Jul 27, 2026
637b1d8
refactor(s3-olci): parameterize reduce/decimate over spatial dims for…
d-v-b Jul 27, 2026
c21e793
feat(s3-olci): add gridded spatial-convention attrs with affine trans…
d-v-b Jul 27, 2026
85045d4
feat(s3-olci): reproject measurements to regular EPSG:4326 grid with …
d-v-b Jul 27, 2026
2d0d6fd
feat(s3-olci): --target-crs flag, regenerated snapshot, gridded-layou…
d-v-b Jul 27, 2026
9967de3
test: cross-product contract - every converted product is a regular g…
d-v-b Jul 27, 2026
2cab010
docs(s3-olci): fix stale notebook intro to describe reprojected grid
d-v-b Jul 27, 2026
863a1f5
fix(test): open contract levels with decode_coords=all, drop bogus S1…
d-v-b Jul 27, 2026
bc65f79
test(s3-olci): cover projected --target-crs end-to-end (EPSG:3857)
d-v-b Jul 28, 2026
1a33666
Merge origin/feat/sentinel3-export: reprojected grid supersedes swath…
d-v-b Jul 28, 2026
547981d
docs(s3-olci): re-execute notebook against merged converter
d-v-b Jul 28, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -218,3 +218,4 @@ uv.lock
# VCS versioning
src/eopf_geozarr/_version.py
analysis/.edh_token
docs/superpowers/
2 changes: 1 addition & 1 deletion .release-please-manifest.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
".": "0.10.1"
".": "0.10.2"
}
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,16 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## 0.10.2 (2026-07-17)

## What's Changed
* ci(deps): bump the actions group with 3 updates by @dependabot[bot] in https://github.com/EOPF-Explorer/data-model/pull/185
* bump zarr conventions; migrate type checking to pyright by @d-v-b in https://github.com/EOPF-Explorer/data-model/pull/199
* fix(conversion): concurrent-safe source opening (native-chunk-aligned reads + atomic CacheStore) for data-pipeline#339 by @lhoupert in https://github.com/EOPF-Explorer/data-model/pull/220


**Full Changelog**: https://github.com/EOPF-Explorer/data-model/compare/v0.10.1...v0.10.2

## 0.10.1 (2026-06-09)

## What's Changed
Expand Down
68 changes: 65 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,6 @@ dt_geozarr = create_geozarr_dataset(
output_path="s3://my-bucket/geozarr-data/output.zarr",
spatial_chunk=4096,
min_dimension=256,
tile_width=256,
max_retries=3
)
```
Expand All @@ -207,7 +206,6 @@ dt_geozarr = create_geozarr_dataset(
output_path="path/to/output/geozarr.zarr",
spatial_chunk=4096,
min_dimension=256,
tile_width=256,
max_retries=3
)
```
Expand All @@ -227,7 +225,6 @@ Create a GeoZarr-spec 0.4 compliant dataset from EOPF data.
- `output_path` (str): Output path for the Zarr store
- `spatial_chunk` (int, default=4096): Spatial chunk size for encoding
- `min_dimension` (int, default=256): Minimum dimension for overview levels
- `tile_width` (int, default=256): Tile width for TMS compatibility
- `max_retries` (int, default=3): Maximum number of retries for network operations

**Returns:**
Expand Down Expand Up @@ -283,6 +280,71 @@ Check if a variable is a grid_mapping variable by looking for references to it.

Validate that a specific band exists and is complete in the dataset.

## Supported Products

### Sentinel-2 MSI

Sentinel-2 MSI (MultiSpectral Instrument) L1C and L2A products are detected
automatically by `eopf-geozarr convert` and routed to the optimized multiscale
layout (`convert-s2-optimized`). The three native resolution groups (10 m, 20 m,
60 m) are reused as-is and coarser overviews (120 m, 360 m, 720 m) are computed
via /2 downsampling.

### Sentinel-3 OLCI L1 EFR

Sentinel-3 OLCI (Ocean and Land Colour Instrument) Level-1 EFR (Full Resolution)
products are detected automatically by `eopf-geozarr convert` and routed to the
dedicated OLCI converter. Unlike Sentinel-2, OLCI data starts out on **native
swath geometry**: measurements are stored on a per-pixel 2-D lat/lon grid. The
exporter warps this curvilinear swath once onto a regular grid (default
`EPSG:4326`) so that the output is a standard GeoZarr raster.

#### Auto-detection

```bash
eopf-geozarr convert S3A_OL_1_EFR.zarr output.zarr
```

#### Dedicated command

```bash
eopf-geozarr convert-s3-olci-optimized S3A_OL_1_EFR.zarr output.zarr \
--spatial-chunk 1024 \
--min-dimension 256 \
--compression-level 3
```

Key flags:

- `--spatial-chunk` — target spatial chunk size in pixels (default: 1024)
- `--compression-level` — Blosc/zstd compression level 1–9 (default: 3)
- `--min-dimension` — stop generating /2 overview levels once either spatial
dimension would drop below this value (default: 256)
- `--enable-sharding` — accepted but not yet wired into encoding (follow-up task)
- `--keep-scale-offset` — accepted but not yet wired into encoding (follow-up task)
- `--target-crs` — target CRS for the reprojected output grid (default: `EPSG:4326`)

#### What is converted

- **`/measurements/r0`**: all 21 OLCI radiance bands warped once from the
native swath onto a regular grid (default `EPSG:4326`, ~300 m preserved),
with 1-D `y`/`x` coordinates, a `spatial_ref` variable, and `grid_mapping`
on every band; the parent `measurements/` group carries the GeoZarr
`multiscales`, `spatial:`, and `proj:` convention metadata.
- **Overview subgroups** (`r2`, `r4`, …): /2 fill-aware block-averaged copies
stored as sibling Zarr groups next to `r0`, each with its own CRS metadata.
- **`/conditions` and `/quality`**: copied through unmodified.

> **Note:** per-scan-line `time_stamp` is not representable on a regular grid
> and is dropped from the converted measurements (it remains in the source
> product).

> **Note:** OLCI support is initial/measurements-focused (v1). Tie-point grid
> groups (`conditions/geometry`, `meteorology`, `instrument`) are copied through but
> not converted to GeoZarr convention. Encoding wiring for `--enable-sharding`,
> `--spatial-chunk`, `--compression-level`, and `--keep-scale-offset` is accepted
> but scheduled as a follow-up task.

## Architecture

The library is organized into the following modules:
Expand Down
62 changes: 62 additions & 0 deletions docs/converter.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,68 @@ dt_optimized = convert_s2_optimized(

The result is a space-efficient multiscale pyramid: `/measurements/reflectance/{r10m, r20m, r60m, r120m, r360m, r720m}` where the native resolutions are preserved as-is and only the coarser levels are computed.

## Sentinel-3 OLCI L1 EFR Conversion

Sentinel-3 OLCI (Ocean and Land Colour Instrument) Level-1 EFR (Full Resolution)
products are supported. OLCI measurements start out on **native swath geometry**:
a per-pixel 2-D lat/lon grid. The converter reprojects (warps) the swath measurements
once onto a regular grid (default `EPSG:4326`) using the swath geolocation arrays,
then generates /2 fill-aware block-averaged overview subgroups for multi-resolution
access.

### Auto-detection

The generic `convert` command detects OLCI products automatically:

```bash
eopf-geozarr convert S3A_OL_1_EFR.zarr output.zarr
```

### Dedicated command

A dedicated command offers fine-grained control:

```bash
eopf-geozarr convert-s3-olci-optimized S3A_OL_1_EFR.zarr output.zarr \
--spatial-chunk 1024 \
--min-dimension 256 \
--compression-level 3
```

| Flag | Default | Description |
|------|---------|-------------|
| `--spatial-chunk` | 1024 | Target spatial chunk size in pixels |
| `--compression-level` | 3 | Blosc/zstd compression level (1–9) |
| `--min-dimension` | 256 | Minimum spatial dimension for overview levels |
| `--enable-sharding` | off | Accepted but not yet wired into encoding (follow-up task) |
| `--keep-scale-offset` | off | Accepted but not yet wired into encoding (follow-up task) |
| `--target-crs` | EPSG:4326 | Target CRS for the reprojected output grid |

### Output layout

```
output.zarr/
├── measurements/ # Carries multiscales + spatial: + proj: convention metadata
│ ├── r0/ # Warped native-resolution OLCI bands (oa01_radiance … oa21_radiance)
│ │ # with 1-D y/x coordinates and spatial_ref
│ ├── r2/ # 1/2-resolution overview
│ ├── r4/ # 1/4-resolution overview
│ └── ...
├── conditions/ # Copied through unmodified (tie-point geometry, meteorology)
└── quality/ # Copied through unmodified (quality flags)
```

Each measurement group carries GeoZarr `spatial:` and `proj:` convention metadata,
with 1-D `y`/`x` dimension coordinates and a `spatial_ref` variable referenced by
`grid_mapping` on every band. Per-scan-line `time_stamp` is dropped, since it has
no representation on a regular grid.

> **Note:** OLCI support is initial/measurements-focused (v1). Tie-point grid
> groups in `conditions/geometry`, `meteorology`, and `instrument` are copied
> through but not converted to GeoZarr convention. Encoding wiring for
> `--enable-sharding`, `--spatial-chunk`, `--compression-level`, and
> `--keep-scale-offset` is accepted but scheduled as a follow-up task.

## Error Handling

The converter includes robust error handling and retry logic for network operations, ensuring reliable processing even in challenging environments.
Expand Down
Loading