feat: Sentinel-3 OLCI L1 EFR → GeoZarr exporter - #2
Draft
d-v-b wants to merge 63 commits into
Draft
Conversation
First Sentinel-3 exporter, scoped to OLCI L1 EFR. Grounded in a real EOPF product introspected from the EODC sample store: 21 radiance bands on a single ~300m curvilinear swath grid geolocated by per-pixel 2D lat/lon (no projected CRS). Decision: preserve native swath geometry with CF 2D coordinates (no reprojection); /2 decimation pyramid; measurements-first scope; mirror the S2 package + data_api model + CLI auto-detection. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
10-task TDD plan mirroring the S2 exporter: OLCI band mapping, data_api model, swath /2 decimation, structural detection, swath GeoZarr metadata, convert_olci_optimized entry point, CLI auto-detect + convert-s3-olci-optimized, real-product JSON fixture, golden-file snapshot, and verification/docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pyproject already pins zarr-cm>=0.4.1; update the lockfile from the git-main dev build to the released 0.4.1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…p type) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ion test Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add swath_spatial_attrs() function to return GeoZarr spatial: convention data for curvilinear swath geometry with pixel registration but no affine transform or bbox (geolocation carried by 2-D lat/lon arrays). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… DataTree omits overviews Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add structure-dump fixture (shapes shrunk to 16x16 / tie-points 16x4) from S3A_OL_1_EFR NT product fetched via HTTPS .zmetadata, add s3_olci_group_example conftest fixture, and extend test_s3_olci.py with is_sentinel3_olci_dataset and Sentinel3OlciRoot round-trip tests. Also update Sentinel3OlciMeasurementsMembers to include time_stamp and fix quality/conditions member types to allow nested GroupSpec children. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e time_stamp rows Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a Supported Products section to README.md and a dedicated Sentinel-3 OLCI L1 EFR Conversion section to docs/converter.md, covering: auto-detection via `eopf-geozarr convert`, the dedicated `convert-s3-olci-optimized` command with all key flags, output layout (measurements + r2/r4/... overviews + conditions/quality passthrough), and the v1 scope note (encoding wiring is a follow-up). Also add `# test: skip` to the first code block in the OLCI implementation plan doc so test_docs.py skips the non-runnable planning snippets. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s passthrough, fixture dims FIX 0 — reduce_swath: genuine 2×2 fill-aware block-average for radiance vars (OLCI_BANDS), stride-decimate for 2-D coord arrays (lat/lon/alt); replaces literal [::2,::2] decimation of radiance that added no information. FIX 1 — declare overviews as GeoZarr multiscales: build_convention_attrs now receives a MultiscalesAttrs layout (asset "." + "r2"/"r4"/… with relative scale=[2,2] transform) and writes the CMO to measurements attrs. FIX 2 — copy measurements/orphans through (was silently dropped because to_dataset() discards child groups); iterate dt_input["/measurements"].children and _copy_subtree each one. FIX 3 — fixture dim consistency: renamed tie-point 'columns' → 'tie_columns' in geometry/meteorology arrays, fixed orphan 2-D arrays from [16,16] to [16,4] (removed_pixels=4), fixed instrument 2-D arrays to [bands,detectors] at consistent sizes, fixed 3-D meteo arrays to [4,16,4], fixed nb_removed_pixels shape; xr.open_datatree now opens the whole tree cleanly. Snapshot regenerated (conditions, quality, measurements/orphans, r2 overview). FIX 4 — wire in OLCI_BANDS: is_sentinel3_olci_dataset uses OLCI_BANDS[0], reduce_swath identifies radiance vars via OLCI_BANDS frozenset. FIX 5 — remove --enable-sharding / --keep-scale-offset from README and docs/converter.md copy-paste example; keep them in flags table with "accepted but not yet wired" note. All 31 OLCI tests pass; pyright 0 errors; ruff clean; no typing.Any. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…vert tie_columns); regen snapshot Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…egen snapshot - CLI and snapshot test now open OLCI source with mask_and_scale=False so radiance stays uint16 with scale_factor/_FillValue in .attrs; CF decoding no longer silently strips scale/offset or widens dtype to float64. - Add _sanitize_data_vars() in olci_converter.py: strips _eopf_attrs, dtype, valid_min, valid_max from measurement data-var attrs before writing to GeoZarr store (both native and overviews); preserves _FillValue, scale_factor, add_offset, units, standard_name, coordinates. - Update sanitize_array_attrs() in conversion/utils.py: always strip _eopf_attrs/dtype/valid_min/valid_max; only strip _FillValue when is_decoded_float=True (raw-integer path keeps it in attrs for downstream fill handling). - Update _clear_encoding docstring: clarify converter expects raw (non-mask-scaled) input; only Zarr v2 codec encoding is stripped. - Regenerate snapshot: oa01_radiance dtype=uint16, scale_factor/add_offset/ _FillValue present, no _eopf_attrs/dtype/valid_min/valid_max. - Add _assert_radiance_dtype_and_attrs() regression guard in snapshot test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…CI-local attr sanitizer The OLCI work modified the shared sanitize_array_attrs in conversion/utils.py, changing S2/S1/generic output attrs and breaking 3 S2 golden-file snapshot tests. - Restore sanitize_array_attrs to c613e24 original: always strips _eopf_attrs + _FillValue; strips dtype/fill_value/valid_min/valid_max and rewrites digital_counts units only when is_decoded_float=True. - Add _sanitize_olci_array_attrs in olci_converter.py: strips _eopf_attrs/dtype/ valid_min/valid_max while preserving _FillValue (needed for raw uint16 data opened with mask_and_scale=False). - Replace sanitize_array_attrs usage in _sanitize_data_vars with the new helper. - Add unit test asserting _FillValue is preserved and stale keys are stripped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…mensions
reduce_swath used slice(None, None, factor) (i.e. [::factor]) for coordinate
arrays, which yields ceil(N/factor) elements when N is odd. coarsen with
boundary="trim" yields floor(N/factor). For real OLCI products with 4865
columns (odd), this caused a 1-element mismatch between radiance and coordinate
arrays in every overview group, making xr.open_dataset raise:
ValueError: conflicting sizes for dimension 'columns':
length 2433 on 'altitude' and length 2432 on 'oa01_radiance'
Fix: pre-compute dim_trim = (N // factor) * factor for each swath dimension
and change coordinate isel slices to slice(0, dim_trim, factor) so both paths
produce exactly floor(N/factor) elements for any N.
Add 4 unit tests (rows=7, cols=5) and 1 integration test (rows=10, cols=9)
that confirm consistent shapes across radiance and coordinate arrays after
reduction, and that overview groups open without conflicting-sizes errors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Notebook walks through opening an EOPF OLCI L1 EFR product (real EODC sample, with an offline fallback to the bundled fixture), detecting it, converting to a multiscale GeoZarr store, and visualizing the pyramid by splitting one field of view into four quadrants each rendered from a different overview level. Adds a `notebooks` dependency group (jupyter, matplotlib, nbformat). Executed end-to-end against the real product; outputs (incl. the quadrant figure) are saved in the notebook. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…r#199) * fix the types * refactor: build all conventions via zarr_cm.create_many Route multiscales, spatial, and proj convention metadata through a single utils.build_convention_attrs() helper that delegates to zarr_cm.create_many, so zarr-cm validates each convention and emits its CMO. Previously the CMOs were hand-placed and the spatial:/proj:/multiscales keys assembled by hand, which skipped zarr-cm validation (e.g. multiscales' layout>=1 and derived_from=>transform rules) and duplicated key strings. Type the helper precisely with zarr-cm's TypedDicts (SpatialAttrs, GeoProjAttrs, MultiscalesAttrs, MultiConventionAttrs) and a CRSLike Protocol instead of dict[str, Any]. Multiscales data is still produced by the project's MultiscaleMeta model (it also covers the TMS encoding zarr-cm doesn't model), but its CMO and validation now go through zarr-cm. Output is byte-identical to before (verified against the golden-file snapshot tests). Add tests/test_conversion/test_convention_attrs.py covering the helper, including that zarr-cm now rejects an invalid multiscales layout. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: gitignore local scratch files and .claude/ Ignore root-level scratch files (test.py, tmp.json, cli_test.sh) and the machine-local .claude/ session directory so they stop showing in git status. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Revert "chore: gitignore local scratch files and .claude/" This reverts commit 4a5b6e4. * fix type checking errors * wire up mypy correctly in pre-commit * build: migrate type checker mypy -> pyright; upgrade zarr-cm zarr-cm's convention TypedDicts now use PEP 728 `extra_items`, which mypy does not support. Switch the type checker to pyright (which does) and upgrade to the zarr-cm main git dep that also ships the supporting fixes: - Mapping-covariant aggregate API + exported JsonType/JsonValue/JsonDict, so build_convention_attrs no longer needs a private `_core` import or a cast. - PEP 695 `type JsonValue` alias, so pydantic resolves MultiscaleGroupAttrs' ConventionMetadataObject field directly (removed the model_rebuild workaround). Toolchain: - pyproject: replace [tool.mypy] with [tool.pyright]; mypy -> pyright dev dep. - .pre-commit-config / CI lint job: run `uv run --frozen pyright`. Type fixes across src/ and tests/ to reach a clean pyright run (0 errors): - remove stale mypy `# type: ignore[...]` comments (pyright-unnecessary); - narrow NotRequired TypedDict access (`.get()` + guard) instead of making keys Required, preserving runtime validation of optional Sentinel-1/2 members; - replace casts on external/untyped data with runtime isinstance/validation; - model construction via `Model.model_validate(...)` instead of `**dict`. No `typing.Any` introduced. Runtime behavior unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: replace unverified casts with runtime checks Three casts asserted a type derived from Any or a union without verifying it. Replace each with an isinstance guard that raises TypeError on violation, so the assumption is enforced at runtime instead of only asserted to the checker: - sentinel1_reprojection: rio.write_crs() returns Any -> verify xr.Dataset. - s2_multiscale: output_group[base_path] is Array | Group -> verify zarr.Group. - s2_multiscale: client.compute() returns Any -> verify distributed.Future. The remaining casts bridge types on data we already validated (model_dump / create_many output), satisfy protocol/TypeVar binding on `self`, or widen a TypedDict for a third-party API — a runtime check there adds no safety. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: use latest version of zarr-cm * fix: re-add CF _FillValue after attr sanitization in s2 multiscale encoding create_measurements_encoding injected the _FillValue=NaN attribute (the workaround for xarray issue #11345) before calling sanitize_array_attrs, which unconditionally strips _FillValue — so the workaround never reached the written dataset. Track the injection with a flag and apply it after sanitization, mirroring the ordering already used in geozarr.py. Also document in sanitize_array_attrs that _FillValue is always removed and that callers needing the CF attribute must re-add it after sanitizing. Assisted-by: ClaudeCode:claude-fable-5 * fix: preserve integer S1 nodata in encoding; document S2 CLI auto-routing Record the determined nodata in the reprojected S1 variable's encoding["_FillValue"]: the downstream attribute sanitization in setup_datatree_metadata_geozarr_spec_compliant strips _FillValue from attrs and only re-adds it for float variables, and explicit_fill_value reads from encoding, so integer nodata was lost from the output metadata. Also document in the convert subcommand help that Sentinel-2 inputs are auto-routed to the optimized flat multiscale layout and that --groups, --crs-groups, --gcp-group and --min-dimension do not apply there. Assisted-by: ClaudeCode:claude-fable-5 * feat: add --no-s2-optimized flag to force generic conversion path The convert command auto-detects Sentinel-2 inputs and routes them to the optimized flat multiscale layout, silently ignoring --groups, --crs-groups, --gcp-group and --min-dimension. Add a --no-s2-optimized escape hatch that disables the auto-detection so those options can still be honored for S2 inputs, and use it in test_cli_convert_with_crs_groups so the test exercises the crs-groups code path again instead of routing away from it. Also remove dead GCP handling in write_geozarr_group: the Sentinel-1 branch materialized dt_input[gcp_group].to_dataset() and immediately discarded it, since reprojection already happened upstream. Assisted-by: ClaudeCode:claude-fable-5 * docs: drop stale tile_width references from README The tile_width parameter was removed from create_geozarr_dataset along with the TMS-based multiscale layout, but the README examples and parameter list still mentioned it; copy-pasting them raised TypeError. Assisted-by: ClaudeCode:claude-fable-5 * fix: log multiscales creation failures at error level with traceback The broad except around create_geozarr_compliant_multiscales logged only a warning with str(e), silently swallowing defects in the rewritten ZCM layout / convention-validation logic. Keep the deliberate continue-with-next-group behavior but log via log.exception (error level, full traceback) so a malformed multiscales block is surfaced instead of hidden. Assisted-by: ClaudeCode:claude-fable-5 * chore: commit zarr-cm re-lock left out of previous commit The pyproject change to zarr-cm>=0.4.1 (e75b3b5) updated the dependency from the git rev to the PyPI release, but the corresponding uv.lock update was never committed. Assisted-by: ClaudeCode:claude-fable-5 * fix(deps): bump tornado to >=6.5.7 and msgpack to >=1.2.1 for CVEs pip-audit in the security workflow flags tornado 6.5.5 (CVE-2026-49853/49854/49855, GHSA-pw6j-qg29-8w7f; fixed in 6.5.7) and msgpack 1.1.2 (GHSA-6v7p-g79w-8964; fixed in 1.2.1). Both are transitive dependencies via distributed; raise the uv constraint-dependencies floors and re-lock. Assisted-by: ClaudeCode:claude-fable-5 * test: cover S1 member accessors, CLI S2 routing, and Dataset validators Raises patch coverage on this branch from ~34% to ~94%: - test_s1.py: reflectively exercise all 96 generated member-accessor properties on the S1 group models (returns member when present, KeyError when absent), plus the root polarization-group helpers. - test_cli_convert_routing.py: in-process tests for _is_sentinel2_input (detected / not detected / exception-swallowing) and convert_command dispatch (S2 auto-routing, --no-s2-optimized escape hatch, generic path for non-S2 inputs). - test_v2.py / test_v3.py: validate Dataset models end-to-end so the model_validator wrappers run, including the missing-grid-mapping failure branch. Assisted-by: ClaudeCode:claude-fable-5 --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # src/eopf_geozarr/conversion/geozarr.py # src/eopf_geozarr/s2_optimization/s2_multiscale.py # tests/test_data_api/test_v2.py # uv.lock
…iew rounding The OLCI auto-detect branch of convert_command passed the CF-decoded DataTree (mask_and_scale=True default) to convert_olci_optimized, whose documented precondition is raw uint16 radiance with scale_factor / _FillValue in attrs. Fill-aware masking in reduce_swath then saw no _FillValue, and np.round quantized decoded physical radiance in every overview level. Re-open the input with mask_and_scale=False in that branch, matching convert_s3_olci_optimized_command, and round in reduce_swath only when packing back into an integer dtype. Also include nested ancillary groups (e.g. conditions/geometry) in the DataTree returned by convert_olci_optimized — they were silently absent because only top-level groups with direct arrays were assembled — and add a routing test asserting the OLCI converter is selected and receives packed (non-CF-decoded) input. Assisted-by: ClaudeCode:claude-fable-5
… collisions The OLCI auto-detect branch of convert_command silently used the default min_dimension=256 instead of the user's --min-dimension; forward the flag so both entry points agree, and assert the forwarding in the routing test. In reduce_swath, a valid block whose mean rounds to the fill sentinel was recoded as fill in the overview; nudge such values one step back into the valid domain on the side the unrounded mean came from. Assisted-by: ClaudeCode:claude-fable-5
…anitizers Mirror the S2 escape hatch for the OLCI auto-detect path: a new --no-s3-olci-optimized flag forces the generic conversion path, with a routing test asserting the opt-out. The CF-decoded detection tree is now closed before re-opening the input raw, and the convert subcommand description documents the OLCI auto-routing. Rename the OLCI attr sanitizer to _sanitize_olci_array_attrs_keep_fill so its divergence from the shared sanitize_array_attrs (which strips _FillValue) is explicit at call sites, and cross-reference the two helpers in both docstrings so edits to one aren't assumed to apply to the other. Assisted-by: ClaudeCode:claude-fable-5
…ict detection The fill-collision nudge in reduce_swath could wrap for a sentinel at a dtype bound (e.g. fill 0 on an unsigned dtype nudging to -1 → dtype max). Clamp the nudge candidates so both sides resolve to the in-range neighbour when the sentinel sits at a bound. Document that OLCI auto-detection is intentionally conservative (strict structural validation against Sentinel3OlciRoot): near-miss products fall through to the generic path, and convert-s3-olci-optimized is the explicit bypass. Noted in both detection helpers and the convert subcommand description. Assisted-by: ClaudeCode:claude-fable-5
Working plan/design documents under docs/superpowers/ are local scaffolding, not project documentation; untrack them and gitignore the directory. The files remain on disk (and in history). Assisted-by: ClaudeCode:claude-fable-5
…allback Validate up front that the measurements group has the rows/columns swath dimensions the whole OLCI pipeline assumes, raising a clear ValueError instead of a bare KeyError deep in the converter. When return-tree assembly hits a DataTree dimension-consistency conflict, re-add groups greedily and exclude only the offending ones instead of collapsing to a measurements-only view. On the convert auto-detect path, warn when a user's --spatial-chunk or --enable-sharding is dropped (those options are not yet applied by the OLCI converter), instead of ignoring them silently. Assisted-by: ClaudeCode:claude-fable-5
…lback Pass skipna=True explicitly to the coarsen mean so the fill-aware promise (one fill pixel must not contaminate its block) does not hinge on a version-dependent default, and add a test asserting a partially filled block averages only its valid pixels. Sort the return-tree greedy fallback by path depth so parents are considered before their nested children, preventing an implicit empty parent from shadowing the real data-bearing one. Assisted-by: ClaudeCode:claude-fable-5
Truncate any pre-existing store before the first write so re-running convert_olci_optimized against the same output path cannot leave stale overview or ancillary groups from a prior run (the per-group writes were mode="w" scoped to measurements plus mode="a" appends). Tested by converting twice with different min_dimension values. Make swath detection in reduce_swath order-insensitive: a band stored transposed as (columns, rows) is normalized to (rows, columns) and fill-aware block-averaged instead of silently falling through to stride decimation. Document the origin-vs-center offset between decimated overview coordinates and block-averaged radiance in the reduce_swath docstring. Assisted-by: ClaudeCode:claude-fable-5
…book The quadrant visualization nearest-neighbour upsampled every overview back onto the native pixel grid, so nothing in the figure was actually shown at its stored resolution. Replace it with a 2x2 panel figure where each panel pcolormeshes one level's radiance against that level's own decimated 2-D latitude/longitude — same swath coverage per panel, visibly larger native cells per level, no resampling. Levels are picked dynamically from the finest affordable (<~600k quads) to the coarsest. Also make the offline fixture fallback independent of the kernel's working directory (nbconvert starts in docs/notebooks/), and silence consolidated-metadata warnings when opening levels. Assisted-by: ClaudeCode:claude-fable-5
…imated Overview coordinates were stride-decimated, labeling each block-averaged radiance cell with its block's top-left corner pixel — up to half a block (~77 km at r512) away from the centroid of the pixels actually averaged, and never converging to the swath center under repeated reduction. reduce_swath now reduces the 2-D latitude/longitude pair (matched by CF standard_name, falling back to variable name) jointly: pixel positions are mapped to unit vectors on the sphere, block-averaged with fill awareness (a pixel fill in either array is excluded from both), and the mean direction is converted back to degrees. Each overview cell is therefore geolocated at the geodesic centroid of its source block, stable across the antimeridian and at the poles, and collapsing the whole swath to one cell yields its geodesic center. Because the converter runs on un-decoded data and OLCI packs geolocation as int32 microdegrees (scale_factor=1e-06), values are CF-unpacked before the trigonometry and re-packed after, preserving dtype and attrs; feeding raw packed integers into the trig collapses every coordinate to ~(0, 0) (caught on the real EODC product, covered by regression test). Other non-band swath variables (e.g. altitude) keep stride decimation. Notebook re-executed against the real product; prose updated to describe the centroid coordinates. Assisted-by: ClaudeCode:claude-fable-5
Address both findings from roborev job 456: The DataTree returned by convert_olci_optimized was reopened with default CF decoding, handing callers float radiance while the store (and the converter's own input) holds packed uint16. Reopen with mask_and_scale=False and document the raw contract. Overview altitude was stride-decimated, sitting ~half a block from the geodesic-centroid lat/lon locating the same cell. Route altitude (matched by standard_name, falling back to name) through the same fill-aware block mean as radiance: it is linear, so the mean commutes with CF packing and is exact on raw values. Assisted-by: ClaudeCode:claude-fable-5
Short-circuit the opt-out flag first so an explicit --no-s3-olci-optimized skips the structural detection (a full model validation of the store) instead of running it and discarding the result. Roborev job 457, finding 1. Assisted-by: ClaudeCode:claude-fable-5
…r a map Replace the four-panel figure with a single cartopy PlateCarree map: the swath is split into four geographic quadrants and each quadrant is drawn from a different pyramid level via xarray's curvilinear plot.pcolormesh against that level's own centroid lat/lon, over coastlines, borders, and labeled gridlines with a shared robust color scale. The seamless tiling across quadrant boundaries makes the geodesic-centroid coordinate scheme visible: levels agree on where things are even though every quadrant is a different stored grid. Adds cartopy to the notebooks dependency group. Uses explicit set_xticks/set_yticks with cartopy's Longitude/LatitudeFormatter for axis labels: Gridliner auto-labels (draw_labels=True) blank the whole GeoAxes when a colorbar axes is present under cartopy 0.25 with matplotlib 3.11. Assisted-by: ClaudeCode:claude-fable-5
…atomic zarr CacheStore (EOPF-Explorer#220) Some S2 conversions fail with "RuntimeError: error during blosc decompression: -1". Root cause is an fsspec bug open since 2021 (fsspec/filesystem_spec#639): simplecache writes a download straight to its final cache filename, so a concurrent read of the same key can observe a half-written file. Products whose quality/atmosphere aot/wvp are stored as one whole-array 10980x10980 chunk trigger it because dask (chunks="auto") splits that single object into ~9 read tasks fetching it concurrently (EOPF-Explorer/data-pipeline#339). open_source_datatree() (conversion/open_source.py) is the safe way to open a source: - opens with dask chunks matching the on-disk zarr chunks, so each object is fetched by exactly one task (removes the race trigger and the ~9x redundant downloads), and - an optional cache_dir replaces the pipeline's simplecache layer with zarr's built-in CacheStore over a LocalStore (suggested by @d-v-b in EOPF-Explorer/data-pipeline#339). LocalStore writes atomically (temp file + rename), so the half-written-file race cannot occur. Cache entries are namespaced per source URL to prevent cross-product collisions. The convert-s2-optimized CLI command now opens sources through this helper (generic convert/info/validate open sites are left for a follow-up). zarr is bounded to >=3.2.0,<3.3.0 while we depend on the experimental CacheStore API; the import is lazy and guarded so only the cache_dir path is affected if zarr moves it. History note: this branch previously carried a hand-rolled atomic fsspec cache (atomiccache) and an output chunk clamp; the former was replaced by the CacheStore wiring, the latter is split out to its own PR. Synthesized from commits acfd142, f950067, cf1a5d2, cd3eb00, da2f2c1. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port EOPF-Explorer#220 to the OLCI converter paths: both OLCI source opens (convert routing re-open and convert-s3-olci-optimized) used chunks="auto", which lets dask sub-split stored zarr chunks into multiple read tasks — duplicate egress/decompression and cache races under EOPF-Explorer/data-pipeline#339. open_source_datatree gains a mask_and_scale passthrough since the OLCI converter requires raw packed input. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…port chore: merge main into feat/sentinel3-export
fix(s3-olci): open OLCI sources with native-chunk-aligned reads
… opens with xr.open_datatree The converter previously wrote the full-resolution arrays directly in measurements/ with overview groups (r2, r4, ...) nested beneath, declared as "asset": "." in the multiscales layout. Overview children then inherited the parent's 2-D latitude/longitude coordinates over the shared rows/columns dims at mismatched sizes, so xr.open_datatree — and any generic GeoZarr reader, e.g. titiler — rejected the store with an alignment error (reported on PR EOPF-Explorer#212 from a real S3A scene). Native-resolution arrays now live in measurements/r0 as a named sibling of the overview groups (the Sentinel-2 pattern); measurements/ itself carries only the multiscales/spatial convention metadata, with the layout referencing "r0" instead of ".". This makes the hand-built DataTree workaround (which silently omitted overview children) unnecessary: convert_olci_optimized now returns xr.open_datatree(output) directly, and a new acceptance test pins that the whole exported store opens cleanly. Snapshot regenerated; docs and demo notebook updated and re-executed against the real EODC product. Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
… contract Assisted-by: ClaudeCode:claude-fable-5
…n arrays Assisted-by: ClaudeCode:claude-sonnet-5
… error Float variables with no pre-existing _FillValue warped to NaN nodata but were skipping the _FillValue attr, breaking the "every warped variable records its fill" contract. Always write _FillValue on the warped path (NaN included for float dtypes) and clarify the docstring to scope the guarantee to warped variables only. Also add the missing test for the 2-D latitude/longitude shape-mismatch ValueError, per the one-test-per-error-case convention. Assisted-by: ClaudeCode:claude-sonnet-5
… gridded pyramids
Add keyword-only dims parameter to decimate_swath and reduce_swath,
defaulting to SWATH_DIMS=("rows", "columns") to preserve existing behavior.
Enables Task 4 to create pyramids on reprojected grids with dims ("y", "x").
Assisted-by: ClaudeCode:claude-haiku-4-5
…form Assisted-by: ClaudeCode:claude-haiku-4-5
…per-level CRS Warp the swath once after sanitize via reproject_olci, build the r0/r2/... pyramid on the grid with reduce_swath(dims=(y, x)), and write per-level spatial/geo-proj convention attrs plus multiscales on the parent. Replace the degenerate collinear lat/lon in the synthetic test fixture with a real axis-aligned grid and restructure size assertions relative to the observed warped r0. swath_spatial_attrs is deleted (replaced by grid_spatial_attrs). The golden-snapshot test intentionally fails until the next commit seeds real geolocation into the zero-filled fixture. Assisted-by: ClaudeCode:claude-sonnet-5
…t docs and notebook Seed real geolocation into the zero-filled golden fixture inside the snapshot test (the warp rejects degenerate extents), regenerate the snapshot for the gridded layout, add --target-crs to the CLI subcommand, update README/converter.md for the reprojected output, and re-execute the demo notebook against the real EODC product. Mark the implementation-plan doc's code blocks test-skip so pytest-examples does not execute them. Assisted-by: ClaudeCode:claude-sonnet-5
…rid with CRS Adds tests/test_geozarr_output_contract.py, a parametrized test over all three converters (OLCI, S1, S2) asserting the hard requirement that every multiscale level is a regular grid with a declared, pyproj-round-trippable CRS. OLCI passes with no marks. S1/S2 whole-store open_datatree is xfailed (legacy asset='.' nested layout). S1/S2 per-level CRS detection is also xfailed: the generic converter (geozarr.py) only stamps zarr-cm proj:/ spatial: convention attrs on the base-resolution group, not on overview sub-groups, so rioxarray can't auto-detect CRS there on a plain xr.open_dataset read. Also extracts the Sentinel-2 fixture body into an importable build_sample_sentinel2_datatree() so the contract test can build the same sample tree without needing the pytest fixture machinery. Assisted-by: ClaudeCode:claude-sonnet-5
Assisted-by: ClaudeCode:claude-fable-5
…/S2 xfail Opening each pyramid level with the default xarray zarr decode never promotes a CF grid_mapping/spatial_ref reference into a coordinate, so ds.rio.crs came back None for S1/S2 overview levels and I wrongly attributed that to a converter gap and added an xfail. The converter's own read path already uses decode_coords="all" for exactly this reason (geozarr.py:1132, :1541). Passing the same flag in the test makes CRS detection succeed for every product's every level (S2 r2 -> EPSG:32632, S1 r2 -> EPSG:4326), so REGULAR_GRID_XFAIL is removed entirely - test_every_level_is_regular_grid_with_crs now genuinely passes for olci/s1/s2 with no marks. DATATREE_XFAIL (asset='.' nested layout) is unrelated and unchanged. Assisted-by: ClaudeCode:claude-sonnet-5
Assisted-by: ClaudeCode:claude-fable-5
… refinements Integrates ~25 upstream commits with the reprojection branch. Resolution policy (maintainer-approved): the reprojected-grid design wins; upstream work invested in swath-native overview geolocation (geodesic block centroids, swath map plots) is dropped as superseded by the warp, while orthogonal improvements are preserved: store truncation on re-run, fill-collision nudge clamped to dtype range, NaN-aware fill masking, explicit skipna, single materialization of the coarsened mean, block-averaged altitude, transposed-band normalization, raw mask_and_scale=False return-tree semantics, detection hardening, CLI --no-s3-olci-optimized opt-out and --min-dimension forwarding, and the docs/superpowers gitignore policy (new plan/spec docs untracked to match). Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Sentinel-3 OLCI L1 EFR → GeoZarr exporter — the first Sentinel-3 product type — modeled on the existing Sentinel-2 exporter and adapted to OLCI's curvilinear swath geometry.
Design:
docs/superpowers/specs/2026-06-21-sentinel3-olci-export-design.md· Plan:docs/superpowers/plans/2026-06-21-sentinel3-olci-export.mdWhat it does
latitude/longitude. The exporter preserves that (CF 2-D coordinates), and emits the GeoZarrspatialconvention with nospatial:transform./2pyramid via fill-aware 2×2 block averaging of the 21uint16radiance bands (scale_factor/_FillValuepreserved); coordinate arrays are decimated (real measured positions, not interpolated). Levels are declared with the GeoZarrmultiscalesconvention (relativetransformonly).measurements/(21 bands + 2-D geolocation) becomes the multiscale group;conditions/quality/orphansare copied through faithfully.is_sentinel3_olci_dataset) + CLI auto-detect inconvert+ dedicatedconvert-s3-olci-optimizedcommand.New code
src/eopf_geozarr/s3_olci_optimization/—olci_band_mapping,olci_multiscale(reduce_swath,swath_spatial_attrs),olci_converter(convert_olci_optimized, detection).src/eopf_geozarr/data_api/s3_olci.py—Sentinel3OlciRootmodel.tests/— band-mapping, multiscale (incl. odd-dimension reduction), data-api/detection, integration + golden-file snapshot; real-product structure-dump fixture intests/_test_data/s3_examples/.docs/notebooks/sentinel3_olci_geozarr.ipynb— end-to-end demo (open EOPF OLCI → convert → 4-quadrant multiscale visualization), executed against the real EODC sample product. Adds anotebooksdep group.README.md/docs/converter.md— OLCI usage.Verification
mask_and_scale=False; a shared-sanitize_array_attrsregression that affected S2 — reverted, S2 snapshot byte-unchanged; odd-column overview size mismatch found by running the notebook on real 4865-col data).typing.Anyin new code; S1/S2/generic paths unaffected.Deferred (documented follow-ups)
keep_scale_offset/enable_sharding/spatial_chunk/compression_levelaccepted but not yet wired into encoding.🤖 Generated with Claude Code