fix(s3-olci): open OLCI sources with native-chunk-aligned reads - #4
Merged
d-v-b merged 5 commits intoJul 17, 2026
Merged
Conversation
…atomic zarr CacheStore (#220) Some S2 conversions fail with "RuntimeError: error during blosc decompression: -1". Root cause is an fsspec bug open since 2021 (fsspec/filesystem_spec#639): simplecache writes a download straight to its final cache filename, so a concurrent read of the same key can observe a half-written file. Products whose quality/atmosphere aot/wvp are stored as one whole-array 10980x10980 chunk trigger it because dask (chunks="auto") splits that single object into ~9 read tasks fetching it concurrently (EOPF-Explorer/data-pipeline#339). open_source_datatree() (conversion/open_source.py) is the safe way to open a source: - opens with dask chunks matching the on-disk zarr chunks, so each object is fetched by exactly one task (removes the race trigger and the ~9x redundant downloads), and - an optional cache_dir replaces the pipeline's simplecache layer with zarr's built-in CacheStore over a LocalStore (suggested by @d-v-b in EOPF-Explorer/data-pipeline#339). LocalStore writes atomically (temp file + rename), so the half-written-file race cannot occur. Cache entries are namespaced per source URL to prevent cross-product collisions. The convert-s2-optimized CLI command now opens sources through this helper (generic convert/info/validate open sites are left for a follow-up). zarr is bounded to >=3.2.0,<3.3.0 while we depend on the experimental CacheStore API; the import is lazy and guarded so only the cache_dir path is affected if zarr moves it. History note: this branch previously carried a hand-rolled atomic fsspec cache (atomiccache) and an output chunk clamp; the former was replaced by the CacheStore wiring, the latter is split out to its own PR. Synthesized from commits acfd142, f950067, cf1a5d2, cd3eb00, da2f2c1. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port #220 to the OLCI converter paths: both OLCI source opens (convert routing re-open and convert-s3-olci-optimized) used chunks="auto", which lets dask sub-split stored zarr chunks into multiple read tasks — duplicate egress/decompression and cache races under EOPF-Explorer/data-pipeline#339. open_source_datatree gains a mask_and_scale passthrough since the OLCI converter requires raw packed input. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
d-v-b
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Only commit 5ea5662 belongs to this PR (3 files, +31/−12) — the other commits are #3 (merge of main), which should be merged first; the extra commits then drop out of this diff.
In this PR open_source_datatree gains a mask_and_scale passthrough since the OLCI converter requires raw packed
input.
Author attestation
AI-assisted: implemented and tested with Claude Code; reviewed by me.