Sub-task of #479. Write side, zarr backend — easiest, do first.
Goal
Let the zarr writer target any fsspec URL (file://, s3://, gs://, http://) instead of being pinned to a local store, and write per sample so peak local disk drops to ~1 sample.
Current state (verified on main)
zarr/writer.py iterates write_sample per sample already, but the store is a concrete zarr.storage.LocalStore.
- The per-sample loop is the right hook; only the store selection is hardcoded.
Scope
Notes / traps
Tests
- Round-trip on
memory:// (write then read) so it is CI-testable without network.
Priority
★ (1/5) — smallest, highest value, unblocks the incremental-write use case from #478/#479.
Sub-task of #479. Write side,
zarrbackend — easiest, do first.Goal
Let the
zarrwriter target any fsspec URL (file://,s3://,gs://,http://) instead of being pinned to a local store, and write per sample so peak local disk drops to ~1 sample.Current state (verified on
main)zarr/writer.pyiterateswrite_sampleper sample already, but the store is a concretezarr.storage.LocalStore.Scope
storage_options(fsspec-aware) instead of pinningLocalStore.s3fs/gcsfs/…) behind a[remote]extra with a clear error when the handler is missing (per fsspec-aware save_to_disk: stream samples to the target during writing (avoid full local materialization before remote push) #479 open questions).Notes / traps
num_proc > 1(independent, process-safe handles per worker) — see fsspec-aware save_to_disk: stream samples to the target during writing (avoid full local materialization before remote push) #479.Tests
memory://(write then read) so it is CI-testable without network.Priority
★ (1/5) — smallest, highest value, unblocks the incremental-write use case from #478/#479.