Skip to content

[fsspec-aware #479] zarr: make save_to_disk write path fsspec-aware (write, priority 1) #485

Description

@xroynard

Sub-task of #479. Write side, zarr backend — easiest, do first.

Goal

Let the zarr writer target any fsspec URL (file://, s3://, gs://, http://) instead of being pinned to a local store, and write per sample so peak local disk drops to ~1 sample.

Current state (verified on main)

  • zarr/writer.py iterates write_sample per sample already, but the store is a concrete zarr.storage.LocalStore.
  • The per-sample loop is the right hook; only the store selection is hardcoded.

Scope

Notes / traps

Tests

  • Round-trip on memory:// (write then read) so it is CI-testable without network.

Priority

★ (1/5) — smallest, highest value, unblocks the incremental-write use case from #478/#479.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions