Skip to content

[fsspec-aware #479] zarr: make init_from_disk read path fsspec-aware (read, priority 2) #486

Description

@xroynard

Sub-task of #479. Read side, zarr backend — easy, mirror of the write-side zarr task. Covers parts ① + ② of the read-side analysis (issuecomment-5166383239).

Goal

Let zarr/reader.py::init_datasetdict_from_disk read from any fsspec URL, not just local disk.

Current state (verified on main)

zarr.open(zarr.storage.LocalStore(local_path / sn), mode="r")

Two hardcoded-local spots in the same function:

  1. Store — pinned to LocalStore. zarr.open() accepts an fsspec URL directly in read mode (checked: zarr.open("memory://…", mode="r") round-trips), so this is just selecting the store from the path/storage_options.
    • ⚠️ Trap: LocalStore("s3://…") does not raise — it treats the string as a literal local path and reads the wrong place. This is a correctness fix, not an exception to guard.
  2. Split discoverylocal_path.iterdir() (is_dir() filtering) is local-only; on a remote target it must go through fs.ls(...) on the fsspec filesystem derived from the URL.

Scope

  • Select the store from the URL/storage_options.
  • Replace iterdir() split discovery with fsspec fs.ls(...).

Tests

  • Round-trip on memory:// (write via the zarr writer, read back via init_from_disk).

Priority

★ (2/5) — small, high value, testable on memory://; symmetric with the write-side zarr task.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions