Skip to content

[fsspec-aware #479] common: remote-aware load_constants_from_disk (read, priority 3) #487

Description

@xroynard

Sub-task of #479. Read side, transverse metadata layer — medium. Covers part ③ of the read-side analysis (issuecomment-5166383239).

Goal

Allow common/reader.py::load_constants_from_disk to read constants from a remote target.

Current state (verified on main)

  • Backed by np.memmap over data.mmap (load_constants_from_disk, l.172).
  • np.memmap requires a real seekable local file — passing an fsspec file handle fails with UnsupportedOperation: fileno (checked empirically).
  • _materialize_memmaps already exists to copy memmaps into plain in-memory arrays, but the source still has to be a local file.

Constraint

Remote constants cannot be read in place. The data.mmap bytes must land on local disk (temp) first, then np.memmap + _materialize_memmaps as today.

Scope

  • When the source is remote (fsspec), stage data.mmap (and companions) to a local temp file, memmap from there, materialize, then discard the temp.
  • Keep the local fast path unchanged (no temp copy when already local).

Tests

  • memory:// source: verify constants load correctly via the staging path.

Priority

★★ (3/5) — transverse blocker shared by all backends on the read side; needed before cgns remote read is meaningful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions