Skip to content

add zarr tests#4228

Open
MMesch wants to merge 1 commit into
emscripten-forge:mainfrom
MMesch:mmesch/zarr-tests
Open

add zarr tests#4228
MMesch wants to merge 1 commit into
emscripten-forge:mainfrom
MMesch:mmesch/zarr-tests

Conversation

@MMesch

@MMesch MMesch commented Jan 12, 2026

Copy link
Copy Markdown
Contributor

Basic zarr tests as in this PR are currrently failing. I'm opening to iterate on it and get it running. Currently this is based on the latests zarr.

Would be itnerest to hear feedback from @DerThorsten and @davidbrochart .

@davidbrochart

davidbrochart commented Jan 12, 2026

Copy link
Copy Markdown
Collaborator

Zarr uses threads to do non-blocking disk I/O, so that can't work in WASM. I don't know if there is a way to configure it so that it doesn't use threads.
I ported watchfiles to WASM in anychange, it would be a similar change.

@MMesch

MMesch commented Jan 12, 2026

Copy link
Copy Markdown
Contributor Author

What does work is:

dependencies:
  - zarr < 3
  - numcodecs < 0.16

Can we expect any problems using zarr < 3?

@davidbrochart

davidbrochart commented Jan 12, 2026

Copy link
Copy Markdown
Collaborator

It looks like in v3 they use a thread pool executor internally, which was not the case in v2.
I don't know if we can still use v2. At least v3 mentions pyodide compatibility: https://github.com/zarr-developers/zarr-python/releases/tag/v3.0.0.

@MMesch

MMesch commented Jan 12, 2026

Copy link
Copy Markdown
Contributor Author

zarr > 3 issue on pyodide pyodide/pyodide-recipes#227

@MMesch

MMesch commented Jan 12, 2026

Copy link
Copy Markdown
Contributor Author

recipe is pinned to <v3 https://github.com/pyodide/pyodide-recipes/blob/main/packages/zarr/meta.yaml

@DerThorsten we should maybe do the same?

@agriyakhetarpal

agriyakhetarpal commented May 17, 2026

Copy link
Copy Markdown

Hi! You may refer to zarr-developers/zarr-python#1903 and pyodide/pyodide-recipes#584 for Zarr v3 support.

@MMesch

MMesch commented May 18, 2026

Copy link
Copy Markdown
Contributor Author

nice, will have a look.

@MMesch
MMesch force-pushed the mmesch/zarr-tests branch from 2851381 to 054391e Compare July 17, 2026 20:23
@MMesch

MMesch commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Reset this branch to a fresh cut at zarr v3.

  • zarr 3.1.3 → 3.2.1
  • Ported the three WASM patches from pyodide-recipes#584 (upstream zarr-python#1903, still open)
  • Swapped crc32c → google-crc32c (zarr 3.2 imports it directly), dropped unused asciitree
  • Tests: import still passes, three sync-API roundtrips are xfail

The reason for the xfails, and a question for you @DerThorsten: zarr 3's sync API seems to need JSPI. Without it every sync call blows up with:

RuntimeError:
Cannot use synchronous Zarr API in browser-based environments without JSPI.
Zarr requires JavaScript Promise Integration (JSPI) to block for asynchronous
operations, which is not supported or enabled in your current environment.
...

Pyodide flipped JSPI on by default in 0.28. Pyjs's CMakeLists doesn't seem to have -sJSPI (or -sASYNCIFY). Could we turn it on? Would that raise the browser floor much? Happy to open an issue on pyjs and prototype if useful.

Once JSPI is around the xfails should flip to XPASS on their own.

@MMesch
MMesch changed the base branch from emscripten-3x to main July 17, 2026 20:28
@MMesch
MMesch force-pushed the mmesch/zarr-tests branch from 054391e to 54d8981 Compare July 17, 2026 20:28
@MMesch
MMesch marked this pull request as ready for review July 20, 2026 08:24
@MMesch
zarr: 3.1.3 -> 3.2.1, port pyodide WASM patches
18f7532 
Bumps zarr to 3.2.1 (was 3.1.3) and vendors the three downstream
patches from pyodide-recipes PR emscripten-forge#584 (upstream zarr-python emscripten-forge#1903, still
open):
- src/zarr/_constants.py + core/sync.py: WASM branch that defers to the
  running event loop and raises a JSPI-explaining RuntimeError if
  synchronous blocking is not available
- core/config.py: threading.max_workers=1 on WASM (single-threaded runtime)
- storage/_local.py: WASM _safe_move that avoids os.link and does a manual
  dst.exists() check before os.rename

Runtime deps: replace crc32c with google-crc32c (zarr 3.2 imports
google_crc32c directly), drop asciitree (unused in v3).

Tests: keep basic import; add three sync-API roundtrip tests marked
xfail with reason "requires JSPI, not enabled in pyjs runtime" — they
will start XPASSing once JSPI is available.

Patch files carry the pyodide-recipes MPL-2.0 header and upstream
attribution to Agriya Khetarpal / zarr-python#1903.
@MMesch
MMesch force-pushed the mmesch/zarr-tests branch from 54d8981 to 18f7532 Compare July 20, 2026 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@MMesch @davidbrochart @agriyakhetarpal