Skip to content

HTTP 500 on every /query endpoint: pyarrow.default_serialization_context removed in pyarrow ≥12 (production flywire_fafb_public down) #229

@luisan00

Description

@luisan00

Summary

All /query endpoints on the production flywire_fafb_public datastack
have been returning HTTP 500 for at least 16+ hours (first
observed 2026-05-28 ~17:00 UTC, still failing 2026-05-29 09:26 UTC).
The error is the same on every request:

{"code": 500,
 "message": "module 'pyarrow' has no attribute 'default_serialization_context'"}

pyarrow.default_serialization_context was removed in pyarrow 12.0 (~2 years
ago, pyarrow #ARROW-14625).
The deployed service appears to be running a pyarrow version that no
longer has the symbol but code that still calls it.

Codex (the user-facing UI) is not affected, so this likely went
unnoticed — only the REST query API is broken.

Reproducer

Plain curl against any table, any column-selection, any limit, all
return the same 500:

curl -X POST \
  -H "Authorization: Bearer $FLYWIRE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"limit": 1}' \
  "https://prod.flywire-daf.com/materialize/api/v3/datastack/flywire_fafb_public/version/783/table/proofread_neurons/query"

The same call worked fine yesterday (downloaded 120,519,649 rows of
synapses_nt_v1 to completion via caveclient 8.0.1).

What still works

  • GET https://prod.flywire-daf.com/materialize/ → 302
  • GET /api/v3/datastack/flywire_fafb_public/versions → 200 [630, 783]
  • GET https://codex.flywire.ai/ → 200

What doesn't

Endpoint Result
POST /api/v3/.../table/<t>/query 500 pyarrow.default_serialization_context
POST /api/v2/.../table/<t>/query 500 'ipc_compress'
POST /api/v3/.../table/<t>/simple_query 404
?arrow_format=False ignored, same 500
return_pyarrow: false in body 500 schema error: "Unknown field"

All four tables I've tried fail the same way:
proofread_neurons, synapses_nt_v1, fly_synapses_neuropil_v6,
hierarchical_neuron_annotations.

Client side

  • caveclient 8.0.1
  • Python 3.14.5

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions