Summary
All /query endpoints on the production flywire_fafb_public datastack
have been returning HTTP 500 for at least 16+ hours (first
observed 2026-05-28 ~17:00 UTC, still failing 2026-05-29 09:26 UTC).
The error is the same on every request:
{"code": 500,
"message": "module 'pyarrow' has no attribute 'default_serialization_context'"}
pyarrow.default_serialization_context was removed in pyarrow 12.0 (~2 years
ago, pyarrow #ARROW-14625).
The deployed service appears to be running a pyarrow version that no
longer has the symbol but code that still calls it.
Codex (the user-facing UI) is not affected, so this likely went
unnoticed — only the REST query API is broken.
Reproducer
Plain curl against any table, any column-selection, any limit, all
return the same 500:
curl -X POST \
-H "Authorization: Bearer $FLYWIRE_TOKEN" \
-H "Content-Type: application/json" \
-d '{"limit": 1}' \
"https://prod.flywire-daf.com/materialize/api/v3/datastack/flywire_fafb_public/version/783/table/proofread_neurons/query"
The same call worked fine yesterday (downloaded 120,519,649 rows of
synapses_nt_v1 to completion via caveclient 8.0.1).
What still works
GET https://prod.flywire-daf.com/materialize/ → 302
GET /api/v3/datastack/flywire_fafb_public/versions → 200 [630, 783]
GET https://codex.flywire.ai/ → 200
What doesn't
| Endpoint |
Result |
POST /api/v3/.../table/<t>/query |
500 pyarrow.default_serialization_context |
POST /api/v2/.../table/<t>/query |
500 'ipc_compress' |
POST /api/v3/.../table/<t>/simple_query |
404 |
?arrow_format=False |
ignored, same 500 |
return_pyarrow: false in body |
500 schema error: "Unknown field" |
All four tables I've tried fail the same way:
proofread_neurons, synapses_nt_v1, fly_synapses_neuropil_v6,
hierarchical_neuron_annotations.
Client side
- caveclient
8.0.1
- Python 3.14.5
Summary
All
/queryendpoints on the productionflywire_fafb_publicdatastackhave been returning HTTP 500 for at least 16+ hours (first
observed 2026-05-28 ~17:00 UTC, still failing 2026-05-29 09:26 UTC).
The error is the same on every request:
pyarrow.default_serialization_contextwas removed in pyarrow 12.0 (~2 yearsago, pyarrow #ARROW-14625).
The deployed service appears to be running a pyarrow version that no
longer has the symbol but code that still calls it.
Codex (the user-facing UI) is not affected, so this likely went
unnoticed — only the REST query API is broken.
Reproducer
Plain
curlagainst any table, any column-selection, any limit, allreturn the same 500:
The same call worked fine yesterday (downloaded 120,519,649 rows of
synapses_nt_v1to completion via caveclient 8.0.1).What still works
GET https://prod.flywire-daf.com/materialize/→ 302GET /api/v3/datastack/flywire_fafb_public/versions→ 200[630, 783]GET https://codex.flywire.ai/→ 200What doesn't
POST /api/v3/.../table/<t>/querypyarrow.default_serialization_contextPOST /api/v2/.../table/<t>/query'ipc_compress'POST /api/v3/.../table/<t>/simple_query?arrow_format=Falsereturn_pyarrow: falsein bodyAll four tables I've tried fail the same way:
proofread_neurons,synapses_nt_v1,fly_synapses_neuropil_v6,hierarchical_neuron_annotations.Client side
8.0.1