[blob-store 4/5] DuckDB read layer + Parquet publish for /v3 (demonstrative)

[chondl]

Track 2 · blob-store stack — PR 4 of 5 (demonstrative, behind API_BACKEND=duckdb; zero default change). Base: state-snapshot (#9); merge after #9. Stack order: #2 → #7 → #9 → #10 → #11. No cross-track conflicts.

---

Stacked on #9 (state-snapshot). Base state-snapshot, head duckdb-api, fork-only. Draft: the default backend is unchanged, so this is zero-risk to merge and purely demonstrative.

This shows the two moves that let us drop CockroachDB entirely: (1) publish the season as Parquet alongside the existing blobs every cycle, and (2) serve the public /v3 API from an in-process DuckDB reading those Parquet files instead of the relational DB. Selection is behind API_BACKEND=duckdb; unset, everything runs exactly as before.

Why each change

Add Parquet/DuckDB table schema introspection (src/db_duckdb/schema.py). The seven entity tables already describe themselves through SQLAlchemy. Rather than hand-maintain column lists, this derives per-column kind (int/float/bool/str/enum/json) from the ORM, so the Parquet writer and the DuckDB reader share one source of truth and stay in lockstep with the models. Enum columns (all str, Enum) are stored as their value; the two JSON columns (team_year.matches, match.pre_epas/epas) as JSON text, coerced back on read so DuckDB-loaded objects are byte-identical to DB-loaded ones. Every SQLAlchemy Integer-family column maps to Parquet int64 — deliberately wider than the ORM's declared Integer (int32) on match.time/predicted_time, event.time, team_event.time, because TBA's ~1900-era placeholder timestamps (-2208988800) underflow int32; a historical load on a strict-int32 backend crashes on exactly this (CockroachDB's 64-bit INT masked it in prod). Verified by round-tripping a synthetic pre-1970 placeholder row: writer -> Parquet (physical int64) -> DuckDB -> model, value preserved exactly. And since BigInteger subclasses Integer, a future ORM Integer->BigInteger fix for those columns changes nothing in the Parquet schema.

Publish the Parquet tables as a manifest-referenced atomic set (src/google/parquet.py, wired in src/data/main.py). Built from the same in-memory objs tuple the snapshot and blobs already use — no extra query. Each of team_years, events, team_events, matches, teams, year is serialized with a columnar pyarrow build and written content-addressed under the same v2/ prefix the versioned site blobs use (v2/parquet/{year}/{table}.parquet.{sha}, immutable long Cache-Control), then recorded in manifest.json under a parquet/{year}/{table}.parquet logical key. For the current-year cycle, the parquet entries are folded into the same, single manifest.json write the site blobs already do (storage.write_objs takes the serialized parquet tables and adds them to its one plan/manifest, written last). So the whole cycle writes the manifest exactly once — verified on the rig: one write_manifest call carrying both the site blobs and the parquet/* entries. This closes the earlier two-write window (site manifest, then a second parquet manifest) where a DuckDB sync landing between the writes served the previous cycle's Parquet while the frontend served the current one, and a crash between them stranded the API a cycle behind. Historical years (reset_all_years backfill) have no co-occurring site publish, so they still write parquet on their own manifest (also last). (The even earlier scheme wrote fixed paths and refreshed each file independently by GCS generation, so a reader syncing mid-publish could pair matches from one run with events from another.) Content-gating is against the manifest digest, so an unchanged table uploads nothing (steady-state = 0 uploads).

A related safety fix: the parquet manifest update no longer does read_manifest() or Manifest(). A transient GCS read failure previously fabricated an empty manifest and wrote back a manifest containing only parquet/* keys — erasing every site-blob reference until the next full re-render. The current-year cycle now can't hit this (parquet rides the site manifest, which is rebuilt from all rendered blobs each cycle, so a failed read just rebuilds fully); the historical writer aborts and logs if the manifest is unreadable rather than clobbering it. Folding into manifest.blobs keeps the reference-aware GC (#7) and the frontend unchanged (both key off blobs).

Add DuckDB read layer over the Parquet set (src/db_duckdb/). Same twelve get_* interfaces the /v3 routers call from src/db/read. An in-process DuckDB queries a local cache resolved from a single manifest.json fetch per sync: the referenced Parquet objects are materialized into a cache dir versioned by manifest generation (unchanged objects reused by hardlink), then the active dir is swapped atomically, so an in-flight query never sees a half-updated set and each query captures its base dir once (a 30s throttle avoids per-request manifest reads). Year-scoped queries read a single file; cross-year queries glob parquet/*/{table}.parquet (the design doc's Query Layer). Sort metric is validated against the column whitelist; every filter is parameterized. Cold start / bootstrap: when no Parquet exists yet (first-ever deploy, before the first cycle), every entity endpoint degrades to an empty result — including team endpoints, whose _teams_source falls back to a glob instead of crashing on max([]). So the read layer 200s with empty data rather than 500ing until the first pipeline cycle writes Parquet.

Serve /v3 from DuckDB behind API_BACKEND flag (src/api/backend.py + the six routers). A one-line facade dispatches to src.db_duckdb or src.db.read at import time. Only the public /v3 routers switch; /v3/site and the pipeline stay on the DB. Default is the relational DB.

Parity (rig, seeded 2026)

In-process, comparing db.read.get_*(...).to_dict() against db_duckdb.get_*(...).to_dict() across a representative sweep (team, teams, year, team_year, team_years with filters/sort/pagination, event, events, team_event, team_events, match, matches with filters):

	value
Fields compared	67,684
Field-level value mismatches	0
teams(country=USA, active) full set	DB 2944 == DuckDB 2944, 0 field mismatches
team_years(2026, country=USA) full set	DB 2944 == DuckDB 2944, 0 field mismatches

The only response-level differences are the order of equal-sort-key rows in paginated queries (ORDER BY <metric> with no unique tiebreaker — the current API contract, engine-agnostic). Confirmed by re-sorting the full result by (metric, pk): 0 field mismatches. The effect scales with how tied the metric is — norm_epa has 2490/2944 duplicate values (integer-rounded) so most pages differ; epa_max (rarely tied) → 0 pages differ. Not a DuckDB defect; the DB shows the same under a plan change.

Shared smoke suite in duckdb mode (--base-url → DuckDB-backed server): 9/9 pass, consistency probe max EPA diff 0.0000.

Latency (rig, alru-cache bypassed via varied params, warmed; ms)

endpoint class	db p50	db p95	dk p50	dk p95
team/{n}	5.8	8.1	1.6	2.6
teams?filter+sort	8.5	13.0	3.5	4.1
year/2026	0.5	0.6	0.5	0.7
team_year/{n}/2026	14.3	16.1	15.0	19.7
team_years?2026+epa+page	123.7	322.3	34.6	42.7
event/{key}	3.4	4.9	1.8	2.7
events?2026+page	9.9	15.9	3.3	4.0
team_event/{}	8.4	10.6	2.9	3.5
team_events?team	10.3	14.6	3.5	4.2
match/{key}	15.0	17.2	11.4	17.5
matches?event	45.0	65.2	29.8	43.0
matches?team+year	50.1	116.3	48.9	86.2

DuckDB is faster on nearly every class and dramatically so on offset pagination (team_years p95 322ms → 43ms), where CockroachDB's OFFSET scan dominates. Point lookups on team_year are marginally slower (full-file scan, no index) but well within budget. Parquet publish adds ~2.3s/cycle (columnar pyarrow; content-gated reruns ~1.6s) — in line with the existing ~2–18s "Write Storage" step.

Storage/httpfs note: the reader uses a local cache of the Parquet blobs rather than DuckDB httpfs. Local files give predicate pushdown plus the glob/listing the fake-gcs emulator can't serve over httpfs; against real GCS this becomes the gcs/httpfs extension with the same SQL. One knob (DUCKDB_SYNC_TTL).

What full cutover would look like

Remaining to drop CockroachDB entirely (beyond this PR):

• Point /v3/site at the DuckDB layer too (the pre-joined _read_* shaping is unchanged; it just needs the same get_*).
• Move etags and the EPA/watermark state fully into the snapshot ([blob-store 3/5] Pipeline state as a snapshot blob; DB becomes a downstream consumer #9) so the pipeline never reads the DB — the cross-year EPA seed read is the last DB dependency.
• Incremental EPA checkpoint/resume and the stale-while-revalidate refresh loop (design doc Round 3) so updates are O(new matches) and the data/ router becomes internal.
• Swap the local Parquet cache for GCS httpfs (or keep the cache with a cold-start prefetch) in production.
• Then remove src/db/, sqlalchemy-cockroachdb, and collapse the three App Engine services to one Cloud Run service.

After those, the relational DB has no readers and no writers and can be deleted; Parquet blobs + the snapshot become the whole source of truth, and new UI surfaces are new SQL over existing blobs — no new blob types.

Verification is rig-only; no staging touched.

[chondl]

Superseded by #22 — reopened as #22 [04] on cph-duckdb-api.