Intermittent: db extension job fails the per-request RSS gate (RSS1/RSS2), ~25% of runs since 8aaf44a

[InauguralPhysicist]

main is currently red on this job (the #764 merge run), and it has blocked three merges in the last day. Splitting it out of #739, where I first reported it — it is not any of that issue's four items.

The failure

db extension (postgres service) → Run full suite with live DATABASE_URL → tests/test_http_rss_growth.sh (suite [45c]):

FAIL: 2 HTTP RSS-growth check(s) failed
FAIL: RSS1 shared_incr leaked 2852 kB over 2000 reqs (1460 B/req; threshold 64 kB)
FAIL: RSS2 authed route leaked 1516 kB over 2000 reqs (776 B/req)

and on another run, RSS2 alone at 1540 kB / 788 B per req. Not noise: the threshold is 64 kB and the observed growth is 20-40x it, measured between two steady-state checkpoints (the harness already excludes the ~1.4 MB one-time arena warmup).

Rate, and a bisection point

Last 20 CI runs, db extension only:

when (UTC)	branch	db job
07-28 06:25 … 21:51	main + 5 PRs	all pass (8 runs)
07-28 23:02	main 8aaf44a	FAIL
07-29 00:36, 00:55	fix/735	pass
07-29 01:32	main cdcb31e	FAIL
07-29 03:20, 03:40	fix/739-trace, main 52aa990	pass
07-29 05:20	fix/739-exit	pass
07-29 05:48	main c913ea9	FAIL
07-29 06:24	fix/739-task	FAIL
07-29 07:30	fix/739-globals	pass
07-29 07:49	main b70e649	FAIL

Every run before 07-28T23:02 is clean; 5 of the 12 runs since have failed. 07-28T23:02 UTC is the CI run for 8aaf44a — fix(json): bound json_encode depth (#730/#757) — whose local commit time is 07-28T18:02 CDT, the same instant. So the onset coincides exactly with that merge.

I want to be careful with that: it is a correlation with a clean before/after boundary, not a demonstrated mechanism, and #757's own PR run passed. Treat it as where to bisect first, not as the answer.

Why it is at least plausible: 8aaf44a changed eigs_json_encode to return NULL on refusal and touched both ext_http.c callers — shared_set and shared_incr — and RSS1 exercises shared_incr. That is the same call-site family that #731 found three ownership bugs in ("if you touch one, re-audit the rest"). Reading the diff I could not find the leak: the new early returns look correct and the encoder frees its strbuf on the failure path. So either it is subtler than a read reveals, or the correlation is coincidental.

What is already ruled out

• Not g_db_conn / per-state db registration. I proposed this in Multi-state boundary: trace.c is entirely process-global and ext_http's per-connection worker calls trace_shutdown(), destroying the process tape on every request #739 and it is wrong: db_connect is only ever called explicitly (ext_db.c:63), and this test drives shared_incr and an authed route — no connection is ever opened. The db builtins are registered per worker env, but as ordinary make_builtin values freed with the env, exactly like the http/model ones that do not leak.
• Not visible in the http variant. Locally, make http (http+model, no db) runs the same gate clean: 0 kB over 2000 reqs on both checks, 5 consecutive runs.
• Not catchable by any sanitizer. Per CLAUDE.md, LeakSanitizer runs atexit and the test server is killed with no SIGTERM handler, so LSan never runs in the server process. make asan-http is clean (3374/3374, leak tally 0) and would stay clean with this bug present. The RSS gate is the only instrument that sees this class — which is also why it must not be quietened.

Suggested next steps

The failing job conflates three variables that no other job combines: the full variant (http+model+db), a container, and a live postgres service. Separate them before touching code:

1. Full variant, container, no DATABASE_URL → isolates the live service.
2. http variant (no db) inside the same container → isolates db linkage from the environment.
3. Full variant on a plain runner → isolates the container.

If (2) fails, it is environmental and the code is fine. If (1) or (3) fails, bisect 8aaf44a directly: revert its ext_http.c hunk on a branch and run the db job a few times — the ~25% rate means a single green run proves nothing, so run it at least 5 times either way.

I could not do any of this here: this box has no libpq at all (no headers, no library), so the full variant cannot be built locally.

Meanwhile

Because the rate is ~25%, a red run on an unrelated PR is not evidence about that PR. Re-run the job; if it goes green the PR is fine. Do not merge past a reproducible red on this gate — a per-request leak in the HTTP server is exactly the class nothing else catches.

[InauguralPhysicist]

Authorship/timing of the bisection point, since it matters for who investigates and what else to suspect.

8aaf44a was committed 07-28 18:02 CDT with Co-authored-by: Claude Fable 5. The four commits that follow it on main — cdcb31e, 52aa990, c913ea9, b70e649 — are from a later session (Claude Opus 5), the first of which began at 18:30 CDT. So the failure onset is 28 minutes before any of that later work existed, and the first failing run is 8aaf44a's own.

Two caveats against reading that as an exoneration of the later commits:

1. Three of the four later commits' runs also failed (one passed). At a ~25% rate that is unremarkable, but 12 runs cannot tell you whether the rate changed.
2. 52aa990 (fix(trace): the tape is the process's, not a connection worker's (#739, #755, #760) #761) modified ext_http.c's per-request connection worker — it removed the trace_shutdown() call so the thread's prev-table is released at eigs_thread_detach instead. That is a per-request path, and per-request RSS growth is the symptom. Against it: the failures predate it by two runs, its own CI run passed, and the http-variant gate is clean locally (0 kB / 2000 reqs, 5 runs) with that change in. But anyone bisecting should know it is there rather than find it by surprise.

So the ordering to try: revert 8aaf44a's ext_http.c hunk first (it is the boundary), and if that does not settle it, 52aa990's worker change is the other per-request edit in the window.