Multi-state boundary: trace.c is entirely process-global and ext_http's per-connection worker calls trace_shutdown(), destroying the process tape on every request

[InauguralPhysicist]

Found in a modularity review of main @ 8e8970a. The EigsState / EigsThread seam is genuinely good and most of the runtime obeys it — these are the subsystems that were never migrated and are now contradicted by the runtime's own in-tree use of multiple states.

Premise correction first, so this isn't read as broader than it is: g_vm, g_try_depth, and g_arena are not globals — they are bridge macros onto struct EigsThread (eigenscript.h:716-784), exactly as docs/EMBEDDING.md:59-61 describes. The seam works. The defects below are a small number of specific opt-outs.

1. trace.c has 24 file-scope globals and no EigsState reference at all

The tape (trace.c:386), the embed sink (:406-410), the replay reader (:536-547), the dedup state (:392-393, 452), the recording flags (:50-54), and the temporal history table (:118-120) are all process-wide.

trace_shutdown() (trace.c:1013-1044) tears all of it down — and it is called from every HTTP connection worker (ext_http.c:1292), each of which owns its own EigsState (ext_http.c:1269-1281: eigs_state_new + eigs_thread_attach per connection).

So with EIGS_TRACE set, the first HTTP request served closes the process's tape, unregisters any sink installed via eigs_set_trace_sink, and frees the entire prev-table — running slot_decref on entries recorded by other, still-live states. Separately, trace_assign (:1102) writes the table unlocked from concurrent workers, and prev_lookup_slot keys purely on the name string (:139, 163-169), so prev of x in one state can read another's history.

This is the same subsystem flagged in the correctness review for being process-global while the embedding contract is multi-state; the HTTP call site is what makes it concrete rather than theoretical.

Minimal fix: make trace_shutdown() take an EigsState* and tear down only that state's tape and table, so ext_http.c:1292 stops destroying the main state's. Full fix: move the globals into a TraceState owned by EigsState, reached by the same bridge macro as everything else.

2. g_exit_requested is never reset, permanently disabling try/catch

builtins.c:625-626 declares it; builtin_exit sets it (:641-642); CHECK_ERROR consults it to make the unwind uncatchable (vm.c:2505, 2515). The only writes in the tree are those two. Neither eigs_close (eigs_embed.c:44-65) nor eigs_state_destroy (state.c:35-61) clears it, and main.c:365 deliberately leaves it set.

Making exit uncatchable is correct. Never resetting it is not: once any script in any state calls exit of N, every subsequent eigs_eval_string in the process — same state or another — has exception handling silently disabled, and a raise inside try goes to vm_error_halt instead of the catch handler. For a long-lived embedder running untrusted snippets (the sandbox_run case), one line of script permanently corrupts the semantics.

I could not build a CLI repro because exit ends the process there; this is an embedding/HTTP-worker path. Fix: clear it at eval entry beside the existing g_parse_errors = 0; g_has_error = 0;, or move it onto EigsThread.

3. g_task_suspend_request is global while its scheduler is per-thread

The cooperative scheduler lives on EigsThread (eigenscript.h:705, freed per-thread at state.c:147), but its suspend request is a plain int (vm.c:5513, extern at vm.h:542), set at four sites and polled by every vm_run on every thread (vm.c:3435). A task_yield on thread A can drive thread B into vm_suspend_halt and hand B's live stack slice to A's task. Mechanical fix: move it into the per-thread TaskScheduler.

4. Smaller opt-outs, same shape

• s_osr_threshold (vm.c:3228-3229) is a plain function static with no __thread, so the first state to cross a back edge freezes the OSR threshold process-wide — defeating the per-state JIT tuning eigenscript.h:570-576 explicitly advertises ("so two co-located embedded states can tune independently"). Entry and iter thresholds are correctly per-state; only OSR opts out.
• g_sandbox_* (vm.c:711, 755-757) is save/restored around one call on the documented premise that "sandbox_run is synchronous / single-threaded" (vm.c:709-711, 752-754) — true per-thread, false the moment two states run concurrently, which ext_http.c:1269 does per connection.
• g_stream_file (builtins.c:2023) is one FILE* per process, and stream_open unconditionally closes whatever is open.
• g_model (model_io.c:8) and g_db_conn (ext_db.c:10) are one-per-process but registered per-state, and nothing frees them at state teardown.

The pattern worth naming

ext_store.c has zero file-scope globals and routes every Store through the per-state handle table (:597, 714). That is the correct seam and it already exists in-tree — the model and DB extensions simply predate it. Likewise g_vm_abort_flag (vm.c:734-735) is a documented, reasoned opt-out (async-signal-safety needs a flat address). The contrast between that explicit opt-out and trace.c's silent one is the whole finding.

[InauguralPhysicist]

Reopening — PR #761 fixed item 1 only, and Closes #739 closed the whole issue by mistake. The remaining items are real and untouched.

Done (#761): trace.c's per-connection trace_shutdown(). The teardown is split into process-wide trace_shutdown() and per-thread trace_thread_release() (called by eigs_thread_detach), and the prev-table moved onto EigsThread. Measured: 1 record per 400 requests before, 336 after.

One premise correction while I was in there: cross-state history reads were not possible. prev_lookup_slot keys on the interned name pointer and g_env_name_interns is per-thread (eigenscript.h), so two states' "x" were never the same key. The damage was entirely the teardown, not the lookup. The unlocked concurrent write to the shared table is likewise moot now that each thread owns its own.

Still open:

• Item 2 — g_exit_requested is never reset, permanently disabling try/catch. The most severe of the remainder: once any script in any state calls exit of N, every subsequent eigs_eval_string in the process has exception handling silently disabled. For a long-lived embedder running untrusted snippets, one line of script permanently corrupts the semantics — and this is the seam EigenOS M11 consumes. Fix is a line at eval entry beside g_parse_errors = 0; g_has_error = 0;, or a move onto EigsThread.
• Item 3 — g_task_suspend_request is global while its scheduler is per-thread. A task_yield on thread A can drive thread B into vm_suspend_halt.
• Item 4 — the smaller opt-outs: s_osr_threshold (defeats the per-state JIT tuning eigenscript.h advertises), g_sandbox_*, g_stream_file, g_model, g_db_conn.

Item 2 is the one worth doing next.

[InauguralPhysicist]

Item 4 has a live symptom, and it is failing CI on main intermittently.

The db extension (postgres service) job — which builds the full variant (http+model+db) — fails on the per-request RSS gate (tests/test_http_rss_growth.sh, suite [45c]):

FAIL: RSS1 shared_incr leaked 2852 kB over 2000 reqs (1460 B/req; threshold 64 kB)
FAIL: RSS2 authed route leaked 1516 kB over 2000 reqs (776 B/req)

This is pre-existing on main, not introduced by a PR. The same job and step failed on main at 2026-07-28T23:02 and 2026-07-29T01:32, and passed at 21:51 and 03:40 — intermittent, but unambiguous when it fires (2852 kB against a 64 kB threshold is not noise). PR #762's run hit it too; PR #761's did not.

It is specific to the db variant. Locally, on make http (http+model, no db), the same gate is clean:

PASS: RSS1 shared_incr steady-state growth 0 kB over 2000 reqs (<= 64 kB)
PASS: RSS2 authed route steady-state growth 0 kB over 2000 reqs (<= 64 kB)

I could not build the full variant here to confirm directly (no libpq headers on this box), so the mechanism below is a hypothesis, not a measurement.

Hypothesis — item 4, stated in this issue before the symptom appeared:

g_model (model_io.c:8) and g_db_conn (ext_db.c:10) are one-per-process but registered per-state, and nothing frees them at state teardown.

Every ext_http connection worker calls eigs_state_new() + register_*_builtins() and destroys the state when the request completes (ext_http.c, http_conn_thread). If db/model registration allocates per state and teardown does not free it, that is a per-request leak reachable only when those extensions are compiled in — which matches both the variant-specificity and the ~1.5 kB/req magnitude.

Worth noting why this class needs the RSS gate at all: per CLAUDE.md, LeakSanitizer runs atexit and the test server is killed with no SIGTERM handler, so no sanitizer gate can see a per-request leak in ext_http.c. make asan-http is clean here (3374/3374, leak tally 0) and always will be for this bug. The RSS gate is the only instrument that sees it.

Next step for whoever takes item 4: build the full variant with libpq, run tests/test_http_rss_growth.sh, and bisect the per-state registrations — register_db_builtins / model loader first. The intermittency is itself worth explaining; a leak that large should fire every run, so something is gating it (possibly whether a live DATABASE_URL connection is actually established).

[InauguralPhysicist]

Correction: my g_db_conn hypothesis for the RSS failure is refuted.

I suggested above that the intermittent db extension RSS-growth failure was per-state db/model registration leaking. That is wrong, and I would rather retract it than let someone chase it:

• db_connect is only ever called explicitly by a script (ext_db.c:63, reading DATABASE_URL at call time). tests/test_http_rss_growth.sh drives shared_incr and an authed route and never touches db, so no connection is ever opened in the failing test. There is no PGconn to leak there.
• The db builtins are registered per worker env in the full variant (register_builtins → register_db_builtins), but they are ordinary make_builtin values freed with the env — the same shape as the http and model builtins, which do not leak. The http+model variant runs the same gate clean locally: 0 kB over 2000 reqs on both checks.

So the mechanism is still unexplained, and it is not item 4. Current evidence:

where	result
main (2026-07-28 23:02, 2026-07-29 01:32)	FAIL
main (21:51, 03:40)	pass
PR #762, PR #763 (neither touches db/http)	FAIL
local make http (no db), 5 runs	pass, 0 kB

Four sightings, all in the db extension job — which is also the only job that runs the full variant inside a container with a live postgres service. The next diagnostic step is to separate the two variables that job conflates: build the full variant without DATABASE_URL set, and build the http variant inside the same container, then see which one moves. I could not do that here — this box has no libpq at all (no headers, no library), so I cannot build the full variant to reproduce.

It is worth filing separately from #739: it is blocking unrelated PRs, and item 4 does not explain it.

---

Item 4 itself — what I did change, and one deliberate non-change:

• s_osr_threshold (vm.c) — was a function static filled by whichever state crossed a back edge first, then frozen process-wide, defeating the per-state JIT tuning eigenscript.h advertises. Now reads the per-state g_osr_threshold directly. The eigs_jit_get_osr_threshold() accessor existed only to feed that cache and is deleted; its comment claimed "once per thread", which was never true — it was once per process.
• g_sandbox_active / _bytes_used / _byte_max / g_sandbox_loop_max — now per-thread. The save/restore in builtin_sandbox_run is correct for one thread's nesting, but its documented premise ("sandbox_run is synchronous") holds per-thread and breaks the moment two states run concurrently, which ext_http does per connection: one worker entering a sandbox capped every other worker's loops and charged their allocations against its budget.
• g_stream_file — now per-thread, and closed at eigs_thread_detach. One FILE* per process meant stream_open's unconditional close tore down another state's in-flight stream; an unclosed stream also lost its buffered tail at exit (verified: 24 bytes now reach disk where they previously did not).
• g_db_conn — moved onto EigsState with a ext_db_state_destroy hook from eigs_state_destroy, mirroring ext_http_state_destroy. Not built or run here (no libpq); syntax-checked against a local stub header only. Worth a careful look in review.

Deliberately not changed: g_model. Its allocations are reachable from a global at exit, so they are "still reachable", not leaked — LSan does not report them and the OS reclaims them. The genuine defect would be cross-state clobber (one state's eigen_model_load replacing another's weights), and the fix for that — per-state weights — multiplies a transformer's memory by the number of states. For a heavy shared resource that reads like the wrong trade, so I left the scope alone rather than "fix" it into a memory regression. Same for g_model_age / g_training_samples. If multi-state model isolation is actually wanted, that is a design decision, not a mechanical move.

[InauguralPhysicist]

All four items are now on main. Ledger, because the PR mapping is not one-to-one:

item	what	landed in
1	trace_shutdown() called per HTTP connection; prev-table ownership	#761
2	g_exit_requested never reset → try/catch permanently disabled	#762
3	g_task_suspend_request global while its scheduler is per-thread	#764 (not #763 — see below)
4	s_osr_threshold, g_sandbox_*, g_stream_file, g_db_conn	#764

Item 3 was reviewed as #763 but merged as part of #764: I branched the item-4 work off the item-3 branch with git checkout -b, so it inherited that commit, and #764's description covered only item 4. #763 is closed with a pointer; both PRs carry the correction. The change itself is unaltered and its gates were run — it simply did not merge under a description that described it.

Two things deliberately left undone, so this issue is not closed on a false premise:

1. g_model (and g_model_age / g_training_samples) are still process-wide. Their allocations are reachable from a global at exit, so they are still-reachable rather than leaked — LSan does not report them. The genuine defect is cross-state clobber: one state's eigen_model_load replaces another's weights. Fixing that means per-state transformer weights, i.e. memory × number of states. That is a design decision about whether multi-state model isolation is wanted, not a mechanical scope move, and it should be decided rather than absorbed into a cleanup PR.

2. g_db_conn was changed but never executed. Moved onto EigsState with an ext_db_state_destroy hook mirroring ext_http_state_destroy, but this box has no libpq (no headers, no library), so ext_db.c and state.c with EXT_DB=1 were only syntax-checked against a stub header I wrote. CI's db extension job is the first real build. If it goes red on a compile error, that hunk is where to look.

Separately, the intermittent db extension RSS-growth failure remains unexplained and is not any of these four — see the retraction above. It has now been observed on main twice and on two unrelated PRs, and it blocked three merges in this sequence. It deserves its own issue; the diagnostic I would run first is to separate the two variables that job conflates (full variant without DATABASE_URL, vs http variant inside the same container), since neither has been isolated.

[InauguralPhysicist]

The intermittent db extension RSS-growth failure now has its own issue: #765. It is not any of this issue's four items, and it carries a bisection point (onset coincides exactly with 8aaf44a, the json_encode depth bound) plus the ruled-out hypotheses so nobody re-runs them.