fix(runtime): four more process-globals that contradict the multi-state contract (#739 item 4)

[InauguralPhysicist]

Last of #739's items. Item 1 (trace tape) → #761, item 2 (exit request) → #762, item 3 (task suspend) → #763.

Each of these is a resource the runtime scopes per state or per thread everywhere else, held in one process-wide slot.

s_osr_threshold — a function static froze per-state JIT tuning

It was filled by whichever state crossed a back edge first, then frozen for the life of the process — defeating the tuning eigenscript.h explicitly advertises ("so two co-located embedded states can tune independently"). Entry and iter thresholds were already per-state; only OSR opted out. It now reads g_osr_threshold directly.

eigs_jit_get_osr_threshold() existed only to fill that cache and is deleted. Its comment claimed the value was read "once per thread" — never true; it was once per process, which is precisely the bug.

g_sandbox_* — one budget shared by every concurrent state

g_sandbox_active / _bytes_used / _byte_max / g_sandbox_loop_max are now per-thread. The save/restore in builtin_sandbox_run is correct for one thread's nesting, but the premise it documented — "sandbox_run is synchronous" — holds per thread and breaks the moment two states run concurrently, which ext_http does per connection. One worker entering a sandbox capped every other worker's loops and charged their allocations against its budget.

g_stream_file — one FILE* per process

Now per-thread, closed at eigs_thread_detach. stream_open unconditionally closes whatever is already open, so two states streaming at once tore down each other's file mid-write. An unclosed stream also lost its buffered tail at exit; verified 24 bytes now reach disk where they previously did not.

The freestanding symbol gate caught fclose leaving the allowlist, so the cleanup is carved out under !EIGENSCRIPT_FREESTANDING alongside the stream_* builtins themselves. Worth noting the gate did exactly its job here.

g_db_conn — per-state, and actually closed

Moved onto EigsState, closed by a new ext_db_state_destroy called from eigs_state_destroy — the same shape ext_http_state_destroy already had. One worker's db_connect used to replace the connection another worker was querying through, and nothing ever closed it.

⚠️ Not built or run. This box has no libpq at all — no headers, no library — so ext_db.c and state.c with EXT_DB=1 are syntax-checked against a local stub header only. That catches compile errors and proves nothing about behavior. This is the one hunk that deserves real review.

Deliberately unchanged: g_model

Its allocations are reachable from a global at exit, so they are still-reachable, not leaked — LSan does not report them and the OS reclaims them. The genuine defect would be cross-state clobber (one state's eigen_model_load replacing another's weights), and the fix for that is per-state transformer weights, which multiplies memory by the number of states. That is a design decision about model isolation, not a mechanical scope move, so I left it rather than trade a non-leak for a memory regression. Same for g_model_age / g_training_samples.

Validation

• release 3274/3274; asan+ubsan 3278/3278, leak tally 0
• TSan 11/11 including its seeded-race self-validation (17 warnings — the gate is live, not silently dead)
• freestanding symbol gate stage 1+2 OK after the carve-out; jit-smoke; sandbox, task, and HTTP suites pass

One transient to record rather than hide: a single test_http_server.sh run showed 41/1 during this work. I did not capture which check, and five consecutive runs since are 42/0. Logging it as an observed transient, not as nothing.

Refs #739 — with this, all four items are addressed. The intermittent db extension RSS-growth failure is not explained by any of them; I retracted my earlier hypothesis in the issue thread and proposed the diagnostic that separates the two variables that job conflates.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR completes the remaining “item 4” fixes from #739 by eliminating several process-global resources that break the runtime’s multi-state / multi-thread contract, and by documenting + regression-testing the task-yield cross-thread suspend bug.

Changes:

• Remove the function-static OSR threshold cache so JIT OSR tuning remains per-state; delete the now-unneeded eigs_jit_get_osr_threshold() accessor.
• Move sandbox caps/budget and stream_open’s target stream from process globals to per-thread (EigsThread) storage, with thread-detach cleanup for unclosed streams.
• Move the DB connection from process-global to per-state (EigsState) storage and close it at state teardown; update docs/spec/changelog and add a regression test for the task-yield bug.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/test_tasks.eigs	Adds a regression test ensuring task_yield on one OS thread cannot suspend other OS threads.
src/vm.h	Updates comments to reflect task-suspend and sandbox controls now being per-thread bridge macros.
src/vm.c	Removes process-global sandbox/task-suspend variables; reads OSR threshold directly from per-state tuning.
src/state.c	Calls ext_db_state_destroy on state teardown; closes per-thread stream handle on thread detach (hosted builds).
src/jit.h	Removes eigs_jit_get_osr_threshold() declaration (no longer used).
src/jit.c	Removes eigs_jit_get_osr_threshold() definition (no longer needed).
src/ext_db.c	Adds ext_db_state_destroy to close the per-state DB connection.
src/ext_db_internal.h	Replaces the process-global DB connection with a per-state accessor macro.
src/eigenscript.h	Adds per-thread fields for sandbox/task/stream and exposes new bridge macros; adds per-state ext_db_conn.
src/builtins.c	Switches stream builtins from a process-global FILE* to the per-thread bridge macro.
docs/SPEC.md	Documents that cooperative tasks are scoped per OS thread.
docs/EMBEDDING.md	Documents per-state/per-thread teardown responsibilities for embedding.
docs/CONCURRENCY.md	Adds explicit documentation that cooperative tasks (and suspend requests) are per-thread.
docs/BUILTINS.md	Documents stream_open as “one stream per thread” and that unclosed streams are closed at thread end.
CHANGELOG.md	Records the fixed process-global violations and the task-yield cross-thread suspend bug.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[InauguralPhysicist]

Correction: this PR also carried #739 item 3, which its description did not mention.

I branched fix/739-remaining-process-globals with git checkout -b while standing on fix/739-task-suspend-per-thread, so the new branch inherited that commit. The merged diff therefore contains both item 3 (per-thread task suspend request) and item 4:

src/vm.h              -extern int g_task_suspend_request;
src/vm.c              (global definition removed, poll site unchanged)
src/eigenscript.h     +int task_suspend_request;  + bridge macro
tests/test_tasks.eigs +27   (the task_yield-vs-OS-thread regression case)
docs/SPEC.md, docs/CONCURRENCY.md   (task scoping rule)

Nobody reading this PR's description would have expected the task-scheduler changes in it. The work itself is the same work that was up for review in #763 — same commit, same gates, same red-then-green validation (expected 400000, got null) — but it landed under a description that did not cover it, and that is on me.

#763 is now redundant and I am closing it with a pointer here. The item-3 rationale, including why the flag went on EigsThread rather than inside TaskScheduler, is in #763's description and its commit message, both of which remain readable.