fix(trace): the tape is the process's, not a connection worker's (#739, #755, #760)

[InauguralPhysicist]

#739 — every HTTP connection worker tore down the process tape

trace_shutdown() is a process-wide teardown: it closes the one tape, drops an embedder's trace sink, and shuts down the replay reader. ext_http.c's per-connection worker called it when it finished a request. Each worker owns its own EigsState, but the tape belongs to the process — so the first request served closed it, and every later request's records were silently dropped.

Measured, 400 requests against a code route that stamps one marker each:

	pre-fix	post-fix
records on tape	1	336 (mid-run)
tape fd open on server	0	1

The same teardown also freed the temporal prev-table, running slot_decref on entries recorded by other, still-live threads.

Fix. The teardown splits. trace_shutdown() keeps the process-wide half; a new trace_thread_release() releases only the calling thread's history, and eigs_thread_detach calls it for every thread — so the HTTP worker no longer touches the trace subsystem at all. Ordering is preserved: the release runs while eigs_current still points at the thread and before the freelists and arena go, and trace_shutdown still releases the process owner's own table before the global env dies (the constraint eigs_close documents).

The prev-table moved onto EigsThread behind bridge macros. That is not a new scope — it is the scope the table always effectively had: it is keyed by interned name pointer, and the intern table (g_env_name_interns) is already per-thread, so two threads' x were never the same key. Process-global storage bought nothing and cost ownership. It needs no lock, because only the owning thread touches its table.

One premise in the issue is wrong, and worth recording: it says prev of x in one state can read another's history. It cannot — pointer-keying already separated them. The damage was the teardown, not the lookup.

No format-version bump. The tape encoding is untouched (#411 applies to encoding changes); this was an ownership bug.

#755 — http_post silently sent no headers for the natural shape

The signature is http_post of [url, headers, body], and a JSON object is what a caller reaches for first. The code tested VAL_LIST while eigs_json_parse_object returns a VAL_DICT, so the loop never ran and the request went out bare — no error, no warning, a normal-looking response body. An Authorization header silently not sent is the worst version of that. docs/BUILTINS.md never said which shape was expected.

Both shapes are accepted now; a headers argument that parses to neither raises type_mismatch rather than dropping them (an unparseable argument, including "", still means "no headers" — the documented idiom). BUILTINS.md now states the accepted shapes.

#760 — the gate that misdiagnosed itself

HS27/HS28 failed one CI run with "aggregate body budget (got 000)" — a DoS-gate regression that had not happened. Both checks had received empty responses because the server was never reachable. Three defects, each measured:

• the listen port was drawn from 50000-59999, entirely inside the kernel's ephemeral range (32768-60999 on Linux), so this suite's own curl clients could hold the port the next server tried to bind;
• the readiness loop's real budget was 3.58s, not the ~33s its --max-time 1 implied — a refused connection returns instantly, so the loop was effectively 30 × sleep 0.1;
• neither failure was detected, so the test proceeded against a server that was not there, while the log naming the real cause went to /tmp and was never read.

Shared pick_port (below the ephemeral floor, read from /proc where available) and wait_ready (wall-clock deadline) helpers now; a server that never comes up fails as a setup failure quoting its own log. The main server in the same file had both defects and now uses the helpers.

Validation

New gate HS33 observes the tape by content — no /proc (macOS runs this suite) and no assumption about stdio buffer size: each request stamps a distinct marker from the shared store first, then writes padding that pushes it out of the buffer, so a marker on disk means that request's records really reached the tape. Validated red against the unfixed runtime (got 1 request markers (expected >= 3)), green with the fix. HS34 covers #755 in all three directions: object shape delivers, array shape still delivers, wrong shape raises.

• release 3273/3273
• asan+ubsan 3277/3277, asan-http 3373/3373 — leak tally 0 in both
• observer corpus 14/14; test_replay.sh 24/24 (it drives EIGS_REPLAY_STRICT=1 per case internally)
• HTTP suite 42/42; freestanding symbol gate stage 1+2 OK; jit-smoke; embed stack soak OK

docs/TRACE.md gains the ownership rule (process tape vs per-thread history, and that a worker must never call trace_shutdown); docs/EMBEDDING.md gains the multi-state consequence for hosts installing a trace sink.

Closes #739
Closes #755
Closes #760

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR fixes trace ownership boundaries and an HTTP client usability bug, and hardens the HTTP integration test harness so setup failures are diagnosed as such rather than misreported as feature regressions. It aligns the runtime with the multi-state/per-thread embedding seam by separating process-wide trace teardown (tape/sink/replay) from per-thread temporal history cleanup.

Changes:

• Split trace teardown into process-wide trace_shutdown() and per-thread trace_thread_release(), moving the temporal prev-table onto EigsThread and removing the erroneous per-request shutdown from HTTP workers.
• Fix http_post so headers provided as a JSON object are actually sent (and wrong parsed shapes raise instead of silently dropping headers), and document accepted header shapes.
• Add shared port-picking/readiness helpers and new regression gates (HS33/HS34) to tests/test_http_server.sh to prevent flaky setup and validate the fixes.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_http_server.sh	Adds port/readiness helpers and HS33/HS34 regression tests for trace-tape survival and http_post header shapes; improves setup-failure diagnosis.
src/trace.h	Documents that trace_shutdown() is process-wide and introduces trace_thread_release() for per-thread history cleanup.
src/trace.c	Moves the temporal prev-table to EigsThread via bridge macros and implements trace_thread_release(); keeps tape/sink/replay process-wide.
src/state.c	Calls trace_thread_release() during thread detach so workers clean up their own history without touching process trace state.
src/ext_http.c	Accepts JSON object headers in http_post, errors on wrong parsed shapes, and removes trace_shutdown() from the connection worker.
src/eigenscript.h	Adds prev_tab/prev_cap/prev_count to EigsThread and bridge macros for trace’s per-thread prev-table.
docs/TRACE.md	Documents the per-thread history vs per-process tape ownership rule and the worker prohibition on trace_shutdown().
docs/EMBEDDING.md	Clarifies that trace sink/tape are process-scoped across multiple states and points to trace_thread_release() for worker cleanup.
docs/BUILTINS.md	Documents http_post header JSON shapes (object or flat array), error behavior, and limits.
CHANGELOG.md	Records fixes for #739/#755/#760 and the associated test harness change.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.