fix(trace): the tape is the process's, not a connection worker's (#739, #755, #760)

CHANGELOG.md

@@ -171,6 +171,42 @@ All notable changes to EigenScript are documented here.
 
 ### Fixed
 
+- **Every HTTP connection worker tore down the process trace tape, so the tape
+  died after the first request (#739).** `trace_shutdown()` is a process-wide
+  teardown — it closes the one tape, drops an embedder's trace sink, and shuts
+  down the replay reader — and `ext_http.c`'s per-connection worker called it
+  when it finished a request. Each worker owns its own `EigsState`, but the
+  tape belongs to the process, so the *first* request served closed it and
+  every later request's records were silently dropped. Measured on the
+  unfixed runtime: **one record for four hundred requests**, with the server's
+  tape fd already closed; after the fix, 336 records had reached disk mid-run
+  and the fd was still open. The same teardown freed the temporal prev-table,
+  running `slot_decref` on entries recorded by other, still-live threads. The
+  teardown is now split: `trace_shutdown()` keeps the process-wide half, and a
+  new `trace_thread_release()` releases only the calling thread's history —
+  called by `eigs_thread_detach` for every thread, so the HTTP worker no longer
+  touches the trace subsystem at all. The prev-table moved onto `EigsThread`
+  behind bridge macros, which is the scope it always effectively had: it is
+  keyed by *interned name pointer* and the intern table is per-thread, so two
+  threads' `x` were never the same key — process-global storage bought nothing
+  and cost ownership. It needs no lock, since only the owning thread touches
+  its table. The tape encoding is untouched, so no format-version bump — this
+  was an ownership bug, not a format one. (Corrects one premise in the issue:
+  cross-state history reads were *not* possible, because pointer-keying already
+  separated them.)
+
+- **`http_post` silently sent NO headers when `headers` was a JSON object
+  (#755).** The documented signature is `http_post of [url, headers, body]`,
+  and an object is the shape a caller reaches for first — but the code tested
+  `VAL_LIST` while `eigs_json_parse_object` returns a `VAL_DICT`, so the loop
+  never ran and the request went out bare. No error, no warning, a normal
+  response body: an `Authorization` header silently not sent. Only the flat
+  alternating array form worked, and `docs/BUILTINS.md` never said which shape
+  was expected. Both shapes are accepted now, and a headers argument that
+  parses to neither raises `type_mismatch` instead of quietly dropping them
+  (an unparseable argument, including `""`, still means "no headers" — the
+  documented idiom). BUILTINS.md now states the accepted shapes.
+
 - **`report` labelled a still-moving value "equilibrium" at a full window,
   violating the agreement guarantee it documents (#735).** `report of x`
   resolves the six windowed bands in priority order and then, if none is true,
@@ -381,6 +417,23 @@ All notable changes to EigenScript are documented here.
 
 ### Changed
 
+- **`tests/test_http_server.sh` no longer reports a setup failure as a feature
+  failure (#760).** `HS27`/`HS28` failed one CI run with "aggregate body budget
+  (got 000)" — a DoS-gate regression that had not happened. Both checks had
+  simply received *empty* responses, because the server was never reachable.
+  Three defects, each measured: the listen port was drawn from `50000-59999`,
+  entirely inside the kernel's ephemeral range (`32768-60999` on Linux), so the
+  suite's own `curl` clients could hold the port the next server tried to bind;
+  the readiness loop's real budget was **3.58s**, not the ~33s its
+  `--max-time 1` implied, because a refused connection returns instantly; and
+  neither failure was detected, so the test proceeded against a server that
+  was not there and the log naming the real cause was written to `/tmp` and
+  never read. There is now a shared `pick_port` (below the ephemeral floor,
+  read from `/proc` where available) and a `wait_ready` with a wall-clock
+  deadline, and a server that never comes up fails as a **setup** failure
+  quoting its own log. The main server in the same file had both defects too
+  and now uses the helpers.
+
 - **`make asan-http` puts the HTTP + model extensions under a sanitizer for the
   first time, and CI runs the suite that way.** `make asan` compiles those
   extensions out, so `ext_http.c` — a network-facing server and client, the most

docs/BUILTINS.md

@@ -538,7 +538,7 @@ axis. Three controls bound it:
 | `http_serve` | `http_serve of port` | Start blocking HTTP server |
 | `http_request_body` | `http_request_body of null` | Get current request body |
 | `http_session_id` | `http_session_id of null` | Get current session ID |
-| `http_post` | `http_post of [url, headers, body]` | HTTP POST via curl (no shell) |
+| `http_post` | `http_post of [url, headers, body]` | HTTP POST via curl (no shell). `headers` is a JSON **string**, either an object (`"{\"X-Key\": \"v\"}"`) or a flat alternating array (`"[\"X-Key\", \"v\"]"`); `""` means no headers. Any other parsed shape raises `type_mismatch` rather than silently sending none (#755). Max 32 headers; CR/LF is stripped from both halves |
 | `http_request_headers` | `http_request_headers of null` | Get current request headers |
 
 ### Per-worker code routes

docs/EMBEDDING.md

@@ -254,6 +254,16 @@ act between evals. While a replay tape is set, nondet builtins return
 the recorded `N` values in order instead of consulting their live
 sources; when the tape runs out they fall back to live.
 
+**The sink and the tape are per-PROCESS, not per-state** (#739). With
+several states co-located, one sink serves them all, and the teardown
+that drops it — `trace_shutdown()`, which `eigs_close` calls — belongs
+to whoever owns the process. A per-request or per-task state must not
+call it: `ext_http`'s connection worker did, so the first request served
+closed the tape and unregistered the host's sink. A worker that wants to
+release its own temporal history calls `trace_thread_release()`, which
+`eigs_thread_detach` already does for every thread. The per-name history
+behind `prev of x` / `at` / `state_at` is per-thread and never shared.
+
 Host builtins participate with the take/record pair, the same contract
 the runtime's own nondet builtins use:
 

docs/TRACE.md

@@ -284,6 +284,26 @@ in-run time travel.
   resolve in O(history/64) instead of O(history). The index adds one
   `int` per 64 history entries and an O(1) min-update per assign.
 - Per-assign cost of the history: one cache line + a pointer compare.
+- **The history is per-thread; the tape is per-process** (#739). The
+  history table is keyed by *interned name pointer*, and the intern
+  table lives on `EigsThread`, so two threads' `x` were never the same
+  key — per-thread is the only scope on which the table is coherent,
+  and it needs no lock because only its owning thread touches it. It is
+  released by `eigs_thread_detach` (and by `trace_shutdown` for the
+  process owner's own thread, which must run before the global env dies
+  — the slots it drops can reach the env).
+- **`trace_shutdown()` is process-wide and a worker must never call
+  it.** It closes the one tape, drops the embed sink, and shuts down
+  the replay reader. Every `ext_http` connection worker used to call it
+  on finishing a request: the first request served closed the tape, so
+  every later request's records were silently dropped (measured: one
+  record for four hundred requests), an embedder's sink was
+  unregistered by whichever request arrived first, and prev-table slots
+  recorded by other still-live threads were decref'd. A worker that
+  wants to clean up after itself wants `trace_thread_release()`, which
+  touches only its own history. Nothing about the tape *encoding*
+  changed here, so no format-version bump: this was an ownership bug,
+  not a format one.
 
 Language-level syntax and examples: [SYNTAX.md](SYNTAX.md),
 [GRAMMAR.md](GRAMMAR.md).

src/eigenscript.h

@@ -586,6 +586,15 @@ struct EigsState {
 struct EigsThread {
     EigsState  *state;
     Arena       arena;
+    /* #739: temporal prev-table (`prev of x`, `at <line>`, `state_at`).
+     * Per-THREAD because it is keyed by interned name pointer and the
+     * intern table above is per-thread — a shared table could never have
+     * merged two threads' "x". Opaque here; the layout is trace.c's.
+     * Released by eigs_thread_detach (and by trace_shutdown, which must
+     * run before the global env dies — see eigs_close). */
+    struct TracePrevEntry *prev_tab;
+    int          prev_cap;      /* power of two */
+    int          prev_count;
     /* Control-flow propagation (return/break/continue out of nested
      * blocks). All three are checked + cleared by the interpreter
      * walk and the VM dispatch loop. */
@@ -773,6 +782,9 @@ extern __thread EigsThread *eigs_current;
 #define g_env_freelist        (eigs_current->env_freelist)
 #define g_env_freelist_count  (eigs_current->env_freelist_count)
 #define g_env_name_interns    (eigs_current->env_name_interns)
+#define g_prev_tab            (eigs_current->prev_tab)
+#define g_prev_cap            (eigs_current->prev_cap)
+#define g_prev_count          (eigs_current->prev_count)
 #define g_parse_depth         (eigs_current->parse_depth)
 #define g_tokenize_depth      (eigs_current->tokenize_depth)
 #define g_vts_depth           (eigs_current->vts_depth)

src/ext_http.c

@@ -409,7 +409,27 @@ Value* builtin_http_post(Value *arg) {
     int hdr_count = 0;
     int jpos = 0;
     Value *hdr_obj = eigs_json_parse_value(headers_json, &jpos);
-    if (hdr_obj && hdr_obj->type == VAL_LIST) {
+    /* #755: a JSON OBJECT is the shape a caller reaches for first, and it used
+     * to send NO headers at all — this tested only VAL_LIST, while
+     * eigs_json_parse_object returns a VAL_DICT, so the loop never ran and the
+     * request went out bare with no error and a normal-looking response. An
+     * Authorization header silently not sent is the worst version of that. Both
+     * shapes are accepted now: object (natural) and flat alternating array
+     * (what already worked). Anything else that PARSED is a caller mistake, not
+     * a no-header request — it says so instead of dropping the headers. */
+    if (hdr_obj && hdr_obj->type == VAL_DICT) {
+        for (int i = 0; i < hdr_obj->data.dict.count && hdr_count < 32 && argc < 90; i++) {
+            char *hk = xstrdup(hdr_obj->data.dict.keys[i]);
+            char *hv = value_to_string(hdr_obj->data.dict.vals[i]);
+            http_strip_crlf(hk);
+            http_strip_crlf(hv);
+            snprintf(header_bufs[hdr_count], sizeof(header_bufs[0]), "%s: %s", hk, hv);
+            free(hk); free(hv);
+            argv[argc++] = "-H";
+            argv[argc++] = header_bufs[hdr_count];
+            hdr_count++;
+        }
+    } else if (hdr_obj && hdr_obj->type == VAL_LIST) {
         for (int i = 0; i + 1 < hdr_obj->data.list.count && hdr_count < 32 && argc < 90; i += 2) {
             char *hk = value_to_string(hdr_obj->data.list.items[i]);
             char *hv = value_to_string(hdr_obj->data.list.items[i + 1]);
@@ -426,6 +446,18 @@ Value* builtin_http_post(Value *arg) {
             argv[argc++] = header_bufs[hdr_count];
             hdr_count++;
         }
+    } else if (hdr_obj && hdr_obj->type != VAL_NULL) {
+        /* Parsed, but neither shape — e.g. a bare string or number. Silently
+         * sending nothing is the #755 failure mode; say so. An unparseable
+         * headers argument (including "") still means "no headers", which is
+         * the documented idiom. */
+        rt_error(EK_TYPE, 0,
+                 "http_post: headers must be a JSON object or a flat "
+                 "[key, value, ...] array (got %s)",
+                 val_type_name(hdr_obj->type));
+        val_decref(hdr_obj);
+        unlink(req_path);
+        TRACE_NONDET_RECORD("http_post", make_str(""));
     }
     /* Released here rather than at the end of the function: the pipe/fork
      * failure paths below return through TRACE_NONDET_RECORD, and hdr_obj is
@@ -1386,7 +1418,12 @@ static void *http_conn_thread(void *arg) {
     handle_request(fd);
     fd = -1;  /* handle_request closed it */
 
-    trace_shutdown();
+    /* #739: NO trace_shutdown() here. This worker owns one connection, not the
+     * process — calling it closed the process tape after the FIRST request
+     * (every later request's records silently lost), dropped an embedder's
+     * trace sink, and decref'd prev-table slots recorded by other still-live
+     * threads. This thread's own prev-table is released by eigs_thread_detach
+     * below, beside the other per-thread destructors. */
     gc_collect_at_exit(global);
     env_decref(global);
     g_global_env = NULL;

src/state.c

@@ -5,6 +5,7 @@
 #include "state.h"
 #include "vm.h"
 #include "jit.h"
+#include "trace.h"   /* #739: trace_thread_release on detach */
 
 #if EIGENSCRIPT_EXT_HTTP
 /* Forward-declared here to avoid pulling ext_http_internal.h (and its
@@ -145,6 +146,7 @@ void eigs_thread_detach(void) {
     jit_thread_destroy(th);
     vm_thread_destroy(th);
     task_sched_thread_free();   /* #408: release the cooperative task scheduler */
+    trace_thread_release();     /* #739: this thread's prev-table (NOT the tape) */
 
     /* Phase 8: release freelist + intern memory before the EigsThread
      * struct itself goes. Must run while eigs_current still points at th

src/trace.c

@@ -97,7 +97,17 @@ typedef struct {
 #define HIST_SEG_SHIFT 6
 #define HIST_SEG       (1 << HIST_SEG_SHIFT)
 
-typedef struct {
+/* #739: the prev-table lives on EigsThread, not in a file static. It is keyed
+ * by INTERNED NAME POINTER, and the intern table is itself per-thread
+ * (`g_env_name_interns` -> `eigs_current->env_name_interns`), so two threads'
+ * "x" were never the same key and a shared table could not have merged them
+ * anyway — process-global storage bought nothing and cost ownership. The
+ * entries hold slots allocated by their own thread, so per-thread is also the
+ * only scope on which "release these" is a well-defined operation, and it needs
+ * no lock: only the owning thread ever touches its table. The tape itself
+ * (FILE*, sink, replay reader, enable flags) stays process-wide below — one
+ * process, one tape — which is the distinction `trace_shutdown` got wrong. */
+typedef struct TracePrevEntry {
     const char    *name;
     EigsSlot       prev;
     EigsSlot       current;
@@ -115,9 +125,9 @@ typedef struct {
     int            obs_cap;
 } PrevEntry;
 
-static PrevEntry *g_prev_tab = NULL;
-static int        g_prev_cap = 0;   /* power of two */
-static int        g_prev_count = 0;
+/* g_prev_tab / g_prev_cap / g_prev_count are bridge macros onto EigsThread
+ * (eigenscript.h), reached only with a thread attached. Every read path below
+ * that can run during teardown or from atexit guards on `eigs_current` first. */
 
 /* g_trace_current_line (exported, see trace.h) replaces the old static
  * line cache: OP_LINE stores it directly instead of paying a call. */
@@ -161,7 +171,7 @@ static void prev_grow(void) {
 }
 
 static void prev_record_assign(const char *name, EigsSlot value) {
-    if (!name) return;
+    if (!eigs_current || !name) return;
     if (g_prev_count * PREV_LOAD_DEN >= g_prev_cap * PREV_LOAD_NUM) {
         prev_grow();
         if (!g_prev_tab) return;
@@ -253,7 +263,7 @@ static void prev_record_assign(const char *name, EigsSlot value) {
  * Gated by g_trace_obs_hist at the call site. */
 void trace_record_obs(const char *name, double entropy, double dH,
                       double last_entropy) {
-    if (!g_prev_tab || !name) return;
+    if (!eigs_current || !name || !g_prev_tab) return;
     PrevEntry *e = prev_lookup_slot(g_prev_tab, g_prev_cap, name);
     if (!e->name || e->hist_count == 0 || !e->obs) return;
     int idx = e->hist_count - 1;
@@ -267,7 +277,7 @@ void trace_record_obs(const char *name, double entropy, double dH,
 }
 
 int trace_query_prev(const char *interned_name, EigsSlot *out) {
-    if (!interned_name || !out || !g_prev_tab) return 0;
+    if (!eigs_current || !interned_name || !out || !g_prev_tab) return 0;
     PrevEntry *e = prev_lookup_slot(g_prev_tab, g_prev_cap, interned_name);
     if (!e->name || !e->has_prev) return 0;
     *out = e->prev;
@@ -311,7 +321,7 @@ static int find_hist_idx_at_or_before(PrevEntry *e, int line) {
 }
 
 int trace_query_at(int kind, const char *interned_name, int line, EigsSlot *out) {
-    if (!interned_name || !out || !g_prev_tab) return 0;
+    if (!eigs_current || !interned_name || !out || !g_prev_tab) return 0;
     PrevEntry *e = prev_lookup_slot(g_prev_tab, g_prev_cap, interned_name);
     if (!e->name) return 0;
 
@@ -1022,6 +1032,46 @@ int trace_replay_take(const char *fn, Value **out) {
     }
 }
 
+/* #739: release THIS THREAD's prev-table. Idempotent, and a no-op with no
+ * thread attached (the atexit path runs detached). Must run while
+ * `eigs_current` still points at the owning thread — the slots it drops are
+ * that thread's, and their destructors read the bridge macros — which is why
+ * eigs_thread_detach calls it beside the other Phase-5 destructors, and why
+ * eigs_close calls trace_shutdown before the global env dies.
+ *
+ * This is the half that used to be fused into trace_shutdown, and the fusion
+ * was the bug: every ext_http connection worker called trace_shutdown when it
+ * finished a request, so one process-wide teardown ran per HTTP request —
+ * closing the tape after the FIRST request (every later request's records
+ * silently lost), unregistering an embedder's sink, and decref'ing prev-table
+ * slots recorded by other, still-live threads. */
+void trace_thread_release(void) {
+    if (!eigs_current || !g_prev_tab) return;
+    PrevEntry *tab = g_prev_tab;
+    int cap = g_prev_cap;
+    /* Clear the thread's view FIRST: a destructor reached from slot_decref
+     * below must not find a half-freed table through the bridge macros. */
+    g_prev_tab = NULL;
+    g_prev_cap = 0;
+    g_prev_count = 0;
+    for (int i = 0; i < cap; i++) {
+        PrevEntry *e = &tab[i];
+        if (!e->name) continue;
+        if (e->has_prev)    slot_decref(e->prev);
+        if (e->has_current) slot_decref(e->current);
+        for (int j = 0; j < e->hist_count; j++)
+            slot_decref(e->history[j].value);
+        free(e->history);
+        free(e->seg_min);
+        free(e->obs);
+    }
+    free(tab);
+}
+
+/* Process-wide teardown: the tape, the sink, the replay reader. One process,
+ * one tape — so this belongs to whoever owns the process (main / eigs_close /
+ * atexit), NEVER to a per-connection or per-task worker. A worker that wants
+ * to clean up after itself wants trace_thread_release. */
 void trace_shutdown(void) {
 #if !EIGENSCRIPT_FREESTANDING
     if (g_trace_fp) {
@@ -1035,23 +1085,7 @@ void trace_shutdown(void) {
     g_trace_sink_ud = NULL;
     g_trace_enabled = 0;
 
-    if (g_prev_tab) {
-        for (int i = 0; i < g_prev_cap; i++) {
-            PrevEntry *e = &g_prev_tab[i];
-            if (!e->name) continue;
-            if (e->has_prev)    slot_decref(e->prev);
-            if (e->has_current) slot_decref(e->current);
-            for (int j = 0; j < e->hist_count; j++)
-                slot_decref(e->history[j].value);
-            free(e->history);
-            free(e->seg_min);
-            free(e->obs);
-        }
-        free(g_prev_tab);
-        g_prev_tab = NULL;
-        g_prev_cap = 0;
-        g_prev_count = 0;
-    }
+    trace_thread_release();
 
     replay_shutdown();
 }