fix(tests): the RSS gate judged the one batch an arena step can land in (#768)

[InauguralPhysicist]

Closes #768

Main went red in run 30439602640 with RSS1 shared_incr leaked 2876 kB over 2000 reqs (1472 B/req). Attempt 2 of the same commit passed, and the PR branch had passed on the same tree 40 minutes earlier.

It was not a leak. A 24,000-request soak grows 0 kB after warmup, bounding any real per-request leak at <3 B/req against the 1472 B/req reported.

Cause. Every connection is served by a fresh EigsState on its own thread (ext_http.c:~1400). The first time two workers' lifetimes overlap, glibc allocates a second per-thread malloc arena and RSS steps by ~one worker's footprint — once, then arenas are recycled and RSS is flat. The overlap is scheduled by wall clock, not request count: the step lands at request 400 / 500 / 600 / 700 across four identical dev-box runs (size 2668–2856 kB, matching CI's 2876 kB), and on a slower runner past request 3000 — inside measured batch 2, which #765 had just made the sole verdict.

#765 fixed the front-loaded warmup shape. A late one-shot step wears the same "did not decay" signature while being equally one-time. The discriminating evidence was already in the failure text: first batch 0 kB — a leak of 1472 B/req cannot be absent from the immediately preceding identical batch.

Fix. Three measured batches, verdict = their median. One step, wherever it lands, moves at most one of three; a real leak moves all three. Threshold unchanged at 64 kB. Cost is one extra 2000-request batch per check (~4s locally).

Validation

• Verdict rule extracted as a pure function of the three numbers, self-tested on every run against 10 planted faults: flat, sub-threshold jitter, the arena step in each of the three positions, RSS falling as well as rising, both historical leak rates ([leak] shared_incr leaks 2 Values per call (160 B/request, measured) — and no ASan build ever compiles ext_http.c #731's 160 B/req, ext_http has no graceful shutdown, so LeakSanitizer never runs in the server process — no sanitizer gate can catch a per-request leak #752's 136 B/req), a leak with a step on top, and a sustained 1472 B/req.
• End-to-end red: a real 256 B/req heap leak planted in shared_incr → FAIL: leaked 532 kB (272 B/req; batches 576/532/532 kB), with RSS2 correctly staying clean.
• End-to-end green: fault reverted, rebuilt, 3/3 clean runs.
• Full suite against make http: 3373/3373.

One note worth carrying forward: the first fault planted to validate this gate — dropping a val_decref in shared_incr — leaked nothing measurable, because make_num serves request-path Values from the arena and they die at arena reset regardless of refcount. A green planted fault is evidence about the fault first.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR fixes flakiness in the HTTP per-request RSS growth leak gate by making it robust against one-time allocator “steps” that can land in any measured window depending on scheduler timing, while keeping the leak threshold unchanged.

Changes:

• Switch the RSS-growth verdict to three consecutive measured batches and judge the median (instead of judging only batch 2).
• Factor the verdict logic into a pure leak_verdict function and add an always-on self-test against planted scenarios.
• Update the test-suite runner label and document the incident and fix in CHANGELOG.md.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
tests/test_http_rss_growth.sh	Implements 3-batch median verdict, extracts the decision rule into a function, and adds self-tests.
tests/run_all_tests.sh	Updates the suite log line to reflect the added verdict self-test.
CHANGELOG.md	Records the intermittent CI failure cause and the new median-based verdict rule.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.