M5-2c1 stage b: reclaim the app slots of clients whose address space is gone - #89
Merged
Conversation
Obligation #3 asks for the sockets of an application that ends without TERMAPI to be reclaimed. Three rounds of measurement established that the liveness this needs is not soundly detectable on 3.8j, so what ships is a best-effort sweep, named that way everywhere it is described. A batch client is never reclaimed. Its recorded ASCB is the initiator's, and an initiator does not end when a job ends -- a normal end and a CANCEL both read LIVE. An STC client is reclaimed only when the check wins the race against ASID reuse, since a later address space taking the same ASID flips the verdict back to LIVE. TERMAPI remains the contract; the design is deferred to issue #88, and ADR-0045 carries the reasoning and Mike's position that an explicit TERMAPI contract is too fragile. Shipping is defensible because the error is one-directional: a false LIVE leaks a slot, and a false DEAD cannot arise from reuse, because an ASID is unique among the address spaces that are alive. That asymmetry is also why the sweep emits NSF817I at the moment of a reclaim -- a run that reclaims nothing is not automatically a failure, and only a message taken while the evidence still exists separates "saw DEAD and acted" from "saw LIVE after reuse". Two triggers, one implementation, the cap as a parameter: periodic from nsfsx_drain, and on demand from do_initapi when the app table is full. A zero interval bypasses the limiter without being a special case, because a zero interval has always elapsed. The interval is real seconds behind a new platform seam, nsf_elapsed_ge. Not ticks: with nothing armed no NSFTMR tick advances, so a tick limiter would never see its interval elapse after an idle period, which is exactly the pass that matters. Not a timer: that reintroduces the idle floor ADR-0043 established is not required. It is pure -- both timestamps are parameters -- so the boundary is host-testable exactly, and it rounds late. The ten seconds bounds the gap between sweeps and promises nothing about latency, since a sweep still only happens on a pass. Placement was three decisions. It lives in nsfsx_drain rather than nsfreq_drain because evt_set_request wires exactly one drain per build, which also makes "Phase 1 sweeps nothing" structural. It sits after step 0, never in front of it, because the wake-ECB reset's argument is positional. And it stands down while a request is in service, since that request may be parked on a socket owned by one of these very apps. The notification is a seam rather than an nsfmsg call: most builds linking nsfreq.c do not link NSFMSG, and a WTO is invisible to a host test. Host 3007 -> 3190 PASS / 0 FAIL, and the new assertions are verified to discriminate: widening the reclaim predicate fails 14, capturing the identity after app_free fails 2, an always-elapsed limiter fails 2, no on-demand sweep fails 4, and a broken sub-second borrow fails 2. Cross-build clean, alias scan 239 unique. NSFRQE stays frozen at 64 bytes, the anchor layout is unmoved and asm/ is untouched. The live gate is not run: arms 1 and 2 and the revert test are still open.
Review found four things the offline gates could not. A run that reclaims nothing was uninterpretable, and that is the expected case rather than an edge one. NSF057I proves "saw DEAD and acted", but "swept, everything read LIVE" and "never swept at all" are both silent -- and never-swept is the likely default, since an idle executive was measured at one pass in 259 s. Two counters now separate them, so an arm reads SWEEPS=12 RECLAIMED=0 against SWEEPS=0 without anyone reasoning about where in a pass the operator drain runs. They are not sts_register counters. The NSFS build already registers ~46 and sts_render fills a fixed 512-byte buffer, so the rendered block truncates well before the end of the list: a counter added there could be one that never reaches the console, which is evidence that silently does not exist. They ride the STATS supplement, which is emitted after that block precisely so it cannot be pushed out. The operator message said RECLAIMED and nothing else, which reads as authoritative cleanup when most dead clients are never reclaimed at all. NSF058I now carries the caveat once per reclaiming sweep. Its width was measured rather than estimated: the Hercules console truncates near 107 characters and eats the tail, the natural phrasing came to 127, so it would have kept the reassuring half and dropped every word that qualifies it. The caveat leads and the count sits inside it, at 103. The two call sites looked contradictory. The periodic caller stands down while a request is in service, yet do_initapi calls the same function from inside dispatch where g_busy is set by construction. It is safe for two different reasons -- in Phase 2 because one request is in flight and it is this INITAPI, which owns no socket; in Phase 1 because no classifier is registered -- and both are now written at that call site, with the note that the Phase-2 half dies if concurrent service lands. The reclaim messages moved from 8xx to 0xx. They are executive actions and belong with NSF050I/NSF051W, which this same file emits when the transport reaps a CSA slot; 814-816 and NSF817I are things an operator asked for. Numbering the sweep with the operator verb that merely reports it conflated the two reclamation paths in the place a reader looks first. Also stated plainly rather than left to be discovered: "Phase 1 untouched" is behavioural, not byte-level -- the NSF module links the changed nsfreq.c and the new nsftime_plat.c -- and a mass reclaim is a burst of up to 64 teardowns inside one run-to-completion pass. Host 3190 -> 3204 PASS / 0 FAIL; counting a SKIPPED sweep as having looked, the exact lie the counters exist to prevent, fails 3.
The renumber moved the per-reclaim line from NSF817I to NSF057I but left three comments naming the old number, one of them in the header that documents the seam. NSF817I is now only the STATS supplement, which is what the numbering rationale says it should be.
Written as the periodic caller's own line, the summary silently exempted the other trigger -- and that trigger is the full table at INITAPI, which is the loudest and most consequential burst there is, up to 64 reclaims at once, and exactly where an operator most needs to be told that most dead clients are never reclaimed at all. A summary that covers the quiet path and not the loud one is worse than none, because it reads as complete. It now lives inside nsfreq_app_sweep behind a seam of its own, so "every sweep that reclaims, summarises" is structural rather than something a call site has to remember. Same rule the sweep itself is built on: one function, two callers, no drift. The ADR and CLAUDE.md claimed the covered behaviour before the code had it; both now describe what ships, and both record that SWEEPS counts either trigger, so arm 1 reads it with no INITAPI traffic in flight or reads it as a ceiling. Host 3204 -> 3342 PASS / 0 FAIL. Returning the summary to the call site fails 4, which is the property the seam exists for.
nsfreq_app_sweep(0) reads as unconditional, and the locked decision calls it the same sweep with the rate limit bypassed. A zero interval does defeat the rate limit, but nsftime.h gives the unbelievable-timestamp rule precedence over secs == 0, so a since later than now returns SKIPPED without looking. That needs the clock to have gone backwards since the last sweep: unreachable on MVS, where nsf_now is STCK and monotonic, reachable on the host. The cost is one INITAPI refusing with EMFILE where it might have reclaimed, and the next one works. The rule earning it is right, so this is recorded at the call site rather than weakened in the contract.
Arm 1: a cancelled STC's slot and socket were reclaimed one second after its address space ended, with the identity in NSF057I matching the STC's own announcement and D A,L identical before and after, so the run measures the sweep rather than ASID reuse. Passes came from the idle heartbeat, and the table was never full, so all 25 sweeps were periodic -- an exact count rather than the ceiling the counter would otherwise be. Arm 2: with the registry full and everything LIVE, 260 INITAPI attempts all returned EMFILE while SWEEPS gained about sixty beyond the periodic rate -- one on-demand scan per refusal, so the scan demonstrably runs before the refusal. Cancelling the STC while a 200-step job was still attempting INITAPI drew the reclaim in the same second, and the slot came back at generation 3 holding the initiator's identity: a step of that job was granted it. The same-second timing makes the on-demand path overwhelmingly likely rather than certain, and that is said rather than rounded up. The revert arm doubles as the control the gate needs. With the periodic call removed the module reports SWEEPS=0 after 66 s and 663 passes, which is impossible on the fixed module, so the revert is confirmed positively; 46 s after the same cancel the slot reads DEAD and still in use with its socket unfreed. That is the defect signature, rendered deliberately. The batch limitation is confirmed at scale in the same round: 63 ended batch clients all report the same initiator identity and every one reads LIVE. Phase 1 was verified rather than assumed -- S NSF starts clean, F NSF,APPS still draws NSF808E, ping 20/20 -- and the record now carries the half that goes missing: the deployed NSF module and main have parted company, because it links the changed nsfreq.c even though its behaviour is unchanged. Also recorded: F NSFS,APPS at a full registry is 66 WTOs, an operational consequence of 16 to 64 that appeared in no trade-off. Zero dumps. Registry left at 0 of 64, stand as found.
mgrossmann
added a commit
that referenced
this pull request
Aug 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage b adds the trigger and the reclaim for obligation #3. It ships best-effort, and
the limits are the most important thing about it —
TERMAPIremains the contract. ADR-0045(new rather than an ADR-0040 annotation), and issue #88 stays deferred.
What it is, and what it cannot do
Three rounds of measurement established that client liveness is not soundly detectable on
3.8j, so the sweep reclaims a narrow class and nothing else:
initiator does not end when a job ends.
Shipping is defensible because the error is one-directional: a false LIVE leaks a slot, and a
false DEAD cannot arise from reuse, since an ASID is unique among live address spaces. Mike's
recorded position — an explicit TERMAPI contract is too fragile, and this is interim — is
in the ADR.
Verified host-side
asm/,include/nsfvsvc.hvsmainNSFRQEfrozen at 64 B,ANCVERNO3Seven discrimination checks: widening the reclaim predicate fails 14, capturing the identity
after
app_freefails 2, an always-elapsed limiter fails 2, no on-demand sweep fails 4, abroken sub-second borrow fails 2, counting a SKIPPED sweep as "looked" fails 3, and returning
the summary to the call site fails 4.
Verified live on MVSCE
Deploy-took-effect needed a new shape, since this round adds no field to an existing line:
NSF817I APPSWEEPpresent at all is the proof, and its complement carried the revert arm.Arm 1 (periodic). Before, client alive:
SLOT 2 ASCB=00FF8D18 ASID=000C LIVE, 3 of 64 inuse,
NSFSOC opens 3 closes 0.C TSTAPPDS→ABEND S222at 06.23.29, and one secondlater
NSF057I APP SLOT 2 (TOKEN=00010002 ASCB=00FF8D18 ASID=000C) RECLAIMED. After: 2 of64 in use,
opens 3 closes 1.D A,Lidentical before and after — nothing started, so therun measures the sweep and not reuse. All 25 sweeps were periodic (the table was never full),
so the count is exact rather than a ceiling.
The batch limitation at scale: 63 ended batch clients all report the same initiator
identity and every one reads LIVE, reclaimed never.
Arm 2 (on demand). Before: registry at 64 of 64, 260 INITAPI attempts all
ERRNO=24,with
SWEEPSgaining ~60 beyond the periodic rate — one on-demand scan per refusal, so thescan runs before the refusal. After: cancelling the STC while a 200-step job was still
attempting INITAPI drew the reclaim in the same second, and the slot returned at
generation 3 holding the initiator's identity — a step of that job was granted it. The
same-second timing makes the on-demand path overwhelmingly likely rather than certain, and
that is stated rather than rounded up.
Revert, three states, one assertion moving — and the reverted arm renders §2.3's defect
signature deliberately:
SWEEPS=0after 66 s and 663 passes (impossible on the fixed module,so the revert is confirmed positively), and 46 s after the same cancel the slot reads DEAD,
still in use, socket unfreed. Restored → reclaims again.
Phase 1 verified, not assumed:
S NSFclean,F NSF,APPS→NSF808E, ping 20/20 0 %.Regression
TSTRQXC/TSTRQXF122 PASS CC 0 batch+TSO. Zero dumps; the fiveIEF450I S222are exactly the five deliberate cancels.F NSFS,APPSat a full registry is 66 WTOs (longest line 61 columns, no truncation) — anoperational consequence of 16 → 64 that appeared in no trade-off.
Not established
Anything about issue #88's eventual design; the TSO client class, unmeasured; and a bound on
the ASID-reuse window. The deployed
NSFmodule andmainhave parted company — inert, noredeploy needed, but any
NSFdeployed before this change predates it.