#92: render every counter, and make truncation visible when it happens - #95
Merged
Conversation
Two defects in two layers, both losing output silently, and both a prerequisite for the throughput round that reads counters. The operator STATS reply put one render into one 512-byte buffer and stopped at the last line that fitted, returning a byte count -- so rendering everything and stopping a third of the way in were the same observation to the caller. Each line carries its counter's value, so the cut point moved as the numbers grew. Measured on the deployed module: a header announcing 52 counters followed by 33 lines, and the missing 19 were the entire UDP set plus most TCP diagnostics. The renderer is now resumable and the caller loops until every counter is out, so the buffer bounds a chunk rather than the whole reply. That alone would have been a fix with the same shape as the bug, so the caller also counts what it emitted against what exists and says so if they differ. That warning cannot occur in production, which is why the chunk size became settable in debug builds: a warning nobody has seen fire is not designed in, it is asserted, and that is this issue one level up. The host capture ring kept its first sixty-four lines and dropped the rest -- the newer ones, not the older ones its comment claimed -- so a full registry report lost its last slot and its summary, and the reader could not tell a dropped line from one that was never emitted. Raising the bound alone would be the same defect at a new threshold, so the overflow is now countable and a test asserts it is zero. Both gates are verified to discriminate by reverting each fix alone, and the counter gate is driven at ten-digit values because a fix that works at small values and fails at large ones has not fixed anything.
The warning is forced in a debug build by shrinking the render chunk, and it has to work in the production build. If the only difference is the chunk constant then the proof carries, but that was a step the reader had to make, and an unstated step is an assumption -- the same class as the defect being fixed here. So it is now said at the test and in the PR: op_stats, its resume loop, the emitted-against-total comparison and the warning itself are all outside any debug guard. The guard changes only whether a setter exists to alter the chunk size, which production leaves at its default. The test forces the branch by varying one constant, not by running different code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#92 — the STATS renderer truncated silently, and the host ring dropped its tail
Two problems in two layers, fixed separately and gated separately. A prerequisite for
(e), which measures throughput by reading counters.
Red lines held: no counter removed or renumbered (verified in the diff), anchor layout
unmoved,
ANCVERNO3,NSFRQEfrozen,asm/nsfvsvc.asmuntouched.Problem 1 —
F NSFS,STATSrendered a third of the registry and said nothingMechanism, from source.
op_stats(src/nsfopr.c) put onests_renderinto onechar buf[512].sts_renderbreaks at the first line that would not fit and returns abyte count — so "rendered everything" and "stopped a third of the way in" are the same
observation to the caller. Each line is
"%.8s %.12s %u\n", and it carries the counter'svalue, so the cut point moves as values gain digits.
Measured live on the deployed module, before any change:
The 19 that were invisible are not a random tail — they are the entire UDP counter set
plus most TCP diagnostics:
NSFUDP in/out/binds/noport/badcksum/badlen/rxfull,NSFTCP rexmit / twreclaim / wndprobe / oooseg / dupack / rxfull / datadrop / …, andNSFSX wakeposts. Earlier rounds that quoted those figures read them throughsts_valueinside thetests, not through this report, so those results stand — but (e) reads the report.
The fix, two halves.
sts_render_from(buf, bufsize, first, *next)renders from an index andreports where it got to;
op_statsresumes until every counter is out. The buffer nowbounds one chunk, not the whole reply.
op_statscounts what it emitted againststs_count()and emitsNSF818W STATS INCOMPLETE -- RENDERED n OF mif they ever differ. This is the durablehalf: without it the next counter added past some future boundary is lost in the same
silence.
nsfsx_stats_extrastays as a mechanism but is no longer the reason a counter is placedoutside the registry — it was a workaround for this, it worked, and it was not a fix.
Problem 2 — the host capture ring dropped its tail (host-only)
Mechanism, from source.
nsfmsg_emitstores only whileg_capn < CAP_MAXbut alwaysincrements, so it retains the first 64 lines and drops the rest — the newer ones, not
the older ones as the old comment claimed.
F NSFS,APPSat a full registry emits 1 + 64 + 1= 66 lines, so the last slot and the
NSF816Isummary were gone, andnsfmsg_cap_line()returnedNULLfor them — which reads as "no such line", not "dropped".Fix:
CAP_MAX64 → 256, andnsfmsg_cap_dropped(), because a bigger number alone isthe same defect at a new threshold. The overflow is now countable, so a test asserts it is
zero instead of inferring completeness from a line count it cannot distinguish from a short
reply.
Verified host-side
make test-host3342 → 3414 PASS / 0 FAIL (TSTOPR 34 → 113).ten-digit values, reverting
op_statsalone givesFAIL: EVERY counter rendered, at ten-digit values (got 22, want 52).That 22 also measures the moving boundary — ~33 at live (small) values, 22 at
ten-digit ones. A fix that worked at small values and failed at large ones would be the
same defect at a new threshold; this is the case that catches it.
CAP_MAXalone givesdropped (got 2, want 0)plus the two missing lines — exactly the last slot and theNSF816Isummary.NSF818Wis proven to FIRE, not merely present. It is unreachable in production (aline is ~33 bytes, the chunk 512), so the chunk size became an
NSF_DEBUG-only settable —because a warning nobody has seen fire is not designed in, it is asserted, which is
this issue's own shape one level up. The test shrinks the chunk, sees
NSF818W ... RENDERED 0 OF 52, restores it, and sees the warning gone and all 52 render.an unstated step is an assumption, which is the same class as the defect being fixed.
op_stats, its resume loop, the emitted-vs-total comparison and theNSF818Wcall areall outside any
NSF_DEBUGguard; the only thing the guard changes is whethernsfopr_set_stats_chunkexists to changeg_statschunk, which production keeps atOPR_STATS_CHUNK. The test forces the branch by varying one constant, not by runningdifferent code.
(
NSFSTRNF,NSFOPSCH,NSFMSGCDnew).NSF_DEBUG-only helpers fromoutside the guard — 4 unresolved externals, invisible to the host build.
Verified live (MVSCE)
Before/after on the same stand, one assertion moving:
NSF810IheaderNSF811IlinesNSF818Wmain)NSFTCP resetsentNSFSX wakepostsNSF817I APPSWEEPstill comes last, so the supplement's ordering is unchanged.TSTSVC/TSTMVCK/TSTUBUF/TSTXFW/TSTDEATHTSTRQXC/TSTRQXFZero dumps; both STCs start and stop clean; stand left with nothing running.
Raw console captures:
docs/measurements/m5-92/stats-before.log,stats-after.log.