feat(perf): take both sides of the PERF-P2 window on one box, and refuse the wrong box (PERF-P2-CEILING) - #65
Open
NSchatz wants to merge 5 commits into
Open
feat(perf): take both sides of the PERF-P2 window on one box, and refuse the wrong box (PERF-P2-CEILING)#65NSchatz wants to merge 5 commits into
NSchatz wants to merge 5 commits into
Conversation
…use the wrong box (PERF-P2-CEILING)
PERF-P2's open question was never the false-alarm RATE. It is whether any constant
RATIO_CEILING sits above every false alarm and below every genuine O(n^2) signal, on
the runner class the gate would actually run on. That is one number,
window = weakest signal / worst false alarm
and a window at or below 1.0x means the two distributions overlap, so no ceiling
separates them at all. Three windows had been read (2.54x, 1.33x, 1.07x) and not one
of them came from a quiet GitHub-hosted 2-vCPU runner. This is O-P2-2, the experiment
that can.
experiments/perf-p2-window/window.mjs derives the figure from the raw rows. It was
computed by hand in prose before, over columns that mislead in the direction that
reads as "it separates": a ratio above the ceiling makes scalingGate THROW, so that
run has no measured row at all, and reading the measured column's max as the worst
false alarm reads the worst NON-FIRING run and drops every fire. The measured and
censored populations are pooled, and a row whose firing axis was inferred rather than
read out of the gate's own diagnostic is reported as unattributed rather than filed
under a guess. Run against the perf-p2-measurements-2026-08-05 branch it reproduces
that branch's hand-computed figures exactly (worst false alarm 8.1700, worst
non-firing 7.6342, count-axis max 5.0354, weakest signal 8.7376 at 500 to 2000 and
6.9568 at 1000 to 4000, 1.07x at hl7's fixture), which is the check that makes a new
number out of it worth anything.
experiments/perf-p2-window/run.sh takes both legs back to back on one box, because a
window is a difference and two sides from two machines are not one. Neither test file
is copied or modified: it launches ../perf-p2-false-alarm/false-alarm.test.ts and
../perf-calibration/signal-check.test.ts through their own vitest configs. What is
duplicated is a for loop, not a measurement. The sibling run.sh is deliberately not
called, because it archives and rewrites its own committed dataset on start.
It REFUSES to run on the wrong box, which is the failure this arc actually paid for.
The container class moved 6x underneath the experiment and two readers in one session
got the wrong answer from nproc. The census reads the cgroup quota when one is set and
the host count otherwise, so it is right inside a container (where nproc reports the
host) AND on a GitHub-hosted VM (where there is no quota and nproc is the only
answer). Both raw values land in the provenance, and ALLOW_ANY_BOX=1 is recorded as
boxAsExpected: false rather than passing silently.
.github/workflows/perf-p2-window.yml is workflow_dispatch only and is NOT a gate, the
same posture as perf-calibration.yml. Nothing here wires test:perf into CI or touches
the umbrella's verify policy: the founder decision of 2026-08-07 merged the kit with
the gate deliberately out of CI, and settling the ceiling is the CONDITION on
adoption, not something adoption can precede. If the window this produces is at or
below 1.0x, the answer is a sampling-shape change (ADR 0001 section 3), which is a
founder call and an ADR revision, not a new constant.
No changeset: no published package surface moves. The root workspace is private and
its CHANGELOG entries are dated rather than versioned.
…at were not true (PERF-P2-CEILING) Pass 1 of the gate-refuter on 1bb3b52 returned REFUTED with two INTRODUCED majors. Both are real, both were re-derived from the source before being fixed, and neither needed the slice re-shaped. MAJOR 1: the decisive pooled figure was narrowed by a default whose justification was FALSE, in the direction that reads as "it separates". window.mjs defaulted --fixture-floor to 500 and printed, as fact, that assertScalingGateFires refuses a fixture smaller than that. It does not. self-check.ts has no fixture-size rule at all: it refuses an output mismatch, a base phase under MIN_PHASE_MS, an unsettled warmup, and a signal that does not clear the ceiling, and that last one is a per-run refusal rather than a size rule. ADR 0001 section 5's own table blesses 250 to 1000 at a 1.22x window and rules out only 125 to 500. On the archive rows the exclusion moved the figure from 0.7247x to 0.8515x. The default is now 0, so every measured size decides the figure; the flag remains, because "what if only fixtures of at least N were allowed" is a real question, but it is now the caller's question and the excluded sizes are printed with the reason the shipped kit does not rule them out. The false sentence is recorded rather than quietly deleted, because the arithmetic it moved is the headline. MAJOR 2: the workflow claimed BOTH sides are measured under the shipped warmup rule. The signal leg is PERF-P0 Experiment C unchanged, and its warmup is three fixed passes: not warmUp(), not assertScalingGateFires, no warmup-stability refusal. ADR 0001 section 2 requires the re-check "on both sides"; this re-checks the noise side. The remedy is to correct the claim, not to rebuild the leg, because running it unchanged is exactly what keeps these rows comparable with P0's and with the 2026-08-05 archive, and changing it is an ADR 0001 question. The workflow header, the README's comparison table and its trap list now all say which side carries the caveat. Four minors, all taken: * the README claimed "every one of those matches that branch's own README" over a list that included the pooled 0.8515x. That figure is a NEW definition and appears nowhere in that branch. The six figures that DO match are now named as the reproduction, and the pooled one is explicitly marked as not corroborated by it. * the per-size window table paired a swept signal against a noise population that is NOT swept: false-alarm.test.ts runs at one fixture, so worst noise is one number repeated down the table. window.mjs now says so in its output, and the README lists it as a trap. * upload-artifact now runs with if: always(). Every refusal in run.sh fires AFTER all the measuring is done, so the failure mode guarded is 90 minutes of runner time with zero rows kept, which is the same argument the file already makes for cancel-in-progress: false. * experiments/perf-p2-window/data/ is gitignored. Unlike its two siblings this directory ships no committed dataset, so a local harness run could otherwise be swept into a commit and read later as evidence from a runner class it was never taken on. Verified after the change: prettier, check-no-emdash and actionlint clean; window.mjs still reproduces the archive's six hand-computed figures; the harness still runs end to end; the gitignore rule matches.
…er than the fact (PERF-P2-CEILING)
Pass 2 returned NOT REFUTED: all six of pass 1's findings closed, no INTRODUCED
blocker and no INTRODUCED major, the slice converged. It named three minors and two
backlog lines, every one of them a sentence scoped wider than what is true. They are
corrected here rather than filed, because all five sit in text this slice itself wrote
and each is a one-sentence narrowing, not a change of behaviour. No production byte
and no arithmetic moves in this commit.
* P0's fixed-count warmup caveat was re-scoped to its NOISE side by the previous
commit. That is wrong: P0's signal side is Experiment C, whose warmup is the same
three fixed passes, so BOTH sides of the 2.54x reading are fixed-count. Scoping it
to noise makes the most optimistic prior reading look better founded than it is,
which is the wrong direction. The README table cell is back to its unscoped
wording, and the README and workflow now say "both sides" where they name it.
* "Every refusal in run.sh fires AFTER all the measuring is done" was a false
universal: four refusals (Node major, missing vitest, missing sibling config, the
box census) fire before any measuring, and the signal loop can exit mid-sweep. The
comment now says which refusal `if: always()` is actually for (the row-count
assert, which does fire last) and what happens on the early ones. The guard is
unchanged; only its argument was over-broad.
* run.sh described data/environment.json as "the committed file". After the
gitignore rule it can never be committed. The prettier --write is still right, and
for a reason worth stating: format:check globs the FILESYSTEM and .prettierignore
does not list this path, so an unformatted local copy reds it whether or not git
tracks it.
* The README claimed none of the three prior windows was taken on the runner the
gate would run on, directly above a table whose first row is GitHub-hosted 2 vCPU.
Two were on the wrong box; the third is disqualified by its warmup rule and sample
depth. Says that now.
* The CHANGELOG's "reproduces every figure that branch computed by hand" is a
universal the tool does not satisfy (the archive also read warmup bands and phase
timings by hand, which this prints none of). It now mirrors the README's precise
version: each of the six figures it prints for that dataset appears in that
branch's README.
Verified after the change: prettier --check over the whole tree, check-no-emdash,
actionlint and shellcheck all clean.
…and two twins of a sentence already corrected (PERF-P2-CEILING) Pass 3 returned REFUTED on one INTRODUCED major, and it is right. Measured before fixing: prettier 3.8.3's default --ignore-path is [.gitignore, .prettierignore], so a gitignored path is never checked. With an unformatted experiments/perf-p2-window/data/environment.json planted, `pnpm format:check` prints "All matched files use Prettier code style!" and exits 0; the same file with --ignore-path .prettierignore (so .gitignore is not consulted) warns. My comment asserted the opposite mechanism, in the repo that ships @cosyte/prettier-config and defines format:check for every scaffolded parser, two commits after scripts/parser-template/.prettierignore recorded the true mechanism from a measurement. Two committed statements in one tree saying opposite things, the false one newer. The remedy is a deletion, not a rewrite. The sibling sweeps normalise their provenance file because theirs IS committed; mine is gitignored, so nothing reads it and the step had no reason to exist once its reason was false. The `prettier --write` line is gone and the comment now records why this driver differs from its siblings, citing the mechanism rather than restating it. JSON.stringify(..., null, 2) is the formatting, and the smoke run confirms the file it writes is already 2-space indented. Also corrected, because they are the same sentence pass 2 fixed in the README and leaving two known-false twins behind would be indefensible: the workflow header and the CHANGELOG both still claimed none of the three prior readings was taken on the runner the gate would run on, directly above material naming PERF-P0's GitHub leg as exactly that runner class. Both now say what the README says: two on the wrong box, the third disqualified by its warmup rule and sample depth. And one name: README said `signal-check.ts`; the file is `signal-check.test.ts`. Pass 3's other two findings are FILED, not remedied. Both are completeness nits on sentences that are true as written: the workflow's refusal taxonomy omits the mid-sweep signal-loop exit (which under-sells its own `if: always()` guard rather than over-selling it), and the CHANGELOG says "the six figures it prints" where the README enumerates them. A fourth remedy round on prose is the runaway ADR 0016 exists to stop. Verified after the change: prettier --check over the whole tree, check-no-emdash, actionlint and shellcheck all clean, and the harness still runs end to end.
…2's box does not exist (PERF-P2-CEILING) STOP AND CUT BACK, per ADR 0016. Four passes are spent, the cap. Pass 4 returned REFUTED on one INTRODUCED major and there is no fifth pass to grade a further remedy, so this commit is UNGRADED and is disclosed as such here, in the PR, and in the experiment's own README. IT IS NOT BEING MERGED. WHAT PASS 4 FOUND, and it reaches past the prose. The previous commit rewrote two sentences to say the third prior window came from "a GitHub-hosted 2-vCPU runner". The repo's own committed measurement says otherwise: experiments/perf-calibration/data/github-hosted/environment.json records "cpuCount": 4 on an AMD EPYC 7763, and ANALYSIS.md section 7 tabulates the same. The BASE sentences pass 3 asked me to "correct" were already true, and my correction made them false. They are restored verbatim rather than reworded, which is the deletion-shaped route: this commit asserts strictly less than the graded state did on that point. THEN I MEASURED THE RUNNER RATHER THAN TRUSTING A THREE-WEEK-OLD CAPTURE, because "the runner is the authority" is the one rule this whole arc paid for. Dispatched perf-calibration.yml --mode bc-only on main (run 31399946419, 2026-08-10): GitHub-hosted ubuntu-latest is 4 vCPU / 15.6 GiB, AMD EPYC 9V74. Note the host generation moved from EPYC 7763 to EPYC 9V74 between P0's leg and today at an UNCHANGED image string 20260720.247.2, which is ADR 0001's review-trigger territory in its own right. SO THE BOX O-P2-2 NAMES DOES NOT EXIST AS ubuntu-latest, and run.sh (EXPECT_CPUS=2) exits 66 before measuring anything. That is the box census doing exactly its job: it caught the mismatch before a figure could be labelled O-P2-2 and filed against a box it was never taken on, which is the failure mode that produced three readings on three classes. The guard earned itself before the first real run. EXPECT_CPUS IS LEFT AT 2 AND THE WORKFLOW IS LEFT REFUSING, DELIBERATELY. Measuring ubuntu-latest as it actually is answers "the class config's CI really uses", which may well be the better question, but it is a different experiment from the one that was authorised, and choosing it is a founder call. Raising the constant to make the job green would be picking the box, which this slice must not do. The refusal message, the workflow header and the README now all carry the measured fact and say so. Also in this commit, both narrowings and neither new: the workflow no longer calls PERF-P0's leg "this runner class"; the README's table cell records the measured 4 vCPU; and run.sh's provenance comment names the ONE sibling that normalises its file rather than saying "the sibling sweeps" (pass 4 finding 2, minor). Pass 4's other observation, filed not fixed: the repo holds two incompatible models of what ubuntu-latest is, and reconciling them is part of the same founder call.
Contributor
Author
🛑 HOLDING THIS PR UNMERGED. The gate refused, and the experiment's target box does not exist.Refuter passes: 4 of 4, the ADR 0016 cap. The blocker, measured rather than reasonedO-P2-2 says "a quiet GitHub-hosted 2-vCPU runner". It is not what
What is NOT being decided here
State of the workThe harness itself is complete and verified: Needs a founder call before this merges. Recorded in the experiment's README, the workflow header and |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
PERF-P2's remaining question is not the false-alarm RATE. It is whether any constantRATIO_CEILINGsits above every false alarm and below every genuine O(n^2) signal, on the runner class the gate would actually run on. That is one number:A window at or below 1.0x means the two distributions overlap, so no ceiling separates them. Three windows had been read and not one came from a quiet GitHub-hosted 2-vCPU runner:
#34's own calibrationThis is O-P2-2, the experiment that can settle it.
What landed
experiments/perf-p2-window/window.mjsderives the figure from the raw rows. It was computed by hand in prose before, over columns that mislead in the direction that reads as "it separates": a ratio above the ceiling makesscalingGatethrow, so that run has no measured row at all, and reading the measured column'smaxas the worst false alarm reads the worst NON-FIRING run and drops every fire. Measured and censored populations are pooled; a row whose firing axis was inferred rather than read out of the gate's diagnostic is reported asunattributedrather than filed under a guess.experiments/perf-p2-window/run.shtakes both legs back to back on ONE box, because a window is a difference and two sides from two machines are not one. Neither test file is copied or modified: it launches the sibling experiments' own test files through their own vitest configs. What is duplicated is aforloop, not a measurement.nprocreports the host) AND on a GitHub-hosted VM (where there is no quota andnprocis the only answer). Both raw values land in the provenance..github/workflows/perf-p2-window.ymlisworkflow_dispatchonly and is NOT a gate, the same posture asperf-calibration.yml.The check that makes a new number worth anything
Run against the
perf-p2-measurements-2026-08-05branch,window.mjsreproduces that branch's hand-computed figures exactly: worst false alarm 8.1700 (the fire, censored out of the measured column), worst non-firing 7.6342, count-axis max 5.0354, weakest signal 8.7376 at500 -> 2000and 6.9568 at1000 -> 4000, and 1.07x athl7's fixture. The README carries the exact commands.What this does NOT do
Nothing here wires
test:perfinto CI or touches the umbrella'sverify-policy.json. The founder decision of 2026-08-07 merged the kit with the gate deliberately out of CI, and settling the ceiling is the CONDITION on adoption. If the window this produces is at or below 1.0x, the answer is a sampling-shape change (ADR 0001 section 3), which is an ADR revision and a founder call, not a new constant.No changeset: no published package surface moves. The root workspace is private and its CHANGELOG entries are dated rather than versioned.