Skip to content

Calibrate the killswitch perf guards to the claim, not to a machine#567

Merged
xmap merged 1 commit into
mainfrom
fix/killswitch-perf-threshold
Jul 16, 2026
Merged

Calibrate the killswitch perf guards to the claim, not to a machine#567
xmap merged 1 commit into
mainfrom
fix/killswitch-perf-threshold

Conversation

@xmap

@xmap xmap commented Jul 16, 2026

Copy link
Copy Markdown
Owner

The symptom

assert float(authz["median"]) < 5.0 is red on any PR merged against main. It failed at 5.418 and 5.453 on #566, whose diff cannot touch authz at all, minutes after passing on main with the identical test. Same code, different runner.

Why it looked safe

This is the part worth knowing, because the ceiling was not careless:

  developer machine   0.253us   <- what it was calibrated against
  old ceiling         5.0us     <- ~20x the dev number; reads as generous
  shared CI runner    ~5.44us   <- ~21x SLOWER than dev for this in-memory call

Written on a machine where it had 20x headroom, it lands a hair under the real cost on a runner that is ~21x slower at a no-I/O Policy.evaluate. No author would guess that ratio, which is why the number survived review. Main cleared it by luck; the next PR did not.

Note the units: authz_decision_us is microseconds (perf_counter_ns() / 1000.0 / batch). The old gate asserted that an in-memory decision completes in under five microseconds on shared infrastructure, and the overage was 0.4us. It was measuring which CPU GitHub assigned.

The fix

Assert the shape of the claim, not a budget. Policy.evaluate is an in-memory decision with no I/O, so it costs microseconds. A regression that matters means someone put a lookup on the decision path, and that costs milliseconds.

   runner noise        ~0.1 us   |
   CI median           ~5.4 us   |=|
   new ceiling         50   us   |=========|      ~9x CI, ~200x dev
   real regression   >1000  us   |=========================...|
                                  ^ I/O on the decision path

Three orders of magnitude separate noise from a real regression, so any ceiling in that gap catches everything this test exists to catch while ignoring the runner. 50us mirrors the headroom the replay floor already had (78k/s observed against a 10k/s floor, ~8x), which is exactly why that assertion never flaked and this one did.

What is not affected

The paper's numbers. They come from a deliberate CORA_PERF_OUT run on the developer testcontainer, which writes data/killswitch_perf.json. This guard only runs when CORA_PERF_OUT is unset. Nothing the actor-symmetry paper reports changes.

The correctness assertion. runs_held == k across the K sweep is untouched: that is the actual claim (one PolicyGrantRevoked holds all K runs) and it is timing-independent.

Verification

Run against real Postgres locally (testcontainers, pgvector/pgvector:pg18): passes, median 0.253us. Both assertions now carry a message naming what a real failure would mean, and the constants carry the measurements above so the next person tightening them knows what they are fighting.

Unblocks #566.

🤖 Generated with Claude Code

`assert authz["median"] < 5.0` is red on main-merged PRs. It failed at 5.418 and
5.453 on PR #566, whose diff cannot touch authz at all, minutes after passing on
main with the same test. Same code, different runner.

The ceiling was never careless, and that is the interesting part:

  developer machine   0.253us   what it was calibrated against
  old ceiling         5.0us     ~20x the dev number; reads as generous
  shared CI runner    ~5.44us   ~21x SLOWER than dev for this in-memory call

Written on a machine where it had 20x headroom, it lands a hair under the actual
cost on a runner that is ~21x slower at a no-I/O `Policy.evaluate`. Nobody would
guess that ratio, which is why the number survived review. Main cleared it by
luck; the next PR did not.

Assert the shape of the claim instead. Policy.evaluate is an in-memory decision,
so it costs microseconds; a regression that matters means someone put a lookup on
the decision path, and that costs milliseconds. Between runner noise and a real
regression sit three orders of magnitude, so any ceiling in that gap catches
everything this test exists to catch while ignoring which CPU GitHub assigned.
50us is ~200x the dev median, ~9x the CI median, and mirrors the headroom the
replay floor already had (78k/s observed against a 10k/s floor) which is exactly
why that assertion never flaked and this one did.

The paper's numbers are unaffected: they come from a deliberate CORA_PERF_OUT run
on the developer testcontainer, not from this guard.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Coverage report

This PR does not seem to contain any modification to coverable code.

@xmap
xmap merged commit 27ef7dd into main Jul 16, 2026
16 checks passed
@xmap
xmap deleted the fix/killswitch-perf-threshold branch July 16, 2026 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant