Skip to content

test: de-flake the search stress benchmark under parallel suite load#10

Merged
resure merged 3 commits into
mainfrom
claude/infallible-easley-8ae5cb
Jul 5, 2026
Merged

test: de-flake the search stress benchmark under parallel suite load#10
resure merged 3 commits into
mainfrom
claude/infallible-easley-8ae5cb

Conversation

@resure

@resure resure commented Jul 5, 2026

Copy link
Copy Markdown
Owner

What changed

The "keeps a 4000-note keystroke burst responsive" test in src/search.stress.test.ts asserted indexed <= onTheFly * 1.1 + 5 from a single timing sample of each path. Under a parallel full-suite vitest run, another worker preempting one burst mid-run inverted the comparison (observed: on-the-fly 37.0ms vs indexed 66.8ms → fail; 2 of 3 full-suite runs on 2026-07-05). It always passed in isolation.

The test now:

  • Interleaves best-of-5 runs of each path and keeps the minimum — the min discards preempted samples, and interleaving keeps the comparison fair under drifting load. On-the-fly still runs first in each pair so the indexed path can't be flattered by a cold JIT.
  • Widens the slack to × 3 + 10ms — absorbs the observed ~1.8× scheduler-noise inversion, but a real order-of-magnitude regression in the indexed path (the thing the 2026-06 perf-audit guard, merged via afa46c2, exists to catch) still fails loudly.
  • Keeps the absolute < 1500ms responsiveness budget unchanged, and renames the assertion wording from "never slower" to "within noise of on-the-fly" to match.

Why

The test's intent is a regression guard on the precomputed-lowercase index (searchNotes's lowerById), not a strict microbenchmark race — the old bound made it a coin flip under machine load.

Reviewer notes

  • Verified: passes in isolation and in two consecutive full-suite runs (868/868), where the best-of-5 numbers show the index winning comfortably (1.16× and 1.25× faster). Lint + typecheck clean.
  • The extra runs add ~300ms of test time.
  • The third test in the file (cappedMs < uncappedMs) is the same class of single-sample comparison but with a ~7–9× real margin, so it's far less exposed — left untouched to keep this scoped to the reported flake.

🤖 Generated with Claude Code

resure and others added 3 commits July 5, 2026 17:23
The keystroke-burst test compared a single timing sample of the indexed
vs on-the-fly search paths with only 1.1x slack, so one vitest-worker
preemption mid-burst inverted the comparison (observed: on-the-fly
37.0ms vs indexed 66.8ms in 2 of 3 full-suite runs). Now it interleaves
best-of-5 runs of each path (the min discards preempted samples) and
allows 3x slack — still loud on an order-of-magnitude regression in the
indexed path, which is what the 2026-06 perf-audit guard is for. The
absolute 1500ms responsiveness budget is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pre-existing sort-imports error from 5374744 that fails the verify job
on any branch; behavior-neutral eslint --fix reorder.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The sibling `caps the body-occurrence count` test still compared single
raw timing samples (`cappedMs < uncappedMs`, `cappedMs < 50`), leaving it
exposed to the exact worker-preemption flakiness the burst test was just
hardened against. Apply the same best-of-5 `min` sampling: interleave the
uncapped reference walk and the real capped `searchNotes` run, keep the
fastest of each. The BODY_FREQ_CAP regression guard is preserved — an
uncapped scorer still blows past both bounds even at best-of-5.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@resure resure merged commit e112eaa into main Jul 5, 2026
2 checks passed
@resure resure deleted the claude/infallible-easley-8ae5cb branch July 5, 2026 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant