Skip to content

Share vitest workers for node-environment tests and cut fixed per-file costs - #2303

Merged
SawyerHood merged 3 commits into
mainfrom
bb/optimize-test-suite-performance-thr_2ezguwcmef
Aug 22, 2026
Merged

Share vitest workers for node-environment tests and cut fixed per-file costs#2303
SawyerHood merged 3 commits into
mainfrom
bb/optimize-test-suite-performance-thr_2ezguwcmef

Conversation

@SawyerHood

@SawyerHood SawyerHood commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator

What was wrong

The test suites spent most of their time outside the tests. Profiling every package uncontended showed that 80–90% of the big suites' CPU was per-file overhead: vitest re-imported the whole module graph, built jsdom, and ran the setup file for each of @bb/app's 422 files; the setup file's @bb/shared-ui/icon-extended import pulled @hugeicons/core-free-icons, a barrel that re-exports 5,122 one-icon modules and takes ~0.7 s to load; and every @bb/server test harness replayed all 107 SQLite migrations (57 of the 61 ms a harness costs, paid by 1,922 tests). @bb/server and @bb/agent-runtime already set isolate: false, but vitest 4 only hands a finished worker the next queued file when that file has the same project and environment, so the interleaved queue still churned workers.

Per-test profiling (JSON reporter) then showed fixed costs inside the slowest suites: the templates external-scaffold test ran four cold npm installs of the packed SDK (~13 s each); db data suites migrated per test (~17 s of their 30 s); every integration harness rebuilt the first-party provider bridges with esbuild (~0.6 s); provider-parity's 43 replay cells are bridge children that mostly wait on settle/drain pacing, so the suite was wall-clock bound by vitest's default maxConcurrency of 5; and two Pi SDK tests slept through eight real 250 ms retry delays each.

What changed

  • vitest.shared.ts exports sharedWorkerProjects(). It scans a package's test files and splits them into shared-worker projects (isolate: false, one per non-DOM environment) and an isolated project for files that mutate worker-global state — vi.mock/vi.stubGlobal/vi.stubEnv/vi.resetModules, process.env writes, process.chdir, assignments to window/globalThis/document/navigator/prototypes (including through a cast), Object.defineProperty/Reflect.* on them — directly or through a test helper they import. Files that run in jsdom/happy-dom stay isolated: pushing the app's jsdom files through one worker failed a different file on every ordering (portals, focus, media-query caches keyed by the shared document), which no source scan can enumerate. Every project keeps the package's own include globs and excludes the other projects' files, so the split never changes which files run.
  • SharedWorkerSequencer (installed by defineWorkspaceTestConfig and the root config) orders the run queue: isolated files first, then each shared project contiguously. This is what turns isolate: false into worker reuse without groupOrder phase barriers, which left workers idle at each phase tail.
  • Tests alias the bare @hugeicons/core-free-icons specifier to the package's self-contained minified bundle (identical exports; 45 ms instead of ~700 ms per worker).
  • 23 package vitest.config.ts files use the helper; @bb/server and @bb/agent-runtime drop their hand-rolled split.
  • apps/server/test/helpers/test-app.ts adds createTestDb(), which migrates once per worker and opens each harness from the serialized image; createConnection accepts a Buffer for that. packages/db/test/helpers/migrated-connection.ts does the same for the db data suites (suites that exercise migrate itself still call it directly).
  • packages/templates/test/plugin-scaffold-external.test.ts installs the packed SDK once per file and symlinks each scaffold's node_modules at it, passes --no-audit --no-fund --prefer-offline, and overlaps the backend and frontend typechecks.
  • apps/server/test/helpers/provider-registry.ts memoizes the first-party bridge artifact build per worker process.
  • packages/provider-parity/vitest.config.ts sets maxConcurrency: 16 (measured pinned to 4 CPUs: 37 s at 5, 21 s at 10, 15 s at 16, all green).
  • Two Pi SDK retry tests drive the retry delay with fake timers.
  • internal-skill-trees.test.ts pins the fixture's file mode (it failed under umask 002), @bb/qa gets a 15 s test timeout, and AGENTS.md documents the convention.

How you verified

Clean, uncontended runs on a 16-core machine: @bb/app 63.0 s → 38.6 s, @bb/server 47.6 s → 32.0 s, @bb/mobile 4.4 s → 2.4 s, @bb/provider-parity 36.5 s → 14.0 s, @bb/integration-tests 24.8 s → 16.6 s, @bb/db 7.5 s → 3.5 s, the templates external test 27.3 s → 23.5 s with a warm npm cache (three cold installs fewer in CI), sdk-session.test.ts 4.0 s → 0.3 s; other packages at parity or slightly faster. pnpm exec turbo run test --force --continue 3 m 27 s → 2 m 22 s with everything contending, 71 of 72 tasks passing. The one failure, @bb/qa's "starts a detached daemon repeatedly" test, times out on the development machine before and after this change. Stress: every shared project run through a single worker (--project <name> --maxWorkers=1) passes for app (149 files), server (184), tasks, host-daemon, and cli; vitest run --shard=1/3 works for the app. pnpm exec turbo run typecheck passes for db, server, app, templates, integration-tests, provider-parity, and agent-runtime; lint passes; tsc --noEmit passes on vitest.shared.ts.

No linked issue.

AGENT GENERATED

kirbyhood and others added 3 commits August 22, 2026 08:30
…e costs

Re-importing the module graph for every test file was 80-90% of the big
suites' CPU. vitest.shared.ts now exports sharedWorkerProjects(), which
splits a package's tests into shared-worker projects (isolate: false) and
an isolated project for files that mutate worker-global state (vi.mock,
stubs, process.env, global assignments, including through test helpers)
or run in a DOM environment. A SharedWorkerSequencer orders the queue so
vitest actually reuses workers. Tests alias @hugeicons/core-free-icons to
its single-file bundle (45ms instead of ~700ms per worker), and the server
harness clones a migrated SQLite template instead of replaying 107
migrations per test.

Co-Authored-By: Claude <noreply@anthropic.com>
…nd Pi suites

The templates external-scaffold test ran four cold `npm install`s of the
packed SDK (~13s each); it now installs once per file, symlinks every
scaffold's node_modules at that install, skips audit/fund round trips,
and overlaps the backend and frontend typechecks. db data suites open a
migrated in-memory template instead of replaying 107 migrations per test.
The integration harness memoizes the first-party bridge artifact build per
worker (~0.6s of esbuild per test). provider-parity raises maxConcurrency
to 16: each replay cell is a bridge child that mostly waits on pacing, so
the suite was wall-clock bound by vitest's default of 5 (37s -> 15s on 4
CPUs). Two Pi SDK retry tests drive the 250ms retry delay with fake timers
instead of sleeping through eight of them.

Co-Authored-By: Claude <noreply@anthropic.com>
Packages whose tsconfig covers their vitest config typecheck
vitest.shared.ts with exactOptionalPropertyTypes, which rejects passing
an undefined value to an optional property.

Co-Authored-By: Claude <noreply@anthropic.com>
@SawyerHood
SawyerHood merged commit 08363a8 into main Aug 22, 2026
15 checks passed
@SawyerHood
SawyerHood deleted the bb/optimize-test-suite-performance-thr_2ezguwcmef branch August 22, 2026 16:09
lnittman added a commit to lnittman/bb that referenced this pull request Aug 22, 2026
…id a spine

Four surfaces, one argument: the page states that bb builds itself and then
never shows you which work was bb's.

**The feed now marks it.** `agentMergedLastMonth` was already being measured —
a search for the "AGENT GENERATED" line the repo requires of agent-created
PRs — and then never rendered. The same query now also writes pr-feed.json, so
each row carries an `agent` flag from the identical source the aggregate is
counted from and the two can never disagree. Both searches had to be ordered
by creation date first: at the default best-match sort their pages shared
almost nothing, which scored a feed of zero agent PRs against a 60% aggregate.
Seventeen of the last eighteen are agent-written, verified against the real
bodies of get-bb#2303, get-bb#2274 and get-bb#2267, with get-bb#2258 correctly unmarked.

The marker is deliberately quiet. On seventeen of eighteen rows it is not a
differentiator, it is a column the eye reads straight down — and the one row
without it is what makes the rest credible.

**The stat block trades forks for it.** 436 sits beside 721 so the adjacency
does the arithmetic without a percentage. The fork count said nothing a reader
could act on; this is the headline restated as a measurement.

**The bento gets spans.** Ten columns instead of two halves, 6-4 / 4-6.
Subagents leads wide because the tree needs room for its indent guides to
mean anything, while a diff hunk is narrow by nature and reads fine at four.
Uniform halves were also why the phones sat in cards built for a window. The
heading becomes a chip, which stops it competing with the app's own headings
inside the mock, and `text-wrap: nowrap` goes — it was truncating every
description that did not happen to fit, a promise unkeepable once the tiles
stopped being the same width. Three stacked shadows replace the single blur.

**The footer becomes an index.** Seven links on one middot-separated line was
a run-on sentence with 17px hit areas; they are now three named groups of
28px rows in monospace uppercase, a voice used nowhere else on the page
because the footer is the one part that is pure address.

Also: the closer mark rests on the riso print. It rested on bb-icon.png — the
192px macOS app icon, whose rounded tile is painted into the artwork, so the
mark sat on a plaque no CSS could remove — and hid the good drawing behind a
click most readers would never make.

Feed tracks widen to two columns: at three, every row had to fit an avatar, a
title, the marker and "get-bb#2303 · Aug 22" in 320px, and the title was what got
the ellipsis.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants