Skip to content

test: scale Windows CI timing budgets from a shared helper - #275

Merged
steipete merged 1 commit into
mainfrom
claude/gifted-vaughan-0ccdf2
Aug 3, 2026
Merged

test: scale Windows CI timing budgets from a shared helper#275
steipete merged 1 commit into
mainfrom
claude/gifted-vaughan-0ccdf2

Conversation

@steipete

@steipete steipete commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Why

Windows CI runners run this suite ~3–4x slower than macOS/Linux (~25s locally, ~100s on a Windows runner), and several tests were calibrated with flat millisecond budgets on fast machines. After three Windows-only flakes surfaced in one session (runtime-stdio-close, e2e-fixture-servers, runtime-inflight-close), this does a systematic pass instead of tuning one flake at a time.

What

Every hardcoded millisecond timeout / timing assertion in the suite was classified into one of three buckets and handled accordingly:

  • Shared helper — new tests/helpers/timing.ts exports CI_SLOWDOWN (3× on Windows, 1× elsewhere) and budget(ms). This is for hang-catcher budgets that exist to fail fast on a hang, not to measure machine speed. The helper's doc comment says so explicitly.
  • Derived from the real budget — bounds now track the source constant they police:
    • The force-exit flush deadline STDOUT_FLUSH_TIMEOUT_MS was moved from cli.ts into the leaf module cli/timeouts.ts (and re-exported), so the truncation test imports it without triggering the CLI's auto-run-on-import.
    • The daemon liveness probe DAEMON_PROBE_TIMEOUT_MS is exposed via __daemonHostInternals and the hung-daemon test derives its budget from it.
    • The e2e stopChild teardown waits and readiness guards scale via budget().
  • Hang-catchers scaled — spawn-heavy per-test budgets and readiness guards across the e2e, daemon, and CLI integration suites now use budget().
  • Genuine perf assertions kept fixedserve.test.ts's 750ms close-promptness check and the in-flight close deadline stay fixed (closing an in-process HTTP server has no per-platform cost) and are documented as such.

Proof

  • The two non-trivial relaxed bounds were mutation-checked: breaking the force-exit fallback and breaking the daemon probe timeout each still fails the corresponding test, so the relaxed assertions retain their teeth.
  • 134 tests across the 12 affected files pass locally; typecheck, oxlint, and oxfmt are clean.
  • autoreview (codex, gpt-5.6-sol, high) clean; TruffleHog clean.

The formatting churn in the diff (e.g. daemon.integration.test.ts) is oxfmt's mandated multiline it(name, fn, timeout) form whenever a trailing timeout arg is present — same reflow the earlier f0bd0d8 commit accepted.

Follow-up

Only a Windows CI run confirms the 3× factor is right. If it still flakes there, bump CI_SLOWDOWN rather than individual budgets.

🤖 Generated with Claude Code

Windows CI runners run the suite ~3-4x slower than macOS/Linux (~25s
locally, ~100s on a Windows runner), and several tests were calibrated
with flat millisecond budgets on fast machines. After three Windows-only
flakes surfaced, do a systematic pass instead of tuning one flake at a
time.

Classify each hardcoded timing value and handle it accordingly:

- Add tests/helpers/timing.ts exporting CI_SLOWDOWN (3x on Windows) and
  budget(ms), for hang-catcher budgets that exist to fail fast on a hang
  rather than to measure machine speed.
- Derive bounds from the real budget they police: the force-exit flush
  deadline (STDOUT_FLUSH_TIMEOUT_MS, moved to the leaf cli/timeouts.ts so
  tests import it without triggering the CLI autorun) and the daemon
  liveness probe (DAEMON_PROBE_TIMEOUT_MS via __daemonHostInternals).
- Scale spawn-heavy per-test budgets and readiness guards via budget()
  across the e2e, daemon, and CLI integration suites.
- Keep genuine real-time assertions (serve close promptness, the
  in-flight close deadline) fixed and document why they are not scaled.

Mutation-checked the two non-trivial relaxed bounds: breaking the
force-exit fallback and the daemon probe timeout each still fails the
corresponding test, so the relaxed assertions retain their teeth.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@clawsweeper

clawsweeper Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: test: scale Windows CI timing budgets from a shared helper This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 385f6c12a2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

},
maxBuffer: 1024 * 1024,
timeout: 10_000,
timeout: budget(10_000),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Add a matching Vitest timeout for the scaled child wait

On Windows, budget(10_000) gives this execFile up to 30s, but the only it that calls runCli in this file still relies on the suite-wide Vitest timeout of 10s. In the slow or hung CLI case, Vitest aborts the test before this scaled child timeout can fire, so the added Windows headroom is ineffective and can leave the spawned CLI running until execFile kills it later; add a matching budget(...) timeout to that test or keep the child timeout within 10s.

Useful? React with 👍 / 👎.

@steipete
steipete merged commit b1640df into main Aug 3, 2026
5 checks passed
@steipete
steipete deleted the claude/gifted-vaughan-0ccdf2 branch August 3, 2026 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant