test: scale Windows CI timing budgets from a shared helper - #275
Conversation
Windows CI runners run the suite ~3-4x slower than macOS/Linux (~25s locally, ~100s on a Windows runner), and several tests were calibrated with flat millisecond budgets on fast machines. After three Windows-only flakes surfaced, do a systematic pass instead of tuning one flake at a time. Classify each hardcoded timing value and handle it accordingly: - Add tests/helpers/timing.ts exporting CI_SLOWDOWN (3x on Windows) and budget(ms), for hang-catcher budgets that exist to fail fast on a hang rather than to measure machine speed. - Derive bounds from the real budget they police: the force-exit flush deadline (STDOUT_FLUSH_TIMEOUT_MS, moved to the leaf cli/timeouts.ts so tests import it without triggering the CLI autorun) and the daemon liveness probe (DAEMON_PROBE_TIMEOUT_MS via __daemonHostInternals). - Scale spawn-heavy per-test budgets and readiness guards via budget() across the e2e, daemon, and CLI integration suites. - Keep genuine real-time assertions (serve close promptness, the in-flight close deadline) fixed and document why they are not scaled. Mutation-checked the two non-trivial relaxed bounds: breaking the force-exit fallback and the daemon probe timeout each still fails the corresponding test, so the relaxed assertions retain their teeth. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: test: scale Windows CI timing budgets from a shared helper This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 385f6c12a2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| }, | ||
| maxBuffer: 1024 * 1024, | ||
| timeout: 10_000, | ||
| timeout: budget(10_000), |
There was a problem hiding this comment.
Add a matching Vitest timeout for the scaled child wait
On Windows, budget(10_000) gives this execFile up to 30s, but the only it that calls runCli in this file still relies on the suite-wide Vitest timeout of 10s. In the slow or hung CLI case, Vitest aborts the test before this scaled child timeout can fire, so the added Windows headroom is ineffective and can leave the spawned CLI running until execFile kills it later; add a matching budget(...) timeout to that test or keep the child timeout within 10s.
Useful? React with 👍 / 👎.
Why
Windows CI runners run this suite ~3–4x slower than macOS/Linux (~25s locally, ~100s on a Windows runner), and several tests were calibrated with flat millisecond budgets on fast machines. After three Windows-only flakes surfaced in one session (
runtime-stdio-close,e2e-fixture-servers,runtime-inflight-close), this does a systematic pass instead of tuning one flake at a time.What
Every hardcoded millisecond timeout / timing assertion in the suite was classified into one of three buckets and handled accordingly:
tests/helpers/timing.tsexportsCI_SLOWDOWN(3× on Windows, 1× elsewhere) andbudget(ms). This is for hang-catcher budgets that exist to fail fast on a hang, not to measure machine speed. The helper's doc comment says so explicitly.STDOUT_FLUSH_TIMEOUT_MSwas moved fromcli.tsinto the leaf modulecli/timeouts.ts(and re-exported), so the truncation test imports it without triggering the CLI's auto-run-on-import.DAEMON_PROBE_TIMEOUT_MSis exposed via__daemonHostInternalsand the hung-daemon test derives its budget from it.stopChildteardown waits and readiness guards scale viabudget().budget().serve.test.ts's 750ms close-promptness check and the in-flight close deadline stay fixed (closing an in-process HTTP server has no per-platform cost) and are documented as such.Proof
The formatting churn in the diff (e.g.
daemon.integration.test.ts) isoxfmt's mandated multilineit(name, fn, timeout)form whenever a trailing timeout arg is present — same reflow the earlierf0bd0d8commit accepted.Follow-up
Only a Windows CI run confirms the 3× factor is right. If it still flakes there, bump
CI_SLOWDOWNrather than individual budgets.🤖 Generated with Claude Code