Skip to content

test: stop CI flakes from tight wall-clock deadlines#103

Merged
lahfir merged 1 commit into
mainfrom
fix/flaky-deadline-tests
Jul 20, 2026
Merged

test: stop CI flakes from tight wall-clock deadlines#103
lahfir merged 1 commit into
mainfrom
fix/flaky-deadline-tests

Conversation

@lahfir

@lahfir lahfir commented Jul 20, 2026

Copy link
Copy Markdown
Owner

Summary

A class of unit tests bounds real retry/resolve/hydration work with a tight wall-clock deadline (100–1000 ms) and asserts a definite outcome. On slow, contended GitHub macOS runners the work overran the budget and the test got a Timeout instead of its asserted outcome, flaking the Test job while every local run passed. Three had already surfaced on CI (selected_hydration, drag_retries...occlusion, terminal_stale_ref...).

This widens those budgets to the suite's existing 5 s idiom so the asserted state is reached deterministically. 13 sites across 12 test files; test-only, no production change.

Genuine deadline-expiry tests keep their tight budgets — permanently-unresolvable adapters, single-shot timeout_ms 0, deadline-value plumbing, and lock-contention tests all expire correctly regardless of runner speed, so widening them would be wrong (or slow) and they were left untouched.

Verification

  • cargo test --lib -p agent-desktop-core — 978 pass, 0 fail
  • cargo test --lib --workspace — all suites pass
  • cargo fmt --all -- --check — clean
  • Diff is 13 one-for-one numeric budget swaps; no production code touched

…akes

A class of unit tests bounds real retry/resolve/hydration work with a
tight wall-clock deadline (100-1000 ms) and asserts a definite outcome
(success, or a specific terminal code after a bounded retry). On slow,
contended GitHub runners the work overran the budget and the test got a
Timeout instead of its asserted outcome, flaking the Test job while every
local run passed. Widen those budgets to the suite's 5 s idiom so the
asserted state is reached deterministically. Genuine deadline-expiry
tests (permanently-unresolvable adapters, single-shot timeout_ms 0,
deadline-value plumbing, lock-contention) keep their tight budgets - a
slow runner still expires them correctly. Test-only; no production change.
@greptile-apps

greptile-apps Bot commented Jul 20, 2026

Copy link
Copy Markdown

Greptile Summary

This PR widens wall-clock test deadlines from 100–1,000 ms to 5,000 ms across 13 sites in 12 test files to eliminate CI flakes caused by slow GitHub macOS runners. No production code is touched.

  • All 13 changes are mechanical one-for-one numeric swaps in unit test setup; the suite's 5 s idiom is already established and now applied consistently.
  • Intentional expiry tests (permanently-unresolvable adapters, timeout_ms: 0, lock-contention, and deadline-value plumbing) are correctly left untouched, as their outcomes are independent of wall-clock budget.

Confidence Score: 5/5

Safe to merge — purely test-only numeric constant changes with no production impact.

Every change is a one-for-one numeric swap in a test file. The root cause of the flakes is clearly traced: TerminalAfterRetryAdapter::get_element_bounds sleeps 120 ms before returning its stale_ref error, which the old 100 ms deadline would expire before the sleep completed, substituting TIMEOUT for the asserted STALE_REF. Widening to 5,000 ms lets the sleep finish and the correct error propagate. The same pattern applies across all 13 sites. Genuine expiry tests (timeout_ms: 0, permanently-failing adapters) are correctly left untouched since their outcomes are deadline-independent.

No files require special attention.

Important Files Changed

Filename Overview
crates/core/src/commands/pointer_action_tests.rs Two deadline widenings: terminal_stale_ref (100 ms → 5,000 ms) and explicitly_retryable_app_unresponsive (1,000 ms → 5,000 ms). The TerminalAfterRetryAdapter sleeps 120 ms then returns stale_ref, which explains the old 100 ms flake — the sleep outlasted the budget yielding TIMEOUT instead of STALE_REF. Both fixes are correct.
crates/core/src/live_locator/selected_hydration_churn_tests.rs Shared selected_request() helper deadline widened from 500 ms to 5,000 ms; affects all churn tests that call this helper.
crates/core/src/live_locator/selected_hydration_contract_tests.rs Same selected_request() deadline widened from 500 ms to 5,000 ms as in churn_tests.
crates/core/src/live_locator/selected_hydration_subtree_tests.rs request() helper deadline widened from 500 ms to 5,000 ms; identical pattern to the other hydration test files.
crates/core/src/live_locator/resolve_query_hydration_tests.rs Inline LocatorResolveRequest deadline widened from 1,000 ms to 5,000 ms for hydration_retry_preserves_failed_attempt_statistics.
crates/core/src/commands/drag_occlusion_retry_tests.rs timeout_ms widened from 500 ms to 5,000 ms for the transient post-focus occlusion retry test.
crates/core/src/commands/drag_tests.rs timeout_ms widened from 500 ms to 5,000 ms for the positive-timeout stable-bounds drag test.
crates/core/src/commands/hover_preflight_tests.rs timeout_ms widened from 500 ms to 5,000 ms for the headed hover transient-occlusion retry test.
crates/core/src/commands/hover_tests.rs timeout_ms widened from 500 ms to 5,000 ms for the positive-timeout stable-bounds hover test.
crates/core/src/ref_action_exactly_once_tests.rs execute(&adapter, 500) widened to 5,000 ms for ambiguity_diagnostic_preserves_adapter_result_details. Adjacent zero_timeout and moved_semantic_target tests are correctly left unchanged.
crates/core/src/ref_action_wait_unresponsive_tests.rs with_timeout_ms(Some(1_000)) widened to 5,000 ms for the transient unresponsive accessibility service auto-wait test.
crates/core/src/renderer_accessibility_tests.rs Deadline::after(500) widened to 5,000 ms for activation_lease_is_dropped_before_observation_retry.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Test sets deadline
100–1000 ms] -->|Slow CI runner| B{Work completes
before deadline?}
    B -->|No — budget overrun| C[TIMEOUT error
❌ Flake]
    B -->|Yes — local/fast runner| D[Expected error or value
✅ Pass]

    E[Test sets deadline
5000 ms] -->|Slow CI runner| F{Work completes
before deadline?}
    F -->|Yes — deterministic| G[Expected error or value
✅ Pass]

    H[Genuine expiry test
timeout_ms 0 / permanent adapter] --> I{Budget expires?}
    I -->|Always, regardless of speed| J[TIMEOUT / expiry
✅ Unchanged — left alone]
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[Test sets deadline
100–1000 ms] -->|Slow CI runner| B{Work completes
before deadline?}
    B -->|No — budget overrun| C[TIMEOUT error
❌ Flake]
    B -->|Yes — local/fast runner| D[Expected error or value
✅ Pass]

    E[Test sets deadline
5000 ms] -->|Slow CI runner| F{Work completes
before deadline?}
    F -->|Yes — deterministic| G[Expected error or value
✅ Pass]

    H[Genuine expiry test
timeout_ms 0 / permanent adapter] --> I{Budget expires?}
    I -->|Always, regardless of speed| J[TIMEOUT / expiry
✅ Unchanged — left alone]
Loading

Reviews (1): Last reviewed commit: "test: widen wall-clock budgets on retry/..." | Re-trigger Greptile

@lahfir
lahfir merged commit 3f3d69a into main Jul 20, 2026
18 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in Issues and PR Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant