Skip to content

Queue 105: make ExecutionMonitor tests deterministic - #287

Open
jbrodovsky wants to merge 1 commit into
mainfrom
v1/p-exec-monitor-clock
Open

Queue 105: make ExecutionMonitor tests deterministic#287
jbrodovsky wants to merge 1 commit into
mainfrom
v1/p-exec-monitor-clock

Conversation

@jbrodovsky

Copy link
Copy Markdown
Owner
Branch v1/p-exec-monitor-clock
Base main
Issues #284
Queue position 105 (parallel)

Why

test_execution_monitor_successful_execution_with_progress slept 10 ms per iteration and then
asserted a 50 ms no-progress budget had not been exceeded. thread::sleep guarantees a minimum
duration, never a maximum, so that assertion is not something the standard library promises. It went
red on GitHub's hosted runners.

It is not caused by any code change: a docs-only branch off main (#279) reproduces it with
every .rs file byte-identical. It was also the only remaining failure on #285 after the ESKF
fixes landed, so it is currently the last thing between this repo and green CI.

What changed

The timeout logic is already a pure function of the instants it is handed; only the source of those
instants was hard-wired to Instant::now(). Added new_at / check_at / mark_progress_at taking
an explicit Instant, with the existing methods delegating. They are pub(crate)the public API
is unchanged
.

Five tests become nine, all deterministic, none sleeping:

  • no-progress and wall-clock timeouts still fire
  • boundary behaviour at exactly the limit and one millisecond past it — untestable with sleeps, and previously uncovered
  • mark_progress restarts the no-progress window
  • mark_progress does not extend the absolute wall-clock budget
  • steady work with progress marks survives 200 iterations across 8 s of simulated time, well past the 50 ms budget — the property the flaky test was reaching for, now actually exercised
  • max_wall_clock_ratio and max_wall_clock_s combine so the tighter wins, in both orders

Verification

  • 240 unit + 12 integration + 63 doc tests pass; clippy and fmt clean
  • This group: ~140 ms of sleeping → 0.00 s
  • 10 consecutive runs under 2x CPU oversubscription: 9/9 passing every time
  • No thread::sleep remains anywhere in core/src/sim.rs

Closes #284

Branches off main; independent of the spine.

Queue: docs/V1_QUEUE.md | Board: https://github.com/users/jbrodovsky/projects/7

…:sleep

test_execution_monitor_successful_execution_with_progress slept 10 ms per
iteration and then asserted a 50 ms no-progress budget had not been exceeded.
thread::sleep guarantees a minimum duration, never a maximum, so that assertion
is not something the standard library promises -- on a contended runner the
sleep overruns and the test fails. It went red on GitHub's hosted runners, and a
docs-only branch off main reproduced it with every .rs file byte-identical,
which is what confirmed it had nothing to do with the code under test.

The timeout logic is already a pure function of the instants it is handed; only
the source of those instants was hard-wired. Add new_at / check_at /
mark_progress_at taking an explicit Instant, with new / check / mark_progress
delegating to them with Instant::now(). They are pub(crate), so the public API
is unchanged.

The five tests become nine, all deterministic and none sleeping:

  - no-progress and wall-clock timeouts still fire
  - boundary behaviour at exactly the limit and one millisecond past it, in both
    directions -- untestable with sleeps, and previously uncovered
  - mark_progress restarts the no-progress window
  - mark_progress does NOT extend the absolute wall-clock budget
  - steady work with regular progress marks survives 200 iterations over 8 s of
    simulated time, well past the 50 ms no-progress budget (the property the
    flaky test was reaching for, now actually exercised)
  - max_wall_clock_ratio and max_wall_clock_s combine so the tighter wins, in
    both orders

Suite time for this group goes from ~140 ms of sleeping to 0.00 s, and 10 runs
under 2x CPU oversubscription pass identically.

Closes #284

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 4, 2026 20:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

ExecutionMonitor::check_at can panic on non-monotonic instants due to duration_since, which should be guarded now that the clock is injectable.

Pull request overview

This PR makes ExecutionMonitor’s timeout logic testable without relying on real wall-clock sleeps by introducing pub(crate) entrypoints that accept an explicit Instant, enabling deterministic, boundary-precise unit tests while keeping the public API unchanged.

Changes:

  • Add ExecutionMonitor::{new_at, check_at, mark_progress_at} with existing new/check/mark_progress delegating to Instant::now().
  • Replace sleep-based ExecutionMonitor unit tests with deterministic tests that drive time via fixed instants, adding boundary coverage.
File summaries
File Description
core/src/sim.rs Adds injectable-clock variants of ExecutionMonitor methods and rewrites/expands unit tests to be deterministic and cover boundaries.
Review details

Suppressed comments (1)

core/src/sim.rs:2747

  • check_at now accepts an arbitrary Instant, but it still uses Instant::duration_since, which panics if now is earlier than start_time or last_progress. Since this is a new injectable-clock API (even if pub(crate)), it’s safer to avoid introducing a panic footgun; use checked_duration_since and return an error on non-monotonic instants.
            if let Some(max_wall_clock) = self.max_wall_clock
                && now.duration_since(self.start_time) > max_wall_clock
            {
                bail!(
                    "Execution timeout ({context}): exceeded wall-clock limit of {:.2} s",
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@jbrodovsky

Copy link
Copy Markdown
Owner Author

CI: this PR and #285 fix exactly what the other still fails on

Neither PR is green alone, because each is missing the other's fix. Together they are.

This PR (#287), branched off main, fixes test_execution_monitor_successful_execution_with_progress
— it no longer appears in any failure. What remains failing here is the ESKF pair:

  • test_eskf_stability_high_dynamics
  • test_filter_comparison

Both are fixed in #285 (golden thresholds of 1905.00 m replaced with physical bounds).

#285, conversely, has every ESKF and RBPF test passing and fails on exactly one thing:
sim::tests::test_execution_monitor_successful_execution_with_progress — the test this PR fixes.

So the merge order does not matter; whichever lands second turns CI green.

One unrelated failure, newly surfaced

macOS also shows:

test_ekf_with_degraded_gnss: RMS altitude error should be less than 140.00m, got 185.66m

This PR cannot reach the EKF. 140.00 is another threshold fitted to an observed value rather than a
physical bound — the same pattern as the 1905.00 and 509.00 numbers #266 was about, and the third
instance found so far. Filed separately so it does not block either PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v1.0] Flaky: ExecutionMonitor tests race the wall clock on loaded CI runners

2 participants