[CI] Collect diagnostic info on LaunchTimedOut simulator failures. Ref #25299 by rolfbjarne · Pull Request #25300 · dotnet/macios

rolfbjarne · 2026-05-04T07:40:59Z

When simulator tests fail with LaunchTimedOut, collect diagnostic info:

Simulator list (simctl list) to check boot state
Simulator boot status for the target UDID
Installed apps on the simulator

Also run simctl diagnose in the pipeline on test failure (not only
when system.debug is true).

…#25299 When simulator tests fail with LaunchTimedOut, collect diagnostic info: - Simulator list (simctl list) to check boot state - Simulator boot status for the target UDID - Installed apps on the simulator Also run simctl diagnose in the pipeline on test failure (not only when system.debug is true). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

rolfbjarne · 2026-05-04T07:41:56Z

CC #25299

Copilot

Pull request overview

Updates CI and xharness to collect additional simulator diagnostics when tests fail due to simulator launch timeouts, improving debuggability of intermittent LaunchTimedOut failures in the Apple-platform test infrastructure.

Changes:

Run xcrun simctl diagnose in the test pipeline on job failure (not only when system.debug is enabled) and publish the resulting artifact.
Add xharness-side diagnostic collection for LaunchTimedOut results (simctl list, bootstatus, listapps) into a dedicated log.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
tools/devops/automation/templates/tests/run-tests.yml	Adjusts step conditions so simulator diagnostics are collected/published on failures as well as debug runs.
tests/xharness/AppRunner.cs	Adds targeted `LaunchTimedOut` diagnostic logging via `simctl` commands.

Log the LaunchTimedOut condition to the Jenkins main log so it's visible in the pipeline output (previously it was only in a per-test Run log file). Also collect load average and recent crash reports in the diagnostic log, since we found sqlite3 crashes correlated with LaunchTimedOut failures. Ref #25299 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

When a simulator test launch times out, retry once with a fresh AppRunner. This addresses the transient failures caused by high system load and sqlite3 crashes on CI machines that prevent simulators from properly launching test apps. Also propagate AppRunner.FailureMessage to the test task so failure details are visible in the test report. Ref #25299 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

vs-mobiletools-engineering-service2 · 2026-05-07T06:57:19Z

✅ [PR Build #23a7934] Build passed (Build packages) ✅

Pipeline on Agent
Hash: 23a7934e38792b5181a671fbf2d68ed66e3d106a [PR build]

vs-mobiletools-engineering-service2 · 2026-05-07T06:58:05Z

✅ [PR Build #23a7934] Build passed (Detect API changes) ✅

Pipeline on Agent
Hash: 23a7934e38792b5181a671fbf2d68ed66e3d106a [PR build]

vs-mobiletools-engineering-service2 · 2026-05-07T07:13:42Z

✅ API diff for current PR / commit

NET (empty diffs)

iOS: vsdrops gist ( No breaking changes )
tvOS: vsdrops gist ( No breaking changes )
MacCatalyst: vsdrops gist ( No breaking changes )
macOS: vsdrops gist ( No breaking changes )

✅ API diff vs stable

NET (empty diffs)

iOS: vsdrops gist ( No breaking changes )
tvOS: vsdrops gist ( No breaking changes )
MacCatalyst: vsdrops gist ( No breaking changes )
macOS: vsdrops gist ( No breaking changes )

ℹ️ Generator diff

Generator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes)

Pipeline on Agent
Hash: 23a7934e38792b5181a671fbf2d68ed66e3d106a [PR build]

vs-mobiletools-engineering-service2 · 2026-05-07T07:18:07Z

✅ [PR Build #23a7934] Build passed (Build macOS tests) ✅

Pipeline on Agent
Hash: 23a7934e38792b5181a671fbf2d68ed66e3d106a [PR build]

vs-mobiletools-engineering-service2 · 2026-05-07T09:18:41Z

🔥 [CI Build #23a7934] Test results 🔥

Test results

❌ Tests failed on VSTS: test results

1 tests crashed, 2 tests failed, 164 tests passed.

Failures

❌ fsharp tests

2 tests failed, 2 tests passed.

Failed tests

fsharp/iOS - simulator/Debug: HarnessException (Harness exception for 'fsharp': System.NullReferenceException: Object reference not set to an instance of an object.
at Xharness.Jenkins.TestTasks.RunSimulator.RunTestAsync() in /Users/builder/azdo/_work/2/s/macios/tests/xharness/Jenkins/TestTasks/RunSimulator.cs:line 135
at Xharness.Jenkins.TestTasks.RunTest.ExecuteAsync() in /Users/builder/azdo/_work/2/s/macios/tests/xharness/Jenkins/TestTasks/RunTest.cs:line 113
at Xharness.Jenkins.TestTasks.TestTask.RunInternalAsync() in /Users/builder/azdo/_work/2/s/macios/tests/xharness/Jenkins/TestTasks/TestTask.cs:line 234)
fsharp/tvOS - simulator/Debug: HarnessException (Harness exception for 'fsharp': System.NullReferenceException: Object reference not set to an instance of an object.
at Xharness.Jenkins.TestTasks.RunSimulator.RunTestAsync() in /Users/builder/azdo/_work/2/s/macios/tests/xharness/Jenkins/TestTasks/RunSimulator.cs:line 135
at Xharness.Jenkins.TestTasks.RunTest.ExecuteAsync() in /Users/builder/azdo/_work/2/s/macios/tests/xharness/Jenkins/TestTasks/RunTest.cs:line 113
at Xharness.Jenkins.TestTasks.TestTask.RunInternalAsync() in /Users/builder/azdo/_work/2/s/macios/tests/xharness/Jenkins/TestTasks/TestTask.cs:line 234)

Html Report (VSDrops) Download

❌ windows tests

🔥 Failed catastrophically on VSTS: test results - windows (no summary found).

Html Report (VSDrops) Download

Successes

✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (iOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (MacCatalyst): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (macOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (tvOS): All 1 tests passed. Html Report (VSDrops) Download
✅ framework: All 2 tests passed. Html Report (VSDrops) Download
✅ generator: All 5 tests passed. Html Report (VSDrops) Download
✅ interdependent-binding-projects: All 4 tests passed. Html Report (VSDrops) Download
✅ introspection: All 6 tests passed. Html Report (VSDrops) Download
✅ linker: All 44 tests passed. Html Report (VSDrops) Download
✅ monotouch (iOS): All 13 tests passed. Html Report (VSDrops) Download
✅ monotouch (MacCatalyst): All 18 tests passed. Html Report (VSDrops) Download
✅ monotouch (macOS): All 18 tests passed. Html Report (VSDrops) Download
✅ monotouch (tvOS): All 13 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ sharpie: All 1 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 4 tests passed. Html Report (VSDrops) Download
✅ xtro: All 1 tests passed. Html Report (VSDrops) Download

macOS tests

✅ Tests on macOS Monterey (12): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Ventura (13): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sonoma (14): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sequoia (15): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Tahoe (26): All 5 tests passed. Html Report (VSDrops) Download

Linux Build Verification

✅ Linux build succeeded

Pipeline on Agent
Hash: 23a7934e38792b5181a671fbf2d68ed66e3d106a [PR build]

rolfbjarne requested a review from Copilot May 4, 2026 07:41

rolfbjarne added the copilot label May 4, 2026

Copilot started reviewing on behalf of rolfbjarne May 4, 2026 07:41 View session

Copilot AI reviewed May 4, 2026

View reviewed changes

Comment thread tests/xharness/AppRunner.cs

This comment has been minimized.

Sign in to view

Merge branch 'main' into dev/rolf/ci-launchtimedout

a909dea

This comment has been minimized.

Sign in to view

rolfbjarne and others added 2 commits May 6, 2026 14:26

This comment has been minimized.

Sign in to view

Build me

23a7934

This comment has been minimized.

Sign in to view

Conversation

rolfbjarne commented May 4, 2026

Uh oh!

rolfbjarne commented May 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

vs-mobiletools-engineering-service2 commented May 7, 2026

✅ [PR Build #23a7934] Build passed (Build packages) ✅

Uh oh!

vs-mobiletools-engineering-service2 commented May 7, 2026

✅ [PR Build #23a7934] Build passed (Detect API changes) ✅

Uh oh!

vs-mobiletools-engineering-service2 commented May 7, 2026

✅ API diff for current PR / commit

✅ API diff vs stable

ℹ️ Generator diff

Uh oh!

vs-mobiletools-engineering-service2 commented May 7, 2026

✅ [PR Build #23a7934] Build passed (Build macOS tests) ✅

Uh oh!

vs-mobiletools-engineering-service2 commented May 7, 2026

🔥 [CI Build #23a7934] Test results 🔥

Test results

Failures

❌ fsharp tests

Failed tests

❌ windows tests

Successes

macOS tests

Linux Build Verification

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants