[CI] Flaky: monotouch-test macOS NativeAOT configurations TimedOut

[rolfbjarne]

Flaky Test Report (automated)

Test: monotouch-test (macOS, NativeAOT configurations)
Platform: macOS
Category: Flaky
Period: June 8–15, 2026

Occurrence Summary

PR	Build	Configuration	Bot	Direct Link
#25657	14376282	NativeAOT	VSCXSDKs-MINI-043.Tahoe.arm64	monotouch_macos → Run tests
#25586	14376290	trimmable static registrar, NativeAOT	VSM-XAM-08.Tahoe.arm64	monotouch_macos → Run tests

Total: Failed in 2 builds across 2 unrelated PRs this week.

Error Details

The monotouch-test suite on macOS completes all NativeAOT configurations successfully except one which times out (TimedOut status in xharness). The specific configuration that times out varies between runs:

Build 14376282 (PR #25657):

NativeAOT → TimedOut

Build 14376290 (PR #25586):

trimmable static registrar, NativeAOT → TimedOut

All other NativeAOT configurations in both builds pass (including NativeAOT ARM64, NativeAOT x64, .NET 11 defaults, trimmable static registrar NativeAOT ARM64/x64). This suggests a timing/resource issue rather than a consistent test failure.

Classification

This failure was identified as flaky because:

• It appeared across 2 unrelated PRs
• Different NativeAOT configurations time out each time (not always the same config)
• All other NativeAOT configs pass in the same run
• The behavior is consistent with resource exhaustion or slow NativeAOT compilation on macOS hitting a timeout threshold

Possible Root Cause

NativeAOT compilation on macOS for the full monotouch-test suite is resource-intensive. Some configurations may occasionally exceed the test timeout when running on heavily loaded bots or when NativeAOT compilation is slightly slower.

---

This issue was automatically generated by CI post-mortem analysis.

[rolfbjarne]

CI Post-Mortem Update (June 15–21, 2026)

This timeout was seen again this week:

PR	Build	Configuration	Bot	Direct Link
#25717	14427640	Debug (trimmable static registrar)	VSCXSDKs-MINI-006.Tahoe.arm64	monotouch_macos → Run tests
#25717	14427640	Release (trimmable static registrar)	VSCXSDKs-MINI-006.Tahoe.arm64	monotouch_macos → Run tests

Total: 2 config failures (same build) in 1 PR this week.

Error

TimedOut (Execution timed out after 1200 seconds. Test run crashed)

Both Debug and Release "trimmable static registrar" configurations timed out on the same bot in the same run.

Trend

• Last week (Jun 8–15): 2 builds, 2 PRs
• This week (Jun 15–21): 1 build, 1 PR

Low volume but still occurring — the pattern is consistent with the NativeAOT/trimmable-static-registrar configs taking too long on macOS.

---

This comment was automatically generated by CI post-mortem analysis.

[rolfbjarne]

CI Post-Mortem Update (June 22–29, 2026)

This failure pattern was seen again this week, along with related macOS/MacCatalyst test crashes:

NativeAOT Crash (matches this issue directly)

PR	Build	Configuration	Bot	Direct Link
#25815	14514853	monotouch-test/macOS/Release (trimmable static registrar, NativeAOT)	AcesShared 32	Run tests

Error: Test run crashed (exit code: 133)

Possibly related: macOS monotouch-test killed (exit code 137)

PR	Build	Configuration	Bot	Direct Link
#25848	14515016	macOS/osx-x64 monotouch-test	VSM-XAM-31.Monterey.arm64	Run tests

Error: Process killed with exit code 137 (SIGKILL — likely OOM). Last output was running MonoTouchFixtures.ObjCRuntime.RegistrarTest.

Possibly related: MacCatalyst "Failed to launch after maximum attempts"

PR	Build	Configuration	Bot	Direct Link
#25815	14514853	MacCatalyst/maccatalyst-x64 monotouch-test	VSM-XAM-20.Tahoe.arm64	Run tests
#25815	14514853	MacCatalyst/maccatalyst-x64 monotouch-test	VSM-XAM-55.Sequoia.arm64	Run tests

Error: Failed to launch after maximum attempts — the MacCatalyst app could not start on either Tahoe or Sequoia.

---

Summary: 1 direct NativeAOT crash occurrence + 2 related macOS/MacCatalyst test execution failures this week across 2 PRs.

[rolfbjarne]

CI Post-Mortem Update (July 8–15, 2026)

The MacCatalyst NativeAOT timeout recurred this week in PR #25919 ([DO NOT MERGE] Instrument __MonoMac_NSAsyncActionDispatcher for issue #25861):

PR	Build	Configuration	Date
#25919	14662367	Mac Catalyst/Release (NativeAOT), (NativeAOT, .NET 11 defaults)	2026-07-15
#25919	14663555	Mac Catalyst/Release (NativeAOT), (NativeAOT, .NET 11 defaults), (trimmable static registrar, NativeAOT)	2026-07-15

All timed out with Execution timed out after 1200 seconds.

Note: These builds are from an instrumentation PR that adds debugging overhead, which may exacerbate the timeout. The timeout is still consistent with this issue's pattern.

[rolfbjarne]

CI Post-Mortem Update (July 13–20, 2026)

This failure was seen again this week:

PR	Build	Date	Bot	Details
#25919	14702838	Jul 20	XAMBOT-1006.Sonoma	macOS osx-x64 tests exited with code 137

Total: 1 occurrence this week (macOS x64 exit 137 on Sonoma bot).

[dalexsoto]

New occurrence on #26247 @ 034423cafa4c5b3bf5e09b6c8fcc645b7de31cf7, build 14734681 attempt 2 — M1 Mac Ventura macOS tests: macOS/osx-arm64 monotouch-test timed out after 600 seconds and was killed with exit code 137 on VSM-XAM-70.Ventura.arm64. The output ended after passing the NSUrlSessionHandler cookie tests; no deterministic assertion failed.

This update adds 1 occurrence across 1 newly affected monitored PR. mac_13_m1 attempt 3 was accepted and is in progress. Reconciled PR budget: 3 lifetime / 3 current-head requests.

[dalexsoto]

Final retry outcome for #26247 @ 034423cafa4c5b3bf5e09b6c8fcc645b7de31cf7: build 14734681 mac_13_m1 attempt 3 completed successfully. The Ventura macOS monotouch tests passed, the overall build succeeded, and the PR merged at 23:56 UTC.

[dalexsoto]

New recurrence on #26278 @ 303f04f2113e74a8a7328fbcc1dc85bbd02fc467: build 14753655 attempt 1 - arm64 Mac Tahoe macOS tests had macOS/osx-arm64 monotouch-test time out after 600 seconds and exit 137 on VSM-XAM-15.Tahoe.arm64. The output ended with repeated CFSocketSetAddress listen failure: 102; no deterministic assertion failed.

This exact arm64 timeout/exit-137 signature now covers 2 occurrences across 2 monitored PRs. It is classified as genuinely flaky because the prior #26247 occurrence recovered when its targeted macOS stage retry passed.

Build 14753655 remains active, so no retry has been submitted yet.

[dalexsoto]

Retry update for #26278 @ 303f04f2113e74a8a7328fbcc1dc85bbd02fc467: build 14753655 simulator_tests attempt 2 was accepted using failed-jobs semantics. It is rerunning the macOS timeout-affected monotouch_macos work together with linker_ios and the jobs canceled by attempt 1 (generator and monotouch_maccatalyst). Retry budget usage: 1 lifetime / 1 current head.

[dalexsoto]

New monitored occurrence on #26278 @ 303f04f2113e74a8a7328fbcc1dc85bbd02fc467: build 14755483 attempt 1 — monotouch_macos / Release (trimmable static registrar, all optimizations) on AcesShared 82.

The process ran for 1200 seconds, was killed with exit 133, and produced no crash report. Its output stopped in MonoTouchFixtures.Contacts.ContactFetchRequestTest after repeated CFSocketSetAddress listen failure: 102; no deterministic assertion failed.

This brings the current monitored signature to 3 occurrences across 2 PRs. Build 14755483 is an automatically triggered PR build and remains active, so no retry has been submitted for this occurrence.