Skip to content

[net11.0] Use scalable Helix job monitor uploader - #37865

Open
kubaflo wants to merge 5 commits into
dotnet:net11.0from
kubaflo:kubaflo/update-net11-helix-monitor
Open

[net11.0] Use scalable Helix job monitor uploader#37865
kubaflo wants to merge 5 commits into
dotnet:net11.0from
kubaflo:kubaflo/update-net11-helix-monitor

Conversation

@kubaflo

@kubaflo kubaflo commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Root cause

PR #37852 correctly isolated the Helix Job Monitor from MAUI's root tool manifest, but its exact-head maui-pr build 1569038 exposed a scalability problem in the branch-pinned monitor version:

  • All 4 Helix jobs and all 36 work items completed successfully.
  • The monitor discovered and processed all four jobs, then waited 5 hours 41 minutes for four pending Azure DevOps result uploads.
  • Azure's Test APIs intermittently returned HTTP 503 or did not respond.
  • The old uploader repeatedly left untagged partial runs: 12 InProgress runs containing 90,650 uploaded result rows remained when the monitor timed out.
  • The monitor was built from 11.0.0-beta.26379.102, which predates Arcade's transient-write retry fix and scalable result-processing rewrite.

Fix

Pin the dedicated monitor job to Microsoft.DotNet.Helix.JobMonitor 11.0.0-beta.26425.2, independently from the Helix submission SDK.

That package includes:

  • dotnet/arcade#17256, which retries transient Azure DevOps transport failures and HTTP 5xx/429 result writes.
  • dotnet/arcade#17331, which replaces the blocking upload queue with bounded parallel result processing, coordinated throttling, streaming batches, and idempotent test-run completion retries.

Arcade intentionally excludes upload failures from the monitor's exit code so an interrupted, untagged upload can be replayed by a later invocation. MAUI has only one monitor invocation, so the pipeline now re-invokes the pinned monitor when its final drain reports untagged jobs. Completed tagged uploads are skipped, incomplete jobs are replayed within the original shared timeout, and any real test failure is preserved until result publication is durable. Missing or changed durable-state output fails closed.

The Helix submission SDK remains unchanged at the branch-pinned version, so this only updates centralized result processing. The monitor package targets net10.0 with rollForward: Major, and runs under MAUI's pinned net11 runtime.

No open MAUI PR addresses this follow-up. Increasing the six-hour timeout would only prolong the old uploader's blocked drain, while suppressing result publication would hide real test failures.

Validation

  • Installed 11.0.0-beta.26425.2 from an empty NuGet cache.
  • Invoked the exact tool DLL under MAUI's pinned net11 runtime and confirmed source commit b49c2eb57bbccbbd4a16e535eb1edc2a9604d723.
  • Verified the existing monitor CLI arguments remain supported, including --max-wait-minutes and --fail-on-failed-tests.
  • Verified the package source commit contains both Arcade fixes.
  • In follow-up build 1569648, all 4 Helix jobs and 36 work items passed while the Test API produced 92 failed request attempts. The scalable uploader processed 12,858 results and drained in 57 minutes instead of hanging for six hours.
  • That stress run also proved the upstream tool can exit successfully with incomplete publication: 3 job uploads remained untagged and 3 test runs remained InProgress. The new replay guard classifies that exact raw summary as incomplete rather than green.
  • Exercised replay-to-success, preserved-test-failure, missing-summary, and nonzero-exit paths against the pinned output contract.
  • Parsed the updated Azure Pipelines template and checked the patch for whitespace errors.

Pin the post-Arcade dotnet#17331 monitor independently from the Helix submission SDK so result uploads use bounded parallelism, coordinated throttling, and resilient finalization.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings August 27, 2026 01:17
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool August 27, 2026 01:17 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 37865

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 37865"

@kubaflo
kubaflo temporarily deployed to copilot-pat-pool August 27, 2026 01:17 — with GitHub Actions Inactive
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

@kubaflo
kubaflo temporarily deployed to copilot-pat-pool August 27, 2026 01:18 — with GitHub Actions Inactive
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool August 27, 2026 01:20 — with GitHub Actions Inactive
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool August 27, 2026 01:21 — with GitHub Actions Inactive
@github-actions github-actions Bot added the area-infrastructure CI, Maestro / Coherency, upstream dependencies/versions label Aug 27, 2026
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool August 27, 2026 01:21 — with GitHub Actions Inactive

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the MAUI Helix test stage template to pin the Helix Job Monitor package version independently from the Helix submission SDK, addressing scalability/reliability issues in Azure DevOps test-result uploads.

Changes:

  • Add a new helixJobMonitorVersion template parameter (defaulting to 11.0.0-beta.26425.2).
  • Stop deriving the monitor version from eng/Versions.props; use the new parameter instead when installing Microsoft.DotNet.Helix.JobMonitor.
  • Adjust the associated validation/error output for invalid monitor version values.

Comment thread eng/pipelines/arcade/stage-helix-tests.yml Outdated
Report rejected values with Bash escaping so pipeline overrides are diagnosable without allowing raw control text into Azure logs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 27, 2026 01:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comment thread eng/pipelines/arcade/stage-helix-tests.yml Outdated
Bind the Azure template parameter through the step environment so quotes and shell metacharacters remain data rather than template-expanded shell syntax.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 27, 2026 01:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comment thread eng/pipelines/arcade/stage-helix-tests.yml Outdated
Use a default expansion under set -u so an absent environment binding reaches the explicit package-version validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 27, 2026 02:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

The rewritten Job Monitor intentionally reports upload failures as warnings and exits successfully. Re-run untagged uploads within the shared timeout, fail closed when durable state is missing, and preserve real test failures until every upload is complete.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 27, 2026 03:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

@kubaflo

kubaflo commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 3 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-infrastructure CI, Maestro / Coherency, upstream dependencies/versions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants