[net11.0] Use Helix job monitor for unit tests (#37852)

<!-- Please let the below note in for people that find this PR -->
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could [test the resulting
artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from
this PR and let us know in a comment if this change resolves your issue.
Thank you!

## Root cause

`maui-pr` builds
[1568023](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1568023)
and
[1568717](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1568717)
failed their Windows Helix lanes even though the affected xUnit commands
completed with exit code 0. The latest run dead-lettered nine work items
after the legacy per-work-item Azure Pipelines reporter hit `TF10216`
service-unavailable responses and 100-second read timeouts.

## Fix

- Opt the unit-test submission jobs into `EnableHelixJobMonitor`, which
disables the legacy per-work-item reporter and hands completion/result
publication to one monitor job.
- Bootstrap the pinned SDK and install the matching
`Microsoft.DotNet.Helix.JobMonitor` version only inside the dedicated
Linux monitor job. This keeps the net11-targeted tool out of the
repository-wide tool manifest, whose restore runs before the pinned SDK
is available.
- Invoke the Arcade bootstrap script through `bash` because MAUI's
generated copy is not executable.
- Give the monitor a 360-minute timeout so it outlives the 240-minute
submission jobs.

The monitor runs alongside the submitters in the same stage and gates on
both the stage timeline and every discovered Helix job. Real test
failures still fail the centralized monitor; only the unreliable legacy
reporting path is removed.

No open `net11.0` PR currently addresses the Helix reporter failure.
This uses the job-monitor path already shipped by the branch-pinned
Arcade SDK instead of suppressing test failures or dropping Azure DevOps
test results.

## Validation

- Restored the unchanged root tool manifest from an empty NuGet cache.
- Installed and invoked `dotnet-helix-job-monitor` version
`11.0.0-beta.26379.102` from an isolated empty cache.
- Verified the normal path remains `monitor=false`, `reporter=true`,
`wait=true`.
- Verified the monitored stage path evaluates to `monitor=true`,
`reporter=false`, `wait=false`.
- Parsed the updated Azure Pipelines YAML and checked the patch for
whitespace errors.

The first PR run,
[1568855](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1568855),
exposed two integration issues in the initial implementation: adding the
net11 tool to the root manifest broke pre-bootstrap restores, and the
generated monitor template directly executed a non-executable script.
Commit `d1ca1493b8` corrects both by isolating installation and invoking
the script through `bash`.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: kubaflo

eng/pipelines/arcade/stage-helix-tests.yml

@@ -47,6 +47,9 @@ parameters:
   default:
   - Debug
   - Release
+- name: helixJobMonitorTimeoutInMinutes
+  type: number
+  default: 360
 - name: helixPool
   type: object
 
@@ -91,9 +94,65 @@ stages:
               DOTNET_TOKEN: $(dotnetbuilds-internal-container-read-token)
               PRIVATE_BUILD: $(PrivateBuild)
 
-          - script: $(_msbuildCommand) "${{ parameters.helixProject }}" -warnAsError 0 -restore /p:Configuration=${{ BuildConfiguration }} /p:HelixInternal="${{ parameters.helixInternal }}" /p:TestRunNameSuffix="_${{ BuildConfiguration }}" /p:SkipInitInternalTooling=true /bl:$(Build.Arcade.LogsPath)${{ BuildConfiguration }}/helix_tests.binlog ${{ parameters.extraHelixArguments }}
+          - script: $(_msbuildCommand) "${{ parameters.helixProject }}" -warnAsError 0 -restore /p:Configuration=${{ BuildConfiguration }} /p:EnableHelixJobMonitor=true /p:HelixInternal="${{ parameters.helixInternal }}" /p:TestRunNameSuffix="_${{ BuildConfiguration }}" /p:SkipInitInternalTooling=true /bl:$(Build.Arcade.LogsPath)${{ BuildConfiguration }}/helix_tests.binlog ${{ parameters.extraHelixArguments }}
             displayName: Run Helix Tests
             env:
-              SYSTEM_ACCESSTOKEN: $(System.AccessToken) # We need to set this env var to publish helix results to Azure Dev Ops
+              SYSTEM_ACCESSTOKEN: $(System.AccessToken)
               HelixAccessToken: ${{ parameters.HelixAccessToken }}
-    
+
+  - job: HelixJobMonitor
+    displayName: Monitor Helix Jobs
+    timeoutInMinutes: ${{ parameters.helixJobMonitorTimeoutInMinutes }}
+    pool:
+      ${{ if eq(variables['System.TeamProject'], 'public') }}:
+        name: $(DncEngPublicBuildPool)
+        demands: ImageOverride -equals build.azurelinux.3.amd64.open
+      ${{ else }}:
+        name: $(DncEngInternalBuildPool)
+        demands: ImageOverride -equals build.azurelinux.3.amd64
+    steps:
+    - checkout: self
+      fetchDepth: 1
+
+    - bash: |
+        set -euo pipefail
+
+        toolPath="$AGENT_TEMPDIRECTORY/helix-job-monitor"
+        bash ./eng/common/dotnet.sh
+        toolVersion=$(bash ./eng/common/dotnet.sh msbuild eng/Versions.props -getProperty:MicrosoftDotNetHelixSdkPackageVersion -nologo | tail -n 1)
+        if [[ -z "$toolVersion" || "$toolVersion" =~ [[:space:]] ]]; then
+          echo "Could not read the Helix SDK package version from eng/Versions.props." >&2
+          exit 1
+        fi
+
+        bash ./eng/common/dotnet.sh tool install \
+          --tool-path "$toolPath" \
+          Microsoft.DotNet.Helix.JobMonitor \
+          --version "$toolVersion"
+
+        toolDll=$(find "$toolPath/.store" -path '*/tools/*/any/Microsoft.DotNet.Helix.JobMonitor.dll' -type f -print -quit)
+        if [ ! -f "$toolDll" ]; then
+          echo "Could not find the Helix Job Monitor DLL in '$toolPath/.store'." >&2
+          exit 1
+        fi
+
+        echo "##vso[task.setvariable variable=HelixJobMonitorDll]$toolDll"
+      displayName: Install Helix Job Monitor
+
+    - bash: |
+        set -euo pipefail
+
+        bash ./eng/common/dotnet.sh exec "$(HelixJobMonitorDll)" \
+          --helix-base-uri 'https://helix.dot.net/' \
+          --polling-interval-seconds 30 \
+          --fail-on-failed-tests true \
+          --max-wait-minutes "$((${{ parameters.helixJobMonitorTimeoutInMinutes }} - 5))" \
+          --stage-name '$(System.StageName)' \
+          --organization dotnet \
+          --repository maui \
+          --build-reason '$(Build.Reason)' \
+          --source-branch '$(Build.SourceBranch)'
+      displayName: Monitor Helix Jobs
+      env:
+        SYSTEM_ACCESSTOKEN: $(System.AccessToken)
+        HELIX_ACCESSTOKEN: ${{ parameters.HelixAccessToken }}