CI: non-blocking Artery MNTR stage - #8373
Conversation
This adds a second Windows multi-node test job, net_mntr_windows_artery, that runs alongside the existing classic MNTR job with the exact same incrementalist scoping, but sets AKKA_MNTR_TRANSPORT=artery so the suite runs over Artery instead of classic DotNetty. The goal is to start tracking Artery's own MNTR green rate as a ship-confidence signal, without putting it in the way of PRs. The job is marked continueOnError:true at the job level (a new parameter on azure-pipeline.mntr-template.yaml, off by default so the classic MNTR job is unaffected), so it still runs and reports its own pass/fail per PR, but can never fail PR validation as a whole. Artery has no failure-injection test-mode yet, so specs that call TestConductor Throttle/Blackhole/PassThrough will fail fast on the RequireTestConductorTransport check instead of exercising real behavior. There's no clean way to exclude just those ~29 specs today (incrementalist filters by project, not by test class), so that failure is accepted as known noise for now; the affected specs are listed in a comment in pr-validation.yaml for when Artery gets a test-mode transport.
|
First-run checklist for this PR's own validation run:
|
MNTR has never had a Linux lane. This adds net_mntr_linux on ubuntu-latest with the same incrementalist scoping and command as the classic Windows MNTR job, non-blocking initially with the same promote-after-sustained-green path as the Artery lane. The only template fix needed was the Copy Build Output match pattern, which used a Windows-style glob; it now matches the generic template that already runs on both platforms. Also reworded the Artery comment to point at a dedicated incrementalist config as the sanctioned lever if that lane ever needs a different project set.
Adds net_mntr_linux_artery, completing the classic/Artery x Windows/Linux matrix with only classic Windows blocking. Also replaces two TBD doc comments on RemoteSettings.TransportNames and RemoteSettings.Transports with real descriptions of the classic transport-selection settings and how they relate to Artery's separate enable switch. Touching Akka.Remote also serves to trigger incrementalist selection of the remoting/cluster multi-node projects, so this PR's validation run exercises all four MNTR lanes with real specs.
|
Four-lane first-run review (build 128977):
Verdict: the matrix wiring is doing its job, but the Artery lane has real triage signal beyond the known ~30-spec noise floor. Do not promote it. After #8378 lands, remove the obsolete failure-injection allowance and compare the residual failure set directly against this run. |
|
Four-lane results from build 129105 — the first run with Artery test-mode active (post-#8378):
The two Artery lanes share 17 failing spec classes across both platforms — deterministic findings, not environment noise. The dominant pattern is the restarted-node family (same address, new UID): all five Remote quarantine/gate-piercing specs ( Also validated mechanically by this run: Linux unit tests failed on one spec, |
|
Rerun (build 129110, after the comment-fix push): PR Validation is green — both prior blocking failures were one-off flakes (StressSpec on Windows MNTR, an Akka.Streams TcpSpec timing case on Linux unit tests) and did not recur. The continueOnError design works end-to-end: both Artery lanes report their own red check while the umbrella passes, so this PR is mergeable with the Artery green-rate tracked visibly per PR. The Artery failure set is stable across both runs and platforms — the restart/quarantine-piercing core (5 piercing specs, 4 cluster restart specs, 4 DurableData specs, DistributedPubSubRestartSpec, UnreachableNodeJoinsAgainSpec) and the router family reproduced identically; only a small tail rotated (PersistentClusterSharding variants, ClusterSingletonManagerLeaveSpec, LeaderElection puppet — flake-class over Artery). The deterministic core is tracked as the restarted-peer/new-incarnation association gap. |
|
Force merging this so we can start working on turning the failing MNTR specs green - we know they have issues, by design. |
1 participant
Adds a second Windows multi-node job that runs the MNTR suite over the Artery transport (
AKKA_MNTR_TRANSPORT=artery, the #8351 switch), parallel to the classic job with identical incrementalist scoping. The job iscontinueOnError: trueso it never gates PR validation — it exists to establish an independent Artery green-rate before ever becoming required.About 30 specs that use TestConductor throttle/blackhole failure injection fail fast on Artery today (
RequireTestConductorTransportthrows before any network activity) — they're listed in a comment in the pipeline YAML and are expected red in this job until the Artery test-mode port lands, absorbed by the non-blocking marking. The MNTR job template gains two optional parameters (continueOnError,env) whose defaults leave the classic job byte-identical.This PR's own validation run doubles as the first live test of the new job — the things to confirm on it are in the comment below.