Merge branch 'dev' into feature/8293-consistenthash-upstream

Author: Aaronontheweb

.config/dotnet-tools.json

@@ -10,7 +10,7 @@
       "rollForward": true
     },
     "incrementalist.cmd": {
-      "version": "1.2.1",
+      "version": "1.2.2",
       "commands": [
         "incrementalist"
       ],

BREAKING_CHANGES_V1.6.md

@@ -42,6 +42,7 @@ of `Behavior`, `Wire`, `API` (combine with `+`).
 |--------|-------------|-----------|------|--------|-----------|
 | Planned | [#8324](https://github.com/akkadotnet/akka.net/pull/8324) | `Akka.Routing` | Behavior | `ConsistentHash<T>` no longer retains the `SortedDictionary` passed to its public constructor — the ring is snapshotted into internal sorted arrays. Mutating that dictionary after construction no longer affects the instance (previously the aliasing was inconsistent: `IsEmpty` and `operator +`/`-` read it live, while `NodeFor` froze it after the first lookup). A `null` dictionary now throws `ArgumentNullException` from the constructor instead of surfacing later as a `NullReferenceException`. No public API removed; the ring built by `ConsistentHash.Create` is byte-identical. (#8293) | None for normal use — `ConsistentHash.Create` already builds the dictionary fully before constructing, so routers/receptionists are unaffected. If you call the `ConsistentHash(SortedDictionary, int)` constructor directly, populate the dictionary before passing it and don't rely on post-construction mutation being visible. |
 | Planned | `feature/default-bounded-shard-rebalancing` | `Akka.Cluster.Sharding` | Behavior | The default `rebalance-absolute-limit` is now `20`, selecting the bounded shard allocation strategy instead of the legacy threshold-based strategy. | To retain the legacy strategy temporarily, explicitly set `akka.cluster.sharding.least-shard-allocation-strategy.rebalance-absolute-limit = 0`. Review `rebalance-threshold` and `max-simultaneous-rebalance`, which do not apply while the bounded strategy is active. |
+| Planned | `fix/artery-inbound-quarantine-check` | `Akka.Remote` (Artery) | Behavior | Quarantine is now enforced on the INBOUND path too. Previously Artery only gated outbound sends -- an envelope arriving FROM a uid this system has quarantined was still delivered, and the quarantined peer was only notified once, proactively, at the moment `Quarantine()` was called. A new `InboundQuarantineCheckStage`, woven into the inbound pipeline right after handshake, now drops every inbound envelope (ordinary or control, including system messages) whose origin uid is quarantined, and reactively re-sends a `Quarantined` control notice to the origin for each drop (except for a heartbeat or the peer's own `Quarantined` notice, to avoid a reply storm). The existing one-shot proactive notice in `Quarantine()` is unchanged. Additionally, an ordinary/large outbound stream that terminates while its association is quarantined no longer wedges permanently: the materialize-once gate is released (timer-driven auto-reconnect stays suppressed), so a quarantine-piercing `ActorSelection` send -- or any send after a new incarnation's handshake lifts the quarantine -- re-materializes the stream on demand and can reach a restarted peer at the same address. | No action required -- this is a bug fix restoring the documented "no further communication" guarantee of quarantine and the documented new-incarnation piercing behavior; code that (incorrectly) depended on a quarantined peer's replies still arriving is unsupported. |
 | Planned | `fix/artery-daemonmsgcreate-control-stream` | `Akka.Remote` (Artery) | Behavior | Remote deployment's `DaemonMsgCreate` now travels over Artery's CONTROL stream (as a plain envelope, no delivery/ack sequencing) instead of the ordinary stream, ordering it ahead of the `Watch` that remote deployment sends immediately afterwards. Previously the two rode independent, unordered TCP connections and `Watch` systematically arrived first, so the receiver replied `DeathWatchNotification(existenceConfirmed: false)` for a not-yet-created actor and the deployer reaped the freshly-deployed routee before its `Supervise` registration landed, emptying cluster router pools. Additionally (Pekko parity), inbound ordinary messages addressed to a remote-deployed recipient that has not been created yet are no longer dead-lettered immediately: the resolve is retried on a bounded schedule (20 attempts x 50ms, buffered per recipient in FIFO order) so first messages that arrive ahead of the in-flight `DaemonMsgCreate` are delivered once the actor exists; paths that never resolve are banned and dead-letter as before. | No action required -- this is a bug fix restoring correct create-before-watch ordering; code that (incorrectly) depended on the old race is unsupported. |
 | Planned | `feature/artery-test-mode` | `Akka.Remote` | Behavior | System UID generation (`AddressUidExtension` / `AddressUid`) now uses a cryptographic RNG instead of the `Environment.TickCount`-seeded `ThreadLocalRandom`. Previously, multiple processes started within the same millisecond tick (e.g. every node of a multi-node test spawning at once) could draw IDENTICAL system UIDs, silently corrupting Artery's uid-keyed identity (handshakes, quarantine, association reverse index). UIDs remain nonzero and in the legacy `[1, int.MaxValue]` range at the default (`use-64bit-system-uids = off`). | Nothing required -- UIDs were always documented as random; only code depending on the (buggy) time-seeded determinism could observe a difference. |
 | Planned | `feature/artery-test-mode` | `Akka.Remote` (Artery) / `Akka.Remote.TestKit` | Behavior | New `akka.remote.artery.advanced.test-mode` setting (default `off`, Pekko key parity) enables failure-injection test stages in Artery's stream pipelines, and `ArteryRemoting.ManagementCommand` now honors `SetThrottle` with `Blackhole`/`Unthrottled` (returning `true`; previously always `false`) so TestConductor `Blackhole`/`PassThrough` work over Artery. Rate throttles (`TokenBucket`) and `ForceDisassociate` remain unsupported on Artery (still `false`, matching Pekko). `MultiNodeConfig.TestTransport = true` now also sets Artery test-mode on, and the TestKit `Player` now treats a `false` `ManagementCommand` result as a failure (previously reported success for a silently ignored command). | Nothing at the default (`test-mode = off` composes byte-identical pipelines). Test code that relied on `ManagementCommand` always returning `false` on Artery, or on the Player treating `false` as success, must account for the new semantics. |

build-system/azure-pipeline.mntr-template.yaml

@@ -7,10 +7,20 @@ parameters:
   command: ''
   outputDirectory: ''
   timeoutInMinutes: 120
+  # Job-level continueOnError: when true, a failure in this job is reported (the job
+  # itself still shows red/green) but does NOT fail the overall stage/pipeline result.
+  # Used to run a parallel, non-blocking MNTR variant (e.g. Artery, Linux)
+  # alongside the classic gating MNTR job without risking PR validation.
+  continueOnError: false
+  # Extra environment variables injected into the test-execution step, e.g.
+  # AKKA_MNTR_TRANSPORT: artery to re-point the suite at the Artery transport
+  # (see src/core/Akka.Remote.TestKit/MultiNodeSpec.cs).
+  env: {}
 
 jobs:
   - job: ${{ parameters.name }}
     displayName: ${{ parameters.displayName }}
+    continueOnError: ${{ parameters.continueOnError }}
     timeoutInMinutes: ${{ parameters.timeoutInMinutes }}
     pool:
       vmImage: ${{ parameters.vmImage }}
@@ -49,6 +59,7 @@ jobs:
 
       - script: ${{ parameters.command }}
         continueOnError: true
+        env: ${{ parameters.env }}
 
       - task: PublishTestResults@2
         inputs:
@@ -60,7 +71,10 @@ jobs:
         displayName: 'Copy Build Output'
         inputs:
           sourceFolder: ${{ parameters.outputDirectory }}
-          contents: '**\*'
+          # '**' (not the Windows-style '**\*') so this template also works on
+          # Linux agents; matches azure-pipeline.template.yaml, which already
+          # runs on both platforms.
+          contents: '**'
           targetFolder: $(Build.ArtifactStagingDirectory)
           continueOnError: boolean  # 'true' if future steps should run even if this step fails; defaults to 'false'
       - task: PublishBuildArtifacts@1

build-system/pr-validation.yaml

@@ -113,6 +113,78 @@ jobs:
       mntrFailuresDir: 'TestResults\\multinode'
       mntrFailuresArtifactName: "net_7_mntr_FAILED_windows-$(Build.BuildId)"
 
+  # First-ever Linux MNTR lane: same suite, same incrementalist scoping and
+  # command as "net_mntr_windows" above, on ubuntu-latest (matching the Linux
+  # unit-test job). NON-BLOCKING initially (continueOnError: true at the job
+  # level) because this lane has never run before -- same
+  # promote-to-blocking-after-sustained-green path as the Artery job below.
+  - template: azure-pipeline.mntr-template.yaml
+    parameters:
+      name: "net_mntr_linux"
+      displayName: ".NET Multi-Node Tests (Linux)"
+      vmImage: "ubuntu-latest"
+      command: "dotnet incrementalist run --config .incrementalist/mutliNodeOnly.json --branch $(IncrementalistBaseBranch) -- test -c Release --no-build --framework net10.0 --logger:trx --results-directory TestResults/multinode"
+      outputDirectory: "TestResults"
+      artifactName: "net_mntr_linux-$(Build.BuildId)"
+      mntrFailuresDir: 'TestResults/multinode'
+      mntrFailuresArtifactName: "net_mntr_FAILED_linux-$(Build.BuildId)"
+      continueOnError: true
+
+  # Independent, parallel, NON-BLOCKING Artery MNTR stage. Runs the exact same
+  # incrementally-scoped multi-node suite as "net_mntr_windows" above, but
+  # re-points the transport at Artery via AKKA_MNTR_TRANSPORT=artery, which
+  # MultiNodeConfig.Config layers in as a low-priority fallback config tier
+  # (see src/core/Akka.Remote.TestKit/MultiNodeSpec.cs). Purpose: track
+  # Artery's own MNTR green-rate as a ship-confidence metric, independent of
+  # classic DotNetty, without gating PR validation on it.
+  #
+  # Non-blocking: continueOnError: true is set at the job level (in
+  # azure-pipeline.mntr-template.yaml), not just on the test step. The job
+  # still runs and reports its own pass/fail per PR (visible in the Checks
+  # list), but it can never fail PR validation as a whole.
+  #
+  # Failure injection runs for real in this lane: AKKA_MNTR_TRANSPORT=artery
+  # also enables akka.remote.artery.advanced.test-mode (see
+  # src/core/Akka.Remote.TestKit/MultiNodeSpec.cs), which supports
+  # TestConductor Blackhole/PassThrough. Rate throttles are not supported by
+  # test-mode, so throttle-using specs (e.g. TestConductorSpec) stay pinned to
+  # classic remoting and fail fast on the RequireTestConductorTransport()
+  # check in src/core/Akka.Remote.TestKit/Conductor.cs. Treat everything else
+  # failing in this lane as a real Artery finding. If this lane ever needs a
+  # different test-project set than the classic job, the clean lever is a
+  # dedicated incrementalist config file (.incrementalist/mutliNodeOnly.json
+  # is the sanctioned mechanism for dictating which test projects run in CI).
+  - template: azure-pipeline.mntr-template.yaml
+    parameters:
+      name: "net_mntr_windows_artery"
+      displayName: ".NET Multi-Node Tests (Windows, Artery)"
+      vmImage: "windows-latest"
+      command: "dotnet incrementalist run --config .incrementalist/mutliNodeOnly.json --branch $(IncrementalistBaseBranch) -- test -c Release --no-build --framework net10.0 --logger:trx --results-directory TestResults/multinode"
+      outputDirectory: "TestResults"
+      artifactName: "net_mntr_windows_artery-$(Build.BuildId)"
+      mntrFailuresDir: 'TestResults\\multinode'
+      mntrFailuresArtifactName: "net_mntr_artery_FAILED_windows-$(Build.BuildId)"
+      continueOnError: true
+      env:
+        AKKA_MNTR_TRANSPORT: artery
+
+  # Fourth lane, completing the classic/Artery x Windows/Linux MNTR matrix
+  # (only classic Windows is blocking). Same rationale and known/accepted
+  # noise as the Windows Artery lane above.
+  - template: azure-pipeline.mntr-template.yaml
+    parameters:
+      name: "net_mntr_linux_artery"
+      displayName: ".NET Multi-Node Tests (Linux, Artery)"
+      vmImage: "ubuntu-latest"
+      command: "dotnet incrementalist run --config .incrementalist/mutliNodeOnly.json --branch $(IncrementalistBaseBranch) -- test -c Release --no-build --framework net10.0 --logger:trx --results-directory TestResults/multinode"
+      outputDirectory: "TestResults"
+      artifactName: "net_mntr_linux_artery-$(Build.BuildId)"
+      mntrFailuresDir: 'TestResults/multinode'
+      mntrFailuresArtifactName: "net_mntr_artery_FAILED_linux-$(Build.BuildId)"
+      continueOnError: true
+      env:
+        AKKA_MNTR_TRANSPORT: artery
+
   - template: azure-pipeline.template.yaml
     parameters:
       name: "nuget_pack"

openspec/changes/artery-tcp-remoting/design.md

@@ -164,7 +164,7 @@ Verified against Pekko `Handshake.scala` / `Association.scala` / `ArteryTranspor
 
 **Association state machine:** `AssociationRegistry` keyed by remote **Address** (one Association per address, CAS-materialized) + an `association(uid)` reverse lookup (None until handshake completes). Per-association `AssociationState` (volatile, CAS-swapped) with `uniqueRemoteAddress`: **Associating** (UID unknown — gates OutboundHandshake) → **Associated** (`completeHandshake` sets it) → **Quarantined**. A **different** incoming UID (remote restart) → `newIncarnation` + atomic swap + clear outbound compression (UID-change → reset); the old UID is not auto-quarantined.
 
-**Quarantine (UID-scoped):** acts only if the uid matches the current `uniqueRemoteAddress().uid` (stale-UID request ignored); swaps `newQuarantined`, emits `QuarantinedEvent`, clears compression, sends `ClearSystemMessageDelivery(incarnation)`. Only `ActorSelectionMessage` + `ClearSystemMessageDelivery` pierce. A **new incarnation re-associates** (keyed by Address; a new UID installs a fresh non-quarantined incarnation while the old UID stays quarantined). Prune after `remove-quarantined-association-after` = 1h.
+**Quarantine (UID-scoped):** acts only if the uid matches the current `uniqueRemoteAddress().uid` (stale-UID request ignored); swaps `newQuarantined`, emits `QuarantinedEvent`, clears compression, sends `ClearSystemMessageDelivery(incarnation)`. Only `ActorSelectionMessage` + `ClearSystemMessageDelivery` pierce. The pierce is not uid-checked and may reach the quarantined incarnation itself, which processes it; the inbound quarantine check drops the reply, so the exchange is one-directional. This is deliberate: the sender cannot learn that the peer restarted without sending to it, and the not-restarted case is the price of that discovery. Classic remoting differs -- it refuses the quarantined uid at handshake (`HopelessAssociation` via `refuseUid`), so a pierce there can only ever reach a new incarnation. A **new incarnation re-associates** (keyed by Address; a new UID installs a fresh non-quarantined incarnation while the old UID stays quarantined). Prune after `remove-quarantined-association-after` = 1h. **Implementation gap:** the config key is not parsed and no pruning exists yet -- `QuarantinedUids` accumulates per association for the life of the process (classic bounds the same state with `prune-quarantine-marker-after = 5 d`, documented there as leak GC, not re-admission).
 
 **Provider integration:** the `RemoteTransport` seam already exists — `RemoteActorRef.Tell → Remote.Send`; the provider creates refs via `new RemoteActorRef(Transport,…)`; `DefaultAddress` / `LocalAddressForRemote` / `Quarantine` all delegate to the transport. So `ArteryRemoting : RemoteTransport` implements **9 abstract members** (two `ManagementCommand` overloads — the design draft said 8) and needs **no change** to `RemoteActorRef` or the ref-creation path. **The one wiring change:** `RemoteActorRefProvider.CreateInternals()` hard-codes `new Remoting(…)` — add a config switch (`akka.remote.artery.enabled = on` → `ArteryRemoting`, else classic) by making it read `RemoteSettings` or overriding in a subclass. **Two nodes must run the same transport** (wire + scheme differ: classic `akka.tcp://`, Artery `akka://`) — homogeneous cluster; fail fast on a mixed config.
 

openspec/changes/artery-tcp-remoting/specs/artery-tcp-remoting/spec.md

@@ -96,6 +96,32 @@ System messages SHALL be delivered with explicit ACK/NACK and resend semantics i
 - **WHEN** the bounded system-message buffer overflows
 - **THEN** the association SHALL be quarantined
 
+### Requirement: Quarantine is one-directional and uid-scoped
+
+Quarantine SHALL block traffic for the quarantined uid in both directions on the
+quarantining system, with a deliberate outbound exemption that lets the sender
+discover a restarted peer. The exemption MAY deliver a message to the quarantined
+incarnation itself; the sender cannot know whether the peer restarted without
+sending to it. The reply from a quarantined incarnation SHALL NOT be delivered.
+
+#### Scenario: ActorSelection pierces outbound quarantine
+- **WHEN** an `ActorSelectionMessage` is sent toward a quarantined association
+- **THEN** it SHALL be transmitted to the peer address, even if the peer is still running as the quarantined uid
+
+#### Scenario: Inbound traffic from a quarantined uid is dropped
+- **WHEN** an envelope arrives whose origin uid this system has quarantined
+- **THEN** the envelope SHALL be dropped before dispatch, system messages included
+- **AND** a quarantine control notice SHALL be sent to the origin, unless the dropped message is itself a quarantine notice or a heartbeat
+
+#### Scenario: New incarnation lifts quarantine
+- **WHEN** a completed handshake installs a different uid for a quarantined association
+- **THEN** traffic to and from the new uid SHALL flow normally
+- **AND** the old uid SHALL remain quarantined
+
+#### Scenario: Quarantined peer does not retaliate
+- **WHEN** a system receives notice that a peer has quarantined it
+- **THEN** it SHALL publish `ThisActorSystemQuarantinedEvent` and SHALL NOT quarantine the peer in return, because a reciprocal quarantine can split a cluster
+
 ### Requirement: Outbound queues are bounded
 
 Artery remoting SHALL use bounded outbound queues for user and control traffic.

src/contrib/cluster/Akka.Cluster.Tools.Tests.MultiNode/Singleton/ClusterSingletonManagerLeaveSpec.cs

@@ -36,6 +36,25 @@ public ClusterSingletonManagerLeaveSpecConfig()
                 akka.actor.provider = ""Akka.Cluster.ClusterActorRefProvider, Akka.Cluster""
                 akka.remote.log-remote-lifecycle-events = off
                 akka.cluster.auto-down-unreachable-after = off
+
+                # The harness already runs gossip and leader actions at 200ms
+                # (MultiNodeClusterSpec.ClusterConfig). Left at the 1s default, the
+                # singleton hand-over ladder needs up to 12 retries to give up, about
+                # 12s. That exceeds this spec's own 10s expects and coordinated
+                # shutdown's 10s cluster-exiting phase, which turns the asserted
+                # stop-before-MemberRemoved order into a race. Match the harness tempo
+                # instead of raising any timeout.
+                akka.cluster.singleton.hand-over-retry-interval = 200ms
+                akka.cluster.singleton-proxy.singleton-identification-interval = 200ms
+
+                # The retry count is derived from min-number-of-hand-over-retries, so the
+                # 200ms interval above also shrinks the manager's give-up patience. The
+                # default count gave only 2.4s, and the artery lane caught the manager
+                # giving up (ClusterSingletonManagerIsStuckException) before the new
+                # oldest took over. 28 hand-over retries give 25 take-over retries: 5s
+                # of patience in 200ms ticks, above the observed take-over latency and
+                # below this spec's own 10s expects.
+                akka.cluster.singleton.min-number-of-hand-over-retries = 28
             ")
             .WithFallback(ClusterSingleton.DefaultConfig())
             .WithFallback(ClusterSingletonProxy.DefaultConfig())