docs: specify quarantine pierce semantics in the artery spec

The spec said only THAT ActorSelectionMessage pierces a quarantined
association, never what the pierce may reach. It now states the resolved
semantics: the pierce is not uid-checked, may reach the quarantined incarnation
itself, and only the reply is dropped - the sender cannot discover a restarted
peer without sending to it. Four scenarios cover the outbound exemption, the
inbound drop-and-renotify, the new-incarnation lift, and the no-retaliation
rule, so the behavior is regression-testable instead of folklore.

Also records in design.md that remove-quarantined-association-after is
specified but not yet implemented (filed separately), and tightens the wording
of the comments this PR adds.

Author: Aaronontheweb

BREAKING_CHANGES_V1.6.md

@@ -41,7 +41,7 @@ of `Behavior`, `Wire`, `API` (combine with `+`).
 | Status | PR / Branch | Component | Type | Change | Migration |
 |--------|-------------|-----------|------|--------|-----------|
 | Planned | `feature/default-bounded-shard-rebalancing` | `Akka.Cluster.Sharding` | Behavior | The default `rebalance-absolute-limit` is now `20`, selecting the bounded shard allocation strategy instead of the legacy threshold-based strategy. | To retain the legacy strategy temporarily, explicitly set `akka.cluster.sharding.least-shard-allocation-strategy.rebalance-absolute-limit = 0`. Review `rebalance-threshold` and `max-simultaneous-rebalance`, which do not apply while the bounded strategy is active. |
-| Planned | `fix/artery-inbound-quarantine-check` | `Akka.Remote` (Artery) | Behavior | Quarantine is now enforced on the INBOUND path too (Pekko's `InboundQuarantineCheck`, ported). Previously Artery only gated outbound sends -- an envelope arriving FROM a uid this system has quarantined was still delivered, and the quarantined peer was only notified once, proactively, at the moment `Quarantine()` was called. A new `InboundQuarantineCheckStage`, woven into the inbound pipeline right after handshake, now drops every inbound envelope (ordinary or control, including system messages) whose origin uid is quarantined, and reactively re-sends a `Quarantined` control notice to the origin for each drop (except for a heartbeat or the peer's own `Quarantined` notice, to avoid a reply storm). The existing one-shot proactive notice in `Quarantine()` is unchanged -- Pekko keeps both. Additionally, an ordinary/large outbound stream that terminates while its association is quarantined no longer wedges permanently: the materialize-once gate is released (timer-driven auto-reconnect stays suppressed), so a quarantine-piercing `ActorSelection` send -- or any send after a new incarnation's handshake lifts the quarantine -- re-materializes the stream on demand and can reach a restarted peer at the same address (Pekko-parity piercing semantics). | No action required -- this is a bug fix restoring the documented "no further communication" guarantee of quarantine and the documented new-incarnation piercing behavior; code that (incorrectly) depended on a quarantined peer's replies still arriving is unsupported. |
+| Planned | `fix/artery-inbound-quarantine-check` | `Akka.Remote` (Artery) | Behavior | Quarantine is now enforced on the INBOUND path too. Previously Artery only gated outbound sends -- an envelope arriving FROM a uid this system has quarantined was still delivered, and the quarantined peer was only notified once, proactively, at the moment `Quarantine()` was called. A new `InboundQuarantineCheckStage`, woven into the inbound pipeline right after handshake, now drops every inbound envelope (ordinary or control, including system messages) whose origin uid is quarantined, and reactively re-sends a `Quarantined` control notice to the origin for each drop (except for a heartbeat or the peer's own `Quarantined` notice, to avoid a reply storm). The existing one-shot proactive notice in `Quarantine()` is unchanged. Additionally, an ordinary/large outbound stream that terminates while its association is quarantined no longer wedges permanently: the materialize-once gate is released (timer-driven auto-reconnect stays suppressed), so a quarantine-piercing `ActorSelection` send -- or any send after a new incarnation's handshake lifts the quarantine -- re-materializes the stream on demand and can reach a restarted peer at the same address. | No action required -- this is a bug fix restoring the documented "no further communication" guarantee of quarantine and the documented new-incarnation piercing behavior; code that (incorrectly) depended on a quarantined peer's replies still arriving is unsupported. |
 | Planned | `fix/artery-daemonmsgcreate-control-stream` | `Akka.Remote` (Artery) | Behavior | Remote deployment's `DaemonMsgCreate` now travels over Artery's CONTROL stream (as a plain envelope, no delivery/ack sequencing) instead of the ordinary stream, ordering it ahead of the `Watch` that remote deployment sends immediately afterwards. Previously the two rode independent, unordered TCP connections and `Watch` systematically arrived first, so the receiver replied `DeathWatchNotification(existenceConfirmed: false)` for a not-yet-created actor and the deployer reaped the freshly-deployed routee before its `Supervise` registration landed, emptying cluster router pools. Additionally (Pekko parity), inbound ordinary messages addressed to a remote-deployed recipient that has not been created yet are no longer dead-lettered immediately: the resolve is retried on a bounded schedule (20 attempts x 50ms, buffered per recipient in FIFO order) so first messages that arrive ahead of the in-flight `DaemonMsgCreate` are delivered once the actor exists; paths that never resolve are banned and dead-letter as before. | No action required -- this is a bug fix restoring correct create-before-watch ordering; code that (incorrectly) depended on the old race is unsupported. |
 | Planned | `feature/artery-test-mode` | `Akka.Remote` | Behavior | System UID generation (`AddressUidExtension` / `AddressUid`) now uses a cryptographic RNG instead of the `Environment.TickCount`-seeded `ThreadLocalRandom`. Previously, multiple processes started within the same millisecond tick (e.g. every node of a multi-node test spawning at once) could draw IDENTICAL system UIDs, silently corrupting Artery's uid-keyed identity (handshakes, quarantine, association reverse index). UIDs remain nonzero and in the legacy `[1, int.MaxValue]` range at the default (`use-64bit-system-uids = off`). | Nothing required -- UIDs were always documented as random; only code depending on the (buggy) time-seeded determinism could observe a difference. |
 | Planned | `feature/artery-test-mode` | `Akka.Remote` (Artery) / `Akka.Remote.TestKit` | Behavior | New `akka.remote.artery.advanced.test-mode` setting (default `off`, Pekko key parity) enables failure-injection test stages in Artery's stream pipelines, and `ArteryRemoting.ManagementCommand` now honors `SetThrottle` with `Blackhole`/`Unthrottled` (returning `true`; previously always `false`) so TestConductor `Blackhole`/`PassThrough` work over Artery. Rate throttles (`TokenBucket`) and `ForceDisassociate` remain unsupported on Artery (still `false`, matching Pekko). `MultiNodeConfig.TestTransport = true` now also sets Artery test-mode on, and the TestKit `Player` now treats a `false` `ManagementCommand` result as a failure (previously reported success for a silently ignored command). | Nothing at the default (`test-mode = off` composes byte-identical pipelines). Test code that relied on `ManagementCommand` always returning `false` on Artery, or on the Player treating `false` as success, must account for the new semantics. |

openspec/changes/artery-tcp-remoting/design.md

@@ -164,7 +164,7 @@ Verified against Pekko `Handshake.scala` / `Association.scala` / `ArteryTranspor
 
 **Association state machine:** `AssociationRegistry` keyed by remote **Address** (one Association per address, CAS-materialized) + an `association(uid)` reverse lookup (None until handshake completes). Per-association `AssociationState` (volatile, CAS-swapped) with `uniqueRemoteAddress`: **Associating** (UID unknown — gates OutboundHandshake) → **Associated** (`completeHandshake` sets it) → **Quarantined**. A **different** incoming UID (remote restart) → `newIncarnation` + atomic swap + clear outbound compression (UID-change → reset); the old UID is not auto-quarantined.
 
-**Quarantine (UID-scoped):** acts only if the uid matches the current `uniqueRemoteAddress().uid` (stale-UID request ignored); swaps `newQuarantined`, emits `QuarantinedEvent`, clears compression, sends `ClearSystemMessageDelivery(incarnation)`. Only `ActorSelectionMessage` + `ClearSystemMessageDelivery` pierce. A **new incarnation re-associates** (keyed by Address; a new UID installs a fresh non-quarantined incarnation while the old UID stays quarantined). Prune after `remove-quarantined-association-after` = 1h.
+**Quarantine (UID-scoped):** acts only if the uid matches the current `uniqueRemoteAddress().uid` (stale-UID request ignored); swaps `newQuarantined`, emits `QuarantinedEvent`, clears compression, sends `ClearSystemMessageDelivery(incarnation)`. Only `ActorSelectionMessage` + `ClearSystemMessageDelivery` pierce. The pierce is not uid-checked and may reach the quarantined incarnation itself, which processes it; the inbound quarantine check drops the reply, so the exchange is one-directional. This is deliberate: the sender cannot learn that the peer restarted without sending to it, and the not-restarted case is the price of that discovery. Classic remoting differs -- it refuses the quarantined uid at handshake (`HopelessAssociation` via `refuseUid`), so a pierce there can only ever reach a new incarnation. A **new incarnation re-associates** (keyed by Address; a new UID installs a fresh non-quarantined incarnation while the old UID stays quarantined). Prune after `remove-quarantined-association-after` = 1h. **Implementation gap:** the config key is not parsed and no pruning exists yet -- `QuarantinedUids` accumulates per association for the life of the process (classic bounds the same state with `prune-quarantine-marker-after = 5 d`, documented there as leak GC, not re-admission).
 
 **Provider integration:** the `RemoteTransport` seam already exists — `RemoteActorRef.Tell → Remote.Send`; the provider creates refs via `new RemoteActorRef(Transport,…)`; `DefaultAddress` / `LocalAddressForRemote` / `Quarantine` all delegate to the transport. So `ArteryRemoting : RemoteTransport` implements **9 abstract members** (two `ManagementCommand` overloads — the design draft said 8) and needs **no change** to `RemoteActorRef` or the ref-creation path. **The one wiring change:** `RemoteActorRefProvider.CreateInternals()` hard-codes `new Remoting(…)` — add a config switch (`akka.remote.artery.enabled = on` → `ArteryRemoting`, else classic) by making it read `RemoteSettings` or overriding in a subclass. **Two nodes must run the same transport** (wire + scheme differ: classic `akka.tcp://`, Artery `akka://`) — homogeneous cluster; fail fast on a mixed config.
 

openspec/changes/artery-tcp-remoting/specs/artery-tcp-remoting/spec.md

@@ -96,6 +96,32 @@ System messages SHALL be delivered with explicit ACK/NACK and resend semantics i
 - **WHEN** the bounded system-message buffer overflows
 - **THEN** the association SHALL be quarantined
 
+### Requirement: Quarantine is one-directional and uid-scoped
+
+Quarantine SHALL block traffic for the quarantined uid in both directions on the
+quarantining system, with a deliberate outbound exemption that lets the sender
+discover a restarted peer. The exemption MAY deliver a message to the quarantined
+incarnation itself; the sender cannot know whether the peer restarted without
+sending to it. The reply from a quarantined incarnation SHALL NOT be delivered.
+
+#### Scenario: ActorSelection pierces outbound quarantine
+- **WHEN** an `ActorSelectionMessage` is sent toward a quarantined association
+- **THEN** it SHALL be transmitted to the peer address, even if the peer is still running as the quarantined uid
+
+#### Scenario: Inbound traffic from a quarantined uid is dropped
+- **WHEN** an envelope arrives whose origin uid this system has quarantined
+- **THEN** the envelope SHALL be dropped before dispatch, system messages included
+- **AND** a quarantine control notice SHALL be sent to the origin, unless the dropped message is itself a quarantine notice or a heartbeat
+
+#### Scenario: New incarnation lifts quarantine
+- **WHEN** a completed handshake installs a different uid for a quarantined association
+- **THEN** traffic to and from the new uid SHALL flow normally
+- **AND** the old uid SHALL remain quarantined
+
+#### Scenario: Quarantined peer does not retaliate
+- **WHEN** a system receives notice that a peer has quarantined it
+- **THEN** it SHALL publish `ThisActorSystemQuarantinedEvent` and SHALL NOT quarantine the peer in return, because a reciprocal quarantine can split a cluster
+
 ### Requirement: Outbound queues are bounded
 
 Artery remoting SHALL use bounded outbound queues for user and control traffic.

src/core/Akka.Remote.Tests.MultiNode/RemoteRestartedQuarantinedSpec.cs

@@ -128,7 +128,7 @@ await RunOnAsync(async () =>
 
                 var actorRef = (await _identifyWithUid(_config.First, "subject")).Item2;
 
-                // See the matching barrier in `first`'s block (Pekko artery-variant parity).
+                // See the matching barrier comment in `first`'s block.
                 await EnterBarrierAsync("before-quarantined");
 
                 await EnterBarrierAsync("quarantined");

src/core/Akka.Remote/Artery/ArteryRemoting.cs

@@ -755,7 +755,7 @@ private void HandleIncomingConnection(Tcp.IncomingConnection connection)
             // would let them reach the acker first.
             //
             // Ordinary, control and large connections all use this one sink, so one instance is
-            // sufficient. Pekko has two sinks and inserts its equivalent stage in each.
+            // sufficient.
             var inboundSink = decoded
                 .Via(Flow.FromGraph(new InboundHandshakeStage(_inboundContext!)))
                 .Via(Flow.FromGraph(new InboundQuarantineCheckStage(_inboundContext!)))

src/core/Akka.Remote/Artery/InboundQuarantineCheckStage.cs

@@ -65,8 +65,6 @@ namespace Akka.Remote.Artery
     /// <c>ArteryInboundProcessingStage.ProcessFrameLaneMode</c> does the same check in its own code.
     /// The two sites call <see cref="ShouldNotifyOrigin"/> so that they stay in agreement.
     /// </para>
-    ///
-    /// <para>This is a port of Pekko's <c>InboundQuarantineCheck</c>.</para>
     /// </summary>
     internal sealed class InboundQuarantineCheckStage : GraphStage<FlowShape<IInboundEnvelope, IInboundEnvelope>>
     {
@@ -116,15 +114,23 @@ internal static bool ShouldNotifyOrigin(object message) =>
             message is not ArteryQuarantined && !IsHeartbeat(message);
 
         /// <summary>
-        /// Pekko's <c>isHeartbeat</c>: matches its own control-channel liveness ping/pong
-        /// (<see cref="ArteryHeartbeat"/>/<see cref="ArteryHeartbeatRsp"/> -- the artery analog
-        /// of Pekko's <c>RemoteWatcher.ArteryHeartbeat</c>/<c>ArteryHeartbeatRsp</c>, both
-        /// <c>HeartbeatMessage</c>) plus <see cref="RemoteWatcher"/>'s own ordinary-stream
-        /// heartbeat (<see cref="IPriorityMessage"/> -- the marker <see cref="RemoteWatcher.Heartbeat"/>/
-        /// <see cref="RemoteWatcher.HeartbeatRsp"/> implement, this codebase's analog of Pekko's
-        /// shared <c>HeartbeatMessage</c> trait), including when it arrives wrapped in an
-        /// <see cref="ActorSelectionMessage"/> (Pekko's <c>ActorSelectionMessage(_: HeartbeatMessage, _, _)</c>
-        /// -- <see cref="RemoteWatcher"/> sends its heartbeat via <c>Context.ActorSelection(...).Tell</c>).
+        /// Tells the caller if <paramref name="message"/> is a heartbeat. Three groups of
+        /// messages match:
+        /// <list type="bullet">
+        /// <item><description>
+        /// The control-channel liveness messages <see cref="ArteryHeartbeat"/> and
+        /// <see cref="ArteryHeartbeatRsp"/>.
+        /// </description></item>
+        /// <item><description>
+        /// The <see cref="RemoteWatcher"/> heartbeats on the ordinary stream. They implement
+        /// the marker interface <see cref="IPriorityMessage"/>.
+        /// </description></item>
+        /// <item><description>
+        /// A <see cref="RemoteWatcher"/> heartbeat inside an <see cref="ActorSelectionMessage"/>.
+        /// <see cref="RemoteWatcher"/> sends its heartbeat through an ActorSelection, so the
+        /// heartbeat arrives in that wrapper.
+        /// </description></item>
+        /// </list>
         /// </summary>
         private static bool IsHeartbeat(object message) => message switch
         {