Version Information
dev (v1.5.x line). Affects Akka.Remote (throttle transport adapter).
Describe the bug
If a ThrottledAssociation actor is ever restarted, an inbound association becomes a permanent one-way black hole: it silently drops every packet, and remoting never notices.
Three facts combine:
- Neither
ThrottlerManager nor ActorTransportAdapterManager declares a SupervisorStrategy, so a child that throws is Restarted by default.
- On restart,
ThrottledAssociation.InitializeFSM() re-enters WaitExposedHandle for an inbound association:
if (Inbound)
StartWith(ThrottlerState.WaitExposedHandle, Uninitialized.Instance);
ThrottlerManager sends the Handle message that leaves that state exactly once, in the InboundAssociation case. It is never re-sent after a restart.
src/core/Akka.Remote/Transport/ThrottleTransportAdapter.cs
Impact
The restarted FSM sits in WaitExposedHandle forever, logging unhandled event InboundPayload(...) and discarding every inbound packet on that association. Remoting does not observe the loss because akka.remote.transport-failure-detector.acceptable-heartbeat-pause defaults to 120s - there is no disassociation and no quarantine, just silent one-way data loss.
Observed in practice: cluster heartbeats and transport heartbeats streaming into a dead association for the remainder of the process's life, with the affected pair of nodes permanently unable to see each other.
Expected behavior
A throttler that dies should result in an observable disassociation, not a silently broken association.
Suggested fix
Give ThrottlerManager an explicit SupervisorStrategy of Stop (or Escalate) for throttler children so remoting sees a disassociation it can recover from. Optionally, make WaitExposedHandle stop on an unexpected InboundPayload rather than log-and-drop, so the failure is loud rather than silent.
Scope
The throttle adapter is only in the transport pipeline when applied-adapters includes trttl/gremlin, which in practice means multi-node test runs rather than production deployments. The impact is therefore on test infrastructure: it is per-node and per-association within the actor remoting stack, and it does not affect the TestConductor control plane (which uses its own connection), so barriers continue to work while two nodes become permanently unable to see each other.
Backport only if the v1.5 multi-node suite needs it.
Version Information
dev(v1.5.x line). AffectsAkka.Remote(throttle transport adapter).Describe the bug
If a
ThrottledAssociationactor is ever restarted, an inbound association becomes a permanent one-way black hole: it silently drops every packet, and remoting never notices.Three facts combine:
ThrottlerManagernorActorTransportAdapterManagerdeclares aSupervisorStrategy, so a child that throws is Restarted by default.ThrottledAssociation.InitializeFSM()re-entersWaitExposedHandlefor an inbound association:ThrottlerManagersends theHandlemessage that leaves that state exactly once, in theInboundAssociationcase. It is never re-sent after a restart.src/core/Akka.Remote/Transport/ThrottleTransportAdapter.csImpact
The restarted FSM sits in
WaitExposedHandleforever, loggingunhandled event InboundPayload(...)and discarding every inbound packet on that association. Remoting does not observe the loss becauseakka.remote.transport-failure-detector.acceptable-heartbeat-pausedefaults to 120s - there is no disassociation and no quarantine, just silent one-way data loss.Observed in practice: cluster heartbeats and transport heartbeats streaming into a dead association for the remainder of the process's life, with the affected pair of nodes permanently unable to see each other.
Expected behavior
A throttler that dies should result in an observable disassociation, not a silently broken association.
Suggested fix
Give
ThrottlerManageran explicitSupervisorStrategyofStop(orEscalate) for throttler children so remoting sees a disassociation it can recover from. Optionally, makeWaitExposedHandlestop on an unexpectedInboundPayloadrather than log-and-drop, so the failure is loud rather than silent.Scope
The throttle adapter is only in the transport pipeline when
applied-adaptersincludestrttl/gremlin, which in practice means multi-node test runs rather than production deployments. The impact is therefore on test infrastructure: it is per-node and per-association within the actor remoting stack, and it does not affect the TestConductor control plane (which uses its own connection), so barriers continue to work while two nodes become permanently unable to see each other.Backport only if the v1.5 multi-node suite needs it.