Fix Cluster Sharding private replicator recovery by Aaronontheweb · Pull Request #8480 · akkadotnet/akka.net

Aaronontheweb · 2026-08-25T22:45:27Z

Summary

Watch each role-scoped Cluster Sharding DData replicator and recreate it in place after unexpected termination.
Give sharding DData consumers an ICanTell actor selection for the replicator's stable path, so existing coordinators and remember-entities stores automatically address the replacement actor.
Preserve persistence-mode behavior, including configurations that specify DData as the remember-entities store.
Add regression coverage for DData recovery, persistence-mode compatibility, and coordinated shutdown in both state-store modes.

Compatibility and scope

DData and custom state-store modes recreate the private replicator.
The replicator remains a direct child at the existing /system/sharding/<role>Replicator path, preserving compatibility with older cluster members.
Persistence state-store mode does not create a private replicator. Its remember-entities provider remains event sourced even if remember-entities-store = ddata is configured.
The lifecycle message is local-only and implements INoSerializationVerificationNeeded.
Replicators are not recreated during cluster shutdown; shutdown detection uses the coordinated-shutdown reason instead of depending on a late cluster event.
This PR does not modify Akka.DistributedData.

Validation

Akka.Cluster.Sharding.Tests net10.0 build: 0 warnings and 0 errors.
Focused recovery, persistence compatibility, and parameterized coordinated-shutdown tests: 4 passed.
The shutdown test covers both persistence and DData across the existing three-node leaving/downing flow and asserts that no private-replicator termination error is logged.
Slopwatch and whitespace validation passed.

Watch role-scoped sharding DData replicators and recreate them at their stable actor paths when they terminate. Use actor selections so existing coordinators and remember-entities stores continue communicating with the replacement incarnation. Add regression coverage for DData remember entities and the persistence compatibility configuration.

Aaronontheweb

Needs some work

Aaronontheweb · 2026-08-26T17:14:13Z

+
+public class ClusterShardingReplicatorResiliencySpec : AkkaSpec
+{
+    private sealed record EntityEnvelope(string EntityId);


why not use ShardEnvelope and the HashCodeMessageExtractor.Create here, no?

Aaronontheweb · 2026-08-26T17:14:26Z

+    }
+
+    private static Config SpecConfig =>
+        ConfigurationFactory.ParseString(@"


LGTM - this is the config we want to test

Aaronontheweb · 2026-08-26T17:15:03Z

+    }
+
+    [Fact]
+    public async Task Private_replicator_should_recover_at_the_same_path_without_restarting_consumers()


Backwards compat test - make sure that the ActorSelection paths have not diverged between deployments, otherwise the deployment + shard rebalancing it triggers is going to be unable to reconcile until the new version is completely cycled in.

Aaronontheweb · 2026-08-26T17:15:48Z

+        shardWatcher.Watch(await Sys.ActorSelection(shard).ResolveOne(3.Seconds()));
+
+        var firstReplicator = await Sys.ActorSelection(replicatorPath).ResolveOne(3.Seconds());
+        Watch(firstReplicator);


``WatchAsync`

Aaronontheweb · 2026-08-26T17:15:58Z

+        var coordinatorWatcher = CreateTestProbe();
+        coordinatorWatcher.Watch(coordinator);
+        var shardWatcher = CreateTestProbe();
+        shardWatcher.Watch(await Sys.ActorSelection(shard).ResolveOne(3.Seconds()));


WatchAsync for all of these calls

Aaronontheweb · 2026-08-26T17:18:06Z

+                    _replicatorSettingsByRole = _replicatorSettingsByRole.SetItem(role, replicatorSettings);
                }
+
+                return Context.ActorSelection(Self.Path / name);


this sucks - we should just stick with the IActorRef and return that rather than an ICanTell + ActorSelection

Aaronontheweb · 2026-08-26T17:18:53Z


+        private IActorRef CreateReplicator(string role, ReplicatorSettings settings)
+        {
+            var replicator = Context.Watch(Context.ActorOf(


Why not use Context.WatchWith and a custom message type so we know exactly which replicator + role died when we receive the termination message? That would avoid some of the messy look-ups inside TryHandleReplicatorTermination

Aaronontheweb · 2026-08-26T17:21:36Z

+        private bool TryHandleReplicatorTermination(IActorRef terminated)
+        {
+            string role = null;
+            foreach (var (candidateRole, replicator) in _replicatorsByRole)


see previous comments about using Context.WatchWith to avoid the lookups here

Aaronontheweb · 2026-08-26T17:21:48Z

+            if (_clusterShuttingDown || _cluster.IsTerminated)
+                return true;
+
+            _log.Warning(


Log an error, not a warning.

Aaronontheweb · 2026-08-26T17:22:23Z


            Receive<Terminated>(msg =>
            {
+                if (TryHandleReplicatorTermination(msg.ActorRef))


One thing we might need here is a flag for when we're shutting down ourselves, so we don't log a bunch of errors when the shard region is being terminated as part of a normal node exit.

Keep IActorRef throughout the sharding DData consumers and publish a local replacement message when the guardian recreates a terminated replicator. Existing coordinators and remember-entities stores swap to the new ref and retry their active operation, while providers use the replacement for newly-created stores. Use WatchWith role-aware termination messages, preserve the peer-visible actor path, suppress shutdown recreation, and simplify the regression test fixtures.

…71-review-fixes

Aaronontheweb · 2026-08-26T18:16:26Z

Addressed the review in 1ca3718:

restored IActorRef throughout the DData consumer APIs
switched replicator death-watch to WatchWith using a role-aware local message
recreated at the unchanged /system/sharding/Replicator path
published a local ReplicatorChanged message so existing consumers swap refs and retry active operations
updated remember-entities providers so newly-created stores receive the replacement ref
marked both internal lifecycle messages INoSerializationVerificationNeeded
changed unexpected termination logging to Error while retaining shutdown suppression
simplified the regression fixture with ShardEnvelope, HashCodeMessageExtractor, and WatchAsync

The focused recovery tests passed four consecutive rebuilt runs. The full sharding suite was 193/194 with only the unchanged RememberEntitiesStarterSpec timing failure noted in the PR description.

Aaronontheweb

LGTM - I refined this and tried some alternative approaches with the BackoffSupervisor but I think this is probably the best option / least disruptive for Akka.Cluster.Sharding.

Aaronontheweb · 2026-08-26T18:33:48Z

Aaronontheweb · 2026-08-26T21:14:25Z

        public ClusterShardingSettings Settings { get; }
        public int MajorityMinCap { get; }
-        public IActorRef Replicator { get; }
+        public ICanTell Replicator { get; }


Moving to ICanTell allows us to communicate with the replicator based on its ActorPath via ActorSelection, so in the event that it dies / gets replaced this will continue to work - unlike an IActorRef which will get invalidated.

I would've gone with the IActorRef-based approach but it requires A LOT more accounting and moving parts, so I settled for this.

* Fix Cluster Sharding private replicator recovery Watch role-scoped sharding DData replicators and recreate them at their stable actor paths when they terminate. Use actor selections so existing coordinators and remember-entities stores continue communicating with the replacement incarnation. Add regression coverage for DData remember entities and the persistence compatibility configuration. * Address sharding replicator recovery review Keep IActorRef throughout the sharding DData consumers and publish a local replacement message when the guardian recreates a terminated replicator. Existing coordinators and remember-entities stores swap to the new ref and retry their active operation, while providers use the replacement for newly-created stores. Use WatchWith role-aware termination messages, preserve the peer-visible actor path, suppress shutdown recreation, and simplify the regression test fixtures. * Simplify remember entities provider tracking * Simplify sharding replicator recovery * Cover sharding replicator coordinated shutdown

Aaronontheweb added 2 commits August 25, 2026 22:44

Merge branch 'v1.5' into fix/8471-sharding-replicator-recovery

31c554c

Aaronontheweb added akka.net v1.5 Issues affecting Akka.NET v1.5 akka-cluster-sharding akka-ddata labels Aug 26, 2026

Aaronontheweb commented Aug 26, 2026

View reviewed changes

Aaronontheweb added 2 commits August 26, 2026 18:15

Merge remote-tracking branch 'refs/remotes/upstream/v1.5' into fix/84…

a75bff3

…71-review-fixes

Simplify remember entities provider tracking

2d09a4d

This was referenced Aug 26, 2026

De-flake ShardedDaemonProcessSpec multinode spec #8489

Closed

Fix Cluster Sharding DData replicator recovery #8490

Closed

Simplify sharding replicator recovery

24fbe7b

Aaronontheweb added this to the 1.5.71 milestone Aug 26, 2026

Cover sharding replicator coordinated shutdown

835176f

Aaronontheweb commented Aug 26, 2026

View reviewed changes

Aaronontheweb marked this pull request as ready for review August 26, 2026 21:14

Aaronontheweb enabled auto-merge (squash) August 26, 2026 21:14

Aaronontheweb disabled auto-merge August 26, 2026 21:15

Aaronontheweb merged commit 885f466 into akkadotnet:v1.5 Aug 26, 2026
10 of 12 checks passed

Aaronontheweb deleted the fix/8471-sharding-replicator-recovery branch August 26, 2026 21:15

This was referenced Aug 26, 2026

Forward-port #8480 to dev: Cluster Sharding private replicator recovery #8491

Merged

Release Akka.NET v1.5.71 #8493

Merged

Uh oh!

Conversation

Aaronontheweb commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Compatibility and scope

Validation

Uh oh!

Aaronontheweb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aaronontheweb commented Aug 26, 2026

Uh oh!

Aaronontheweb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Aaronontheweb commented Aug 25, 2026 •

edited

Loading