De-flake ClusterSingletonProxySpec, ClusterSingletonRestartSpec, and DistributedPubSubRestartSpec

[Aaronontheweb]

Wave-1 de-flake batch: three flaky test families, test-only changes, no product code touched. No timeout was raised, no sleep or retry backoff added anywhere — every fix removes a structural race or a blocking wait.

ClusterSingletonProxySpec (failed build 130872, Linux, ~36s in-test)

The identify test was synchronous end to end: TestProxy blocked on ExpectMsg for up to 25s per node while five in-process ActorSystems competed for the same starved thread pool, and nothing waited for the singleton to exist before sending — cluster formation, singleton startup, and identification all had to fit inside one message timeout.

• The test is now async, with two explicit gates: all five nodes see five members Up, then each proxy publishes IdentifySingletonResult.Success. The event-stream subscription is made before the proxy actor is created, so the result cannot be missed; the wait fishes for Success because the proxy also publishes Timeout results on a zero-initial-delay timer.
• The zero-buffering test gated on membership, then sent once into a proxy with BufferSize = 0 — which drops what it cannot forward. Membership does not imply identification, so the only message could vanish silently. It now gates on the identification result, and terminates its seed system instead of leaving it gossiping through the rest of the assembly.
• Two waits in the timeout tests inherited single-expect-default (3s) on different TestKit instances, so their budgets did not nest and the retry loop got roughly one attempt against a queue pre-loaded with 500ms-cadence Timeout results. Both now fish with explicit 30s bounds.
• Teardown awaits Task.WhenAll — the old Wait(30s) discarded its result, letting five cluster systems keep shutting down into the next test.

ClusterSingletonRestartSpec (failed build 130838, Windows, 15s removal window)

• Shutdown(_sys1) waits 5s, then silently force-stops the user guardian — which does not stop /system, so remoting kept sys1's listener bound while sys3 was created on the same host:port. tcp-reuse-addr = off-for-windows makes that rebind a Windows-only failure, matching the Windows-only flake. Now await _sys1.Terminate(): the hand-over completes and the port is released.
• Each proxy assertion created a TestProbe inside its retry loop. CreateTestProbe blocks the caller until the probe's PreStart runs and swaps the calling thread's SynchronizationContext — a blocking wait on work that itself needs a pool thread, once per ~1.1s attempt, at four sites. One probe per phase now; only the send is retried.
• The reported 15s failure waits for sys2 to be fully Removed, which runs sys2's cluster-exiting phase — and that phase blocks on the singleton hand-over with a 10s timeout. JoinAsync only proves sys3's own view, so sys2 could still see sys3 as Joining when it left: no hand-over target, phase runs to timeout, removal misses the window. A mutual-convergence gate now requires both sys2 and sys3 to see the same two-member Up cluster before the Leave.
• Join retries moved from 100ms to 500ms: a JoinTo arriving while the daemon is in TryingToJoin drops it back to Uninitialized and restarts the handshake, so ten retries a second was load on the very daemon the test was waiting on. The join must still be re-issued — sys3 reuses sys1's address and is refused until the old incarnation is removed.

DistributedPubSubRestartSpec (failed build 130808)

The confirmed occurrence livelocked on a reused Identify probe: association establishment flushes the buffered Identifies as a burst of ActorIdentity(null), and each retry then read one stale null forever — 80 attempts in 25s, none of them waiting.

• The restart-kill loop now uses ActorSelection.ResolveOne inside the retry: the Identify round trip is the delivery confirmation, its temp actor is fresh per attempt (structurally immune to the stale-null livelock), and final failure raises ActorNotFoundException naming the unresolved path instead of a bare timeout on a null Subject.
• Resolve and kill stay in one retry on purpose: splitting them lets the outbound stream drop the kill in the gap between a successful resolve and the Tell, with no resend — a local soak reproduced exactly that. The comment in the spec documents it so nobody re-splits it.
• The 45s bound is derived, not guessed: third's restart (~15s) + one association cycle (15s associate timeout + 5s gate) + handshake and ack (~5s) ≈ 40s, nesting inside the 120s end barrier with 45s of headroom.
• Baseline DeltaCount reads moved off TestActor (subscribed to topic1) onto probes with explicit bounds; CountAsync retries on the 500ms gossip tick with a fresh probe and a 1s bound per attempt instead of spending its 10s window on three 3s waits.

Verification

Check	Result
Proxy identify, DOTNET_ThreadPool_ForceMaxWorkerThreads=3, ×10	10/10 green, tail latency gone (12.6–13.7s vs 12.8–18.6s before)
Restart spec, same 3-thread cap, ×12	12/12 green
Both singleton classes, same cap, independent re-run	5/5 green
Full Akka.Cluster.Tools.Tests	98/98 green
DistributedPubSubRestartSpec, artery, ×6 + independent re-run	7/7 green
DistributedPubSubRestartSpec, classic, ×3	3/3 green
Both projects at -warnaserror	0 warnings, 0 errors