Skip to content

Commit 5dd08e0

Browse files
eordanoclaude
andcommitted
fix: heal missed Pulse peer discovery without widening LiveKit fan-out
Remote avatars intermittently disappear after teleporting to a World and back to Genesis Plaza (and, per the 2026-08-05 recurrence, even without teleporting, under archipelago fragmentation). The remote-avatar pipeline is announcement-driven with no reconciliation loop: since the Pulse integration (PR #7291), a peer is announced to an observer exactly once per interest-set entry, and while Pulse is active LiveKit's AnnounceProfileVersion broadcast is targeted only at wallets that already announced to us. Any dropped, negated, or realm-raced Pulse join therefore produces a permanently invisible peer -- there is no client- or server-side membership reconciliation. (The filed v0.154/ v0.155 100%-repro mechanism, PR #9041's ForceRelease of Pulse-sourced avatars on teleport, is already reverted at the pin; this addresses the residual one-shot-discovery fragility class that still recurs.) - LiveKitMessagesBroadcaster: while Pulse is active, fall back to an untargeted AnnounceProfileVersion broadcast on the island+scene pipes at a slow cadence (~10s) instead of only the targeted send, so any peer whose Pulse join we missed self-heals over a LiveKit room we demonstrably share. To avoid duplicating the Pulse movement/emote stream room-wide, Send's recipient lists now skip any wallet with a live PeerIdCache session -- recruitment into announcedWallets stays unconditional, but effective fan-out composition for Pulse-live peers is unchanged from HEAD. - PulseMultiplayerBus.PlayerState: HandlePlayerLeft now ignores a leave whose SubjectId no longer matches the wallet's current peer id (a late/re-ordered leave for a superseded session), closing the window where a stale leave deletes a freshly re-joined avatar. - PeerIdCache: Remove/RemoveWhereNotInRealm only clear the wallet's reverse mapping when it still points at the peer id being removed, so a dangling forward entry from a re-join can't destroy the live session's reverse lookup out from under the new guard. - Thread PeerIdCache through PulseContainer -> MultiplayerContainer -> LiveKitMultiplayerContainer so the broadcaster can filter by live session. Adversarial review (NEEDS-FIX) found the first pass's announce fallback recruited every Pulse-active peer into announcedWallets within one interval, turning targeted LiveKit movement/emotes into full-room fan-out that duplicated the Pulse stream. This revision keeps the fallback announce-only and filters Send by live PeerIdCache membership, so only genuinely missing peers get materialized and fan-out composition matches HEAD. Add EditMode tests for the join-epoch guard (join/re-join/stale-leave sequencing) and for the broadcaster's fallback-materializes / send-path-excludes-live-session behavior. Fixes #9337. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent df35da1 commit 5dd08e0

13 files changed

Lines changed: 335 additions & 32 deletions

Explorer/Assets/DCL/Multiplayer/Connections/Pulse/PeerIdCache.cs

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -31,11 +31,18 @@ public void Remove(uint peerId)
3131
{
3232
lock (sync)
3333
{
34-
if (peersByWallet.Remove(peerId, out (Web3Address wallet, string realm) entry))
35-
walletsByPeerId.Remove(entry.wallet);
34+
RemoveEntry(peerId);
3635
}
3736
}
3837

38+
private void RemoveEntry(uint peerId)
39+
{
40+
if (peersByWallet.Remove(peerId, out (Web3Address wallet, string realm) entry)
41+
&& walletsByPeerId.TryGetValue(entry.wallet, out uint currentPeerId)
42+
&& currentPeerId == peerId)
43+
walletsByPeerId.Remove(entry.wallet);
44+
}
45+
3946
/// <summary>
4047
/// Atomically iterates all wallets, invokes the callback for each, then clears both caches.
4148
/// </summary>
@@ -121,8 +128,7 @@ public void RemoveWhereNotInRealm(string realm, Action<uint> onPeerRemoved)
121128

122129
foreach (uint peerId in removalBuffer)
123130
{
124-
if (peersByWallet.Remove(peerId, out (Web3Address wallet, string realm) entry))
125-
walletsByPeerId.Remove(entry.wallet);
131+
RemoveEntry(peerId);
126132

127133
onPeerRemoved(peerId);
128134
}

Explorer/Assets/DCL/Multiplayer/Movement/Systems/LiveKitMultiplayerContainer.cs

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,9 +28,10 @@ internal LiveKitMultiplayerContainer(
2828
ISelfProfile selfProfile,
2929
IUserBlockingCache userBlockingCache,
3030
MultiplayerDebugSettings multiplayerDebugSettings,
31-
PulseActivation pulseActivation)
31+
PulseActivation pulseActivation,
32+
PeerIdCache peerIdCache)
3233
{
33-
var broadcaster = new LiveKitMessagesBroadcaster(roomHub.SceneRoom(), messagePipesHub, pulseActivation);
34+
var broadcaster = new LiveKitMessagesBroadcaster(roomHub.SceneRoom(), messagePipesHub, pulseActivation, peerIdCache);
3435

3536
RemoteAnnouncements = new LiveKitRemoteAnnouncements(messagePipesHub, broadcaster);
3637
ProfileBroadcast = new DebounceLiveKitProfileBroadcast(new LiveKitProfileBroadcast(selfProfile, broadcaster));

Explorer/Assets/DCL/Multiplayer/Movement/Systems/MultiplayerContainer.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -234,7 +234,7 @@ public static async UniTask<MultiplayerContainer> CreateAsync(
234234
var pulseActivation = new PulseActivation(FeaturesRegistry.Instance.IsEnabled(FeatureId.Pulse));
235235

236236
PulseContainer pulseContainer = await PulseContainer.CreateAsync(pluginSettingsContainer, identityCache, movementInbox, landscapeData, urlsSource, selfProfile, realmData, pulseActivation, ct);
237-
var liveKitContainer = new LiveKitMultiplayerContainer(roomHub, messagePipesHub, movementInbox, selfProfile, userBlockingCache, multiplayerDebugSettings, pulseActivation);
237+
var liveKitContainer = new LiveKitMultiplayerContainer(roomHub, messagePipesHub, movementInbox, selfProfile, userBlockingCache, multiplayerDebugSettings, pulseActivation, pulseContainer.peerIdCache);
238238

239239
return new MultiplayerContainer(pulseContainer, liveKitContainer, selfProfile, pulseActivation);
240240
}

Explorer/Assets/DCL/Multiplayer/Movement/Systems/PulseContainer.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ internal class PulseContainer : DCLWorldContainer<PulseContainer.Settings>
2020
{
2121
private readonly IWeb3IdentityCache identityCache;
2222
private readonly MovementInbox movementInbox;
23-
private readonly PeerIdCache peerIdCache = new ();
23+
internal readonly PeerIdCache peerIdCache = new ();
2424
private readonly MessagePipe messagePipe = new ();
2525

2626
internal ENetTransport? transport;

Explorer/Assets/DCL/Multiplayer/Movement/Systems/PulseMultiplayerBus.PlayerState.cs

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,12 @@ private void HandlePlayerLeft(IncomingMessage message)
8383

8484
// Realm-agnostic on purpose: removals must process even for peers whose stored realm is stale
8585
if (peerIdCache.TryGetWallet(playerLeft.SubjectId, out Web3Address wallet))
86-
removeIntentions.Enqueue(wallet);
86+
{
87+
if (peerIdCache.TryGetPeerId(wallet, out uint currentPeerId) && currentPeerId != playerLeft.SubjectId)
88+
ReportHub.Log(ReportCategory.MULTIPLAYER, $"Ignoring stale PlayerLeft for {playerLeft.SubjectId}: {wallet} re-joined as {currentPeerId}");
89+
else
90+
removeIntentions.Enqueue(wallet);
91+
}
8792

8893
peerIdCache.Remove(playerLeft.SubjectId);
8994
PurgeQueues(playerLeft.SubjectId);

Explorer/Assets/DCL/Multiplayer/Movement/Tests/PulseMultiplayerBusRealmFilteringShould.cs

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -292,6 +292,63 @@ public void ProcessPlayerLeftForStaleRealmPeer()
292292
Assert.IsFalse(peerIdCache.TryGetWallet(7, out _));
293293
}
294294

295+
// Regression coverage for unity-explorer#9337 (join-epoch guard, potential-fix.patch site 3+4):
296+
// a PlayerLeft for a superseded session (a wallet that already re-joined under a new subject id)
297+
// must not delete the freshly re-joined avatar. At pin, HandlePlayerLeft enqueues the remove
298+
// unconditionally from the dangling forward entry, and PeerIdCache.Remove(7) then deletes the
299+
// *live* wallet->peerId reverse mapping too (peersByWallet[7] still resolves to the wallet even
300+
// though walletsByPeerId[wallet] already points at 9) - exactly the "re-join cancels the stale
301+
// pending leave" gap the report's [INVISIBLE_AVATAR] diagnosis (03b82789c) named.
302+
[Test]
303+
public void IgnoreStalePlayerLeftForSupersededSession()
304+
{
305+
Handle(PlayerJoinedMessage(7, WALLET_1, REALM_A));
306+
DrainAnnouncements();
307+
308+
// Wallet re-joins under a new subject id (e.g. a reconnect burst) before the old session's
309+
// leave is processed; the routing thread always serializes these messages in arrival order.
310+
Handle(PlayerJoinedMessage(9, WALLET_1, REALM_A));
311+
DrainAnnouncements();
312+
313+
// Late/re-ordered leave for the superseded session (subject id 7).
314+
Handle(new ServerMessage { PlayerLeft = new PlayerLeft { SubjectId = 7 } });
315+
316+
using (OwnedBunch<RemoveIntention> bunch = removeIntentions.Bunch())
317+
Assert.IsFalse(bunch.Available(),
318+
"A PlayerLeft for a superseded session must not delete the peer's live re-joined avatar.");
319+
320+
Assert.IsTrue(peerIdCache.TryGetWallet(9, out Web3Address wallet));
321+
Assert.IsTrue(wallet.Equals(WALLET_1));
322+
323+
Assert.IsTrue(peerIdCache.TryGetPeerId(new Web3Address(WALLET_1), out uint currentPeerId),
324+
"The re-join's reverse mapping (wallet -> current peer id) must survive the stale leave; " +
325+
"at pin, PeerIdCache.Remove(7) deletes the *live* session's wallet->peerId entry too.");
326+
Assert.AreEqual(9u, currentPeerId);
327+
}
328+
329+
// Companion to IgnoreStalePlayerLeftForSupersededSession: the guard must only reject leaves for
330+
// superseded sessions, not swallow every future leave for a wallet that has ever re-joined.
331+
[Test]
332+
public void ProcessPlayerLeftForCurrentSessionAfterRejoin()
333+
{
334+
Handle(PlayerJoinedMessage(7, WALLET_1, REALM_A));
335+
DrainAnnouncements();
336+
337+
Handle(PlayerJoinedMessage(9, WALLET_1, REALM_A));
338+
DrainAnnouncements();
339+
340+
// Stale leave for the superseded session - ignored (see IgnoreStalePlayerLeftForSupersededSession).
341+
Handle(new ServerMessage { PlayerLeft = new PlayerLeft { SubjectId = 7 } });
342+
343+
// The eventual leave for the CURRENT session (9) must still be processed normally.
344+
Handle(new ServerMessage { PlayerLeft = new PlayerLeft { SubjectId = 9 } });
345+
346+
using (OwnedBunch<RemoveIntention> bunch = removeIntentions.Bunch())
347+
CollectionAssert.AreEquivalent(new[] { new RemoveIntention(WALLET_1, RoomSource.Pulse) }, bunch.Collection());
348+
349+
Assert.IsFalse(peerIdCache.TryGetWallet(9, out _));
350+
}
351+
295352
private void Handle(ServerMessage serverMessage)
296353
{
297354
byte[] bytes = serverMessage.ToByteArray();

Explorer/Assets/DCL/Multiplayer/Profiles/BroadcastProfiles/LiveKitMessagesBroadcaster.cs

Lines changed: 57 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
using DCL.Multiplayer.Connections.Messaging.Pipe;
66
using DCL.Multiplayer.Connections.Pulse;
77
using DCL.Multiplayer.Connections.Rooms;
8+
using DCL.Web3;
89
using Google.Protobuf;
910
using LiveKit.Rooms;
1011
using System;
@@ -35,13 +36,36 @@ public class LiveKitMessagesBroadcaster
3536
/// </summary>
3637
private readonly PulseActivation pulseActivation;
3738

38-
private readonly Dictionary<string, RoomSource> announcedWallets = new ();
39+
private readonly PeerIdCache peerIdCache;
3940

40-
public LiveKitMessagesBroadcaster(IGateKeeperSceneRoom sceneRoom, IMessagePipesHub messagePipesHub, PulseActivation pulseActivation)
41+
private readonly Dictionary<string, (RoomSource rooms, Web3Address wallet)> announcedWallets = new ();
42+
43+
private readonly TimeSpan untargetedAnnounceInterval;
44+
private DateTime previousUntargetedAnnounce;
45+
46+
public LiveKitMessagesBroadcaster(IGateKeeperSceneRoom sceneRoom, IMessagePipesHub messagePipesHub, PulseActivation pulseActivation, PeerIdCache peerIdCache)
47+
: this(sceneRoom, messagePipesHub, pulseActivation, peerIdCache, TimeSpan.FromSeconds(10)) { }
48+
49+
public LiveKitMessagesBroadcaster(IGateKeeperSceneRoom sceneRoom, IMessagePipesHub messagePipesHub, PulseActivation pulseActivation, PeerIdCache peerIdCache, TimeSpan untargetedAnnounceInterval)
4150
{
4251
this.sceneRoom = sceneRoom;
4352
this.messagePipesHub = messagePipesHub;
4453
this.pulseActivation = pulseActivation;
54+
this.peerIdCache = peerIdCache;
55+
this.untargetedAnnounceInterval = untargetedAnnounceInterval;
56+
}
57+
58+
public void SendProfileAnnouncement<TInput, TMessage>(Action<TInput, TMessage> buildMessage, TInput args,
59+
LKDataPacketKind packetKind, CancellationToken ct) where TMessage: class, IMessage, new()
60+
{
61+
if (pulseActivation.IsActive && DateTime.UtcNow - previousUntargetedAnnounce < untargetedAnnounceInterval)
62+
{
63+
Send(buildMessage, args, packetKind, ct);
64+
return;
65+
}
66+
67+
previousUntargetedAnnounce = DateTime.UtcNow;
68+
SendUntargeted(buildMessage, args, packetKind, ct);
4569
}
4670

4771
public void Send<TInput, TMessage>(Action<TInput, TMessage> buildMessage, TInput args,
@@ -54,8 +78,11 @@ public void Send<TInput, TMessage>(Action<TInput, TMessage> buildMessage, TInput
5478
using PooledObject<List<string>> _ = ListPool<string>.Get(out List<string>? islandList);
5579
using PooledObject<List<string>> __ = ListPool<string>.Get(out List<string>? sceneList);
5680

57-
foreach ((string walletId, RoomSource rooms) in announcedWallets)
81+
foreach ((string walletId, (RoomSource rooms, Web3Address wallet)) in announcedWallets)
5882
{
83+
if (peerIdCache.TryGetPeerId(wallet, out uint _))
84+
continue;
85+
5986
if (EnumUtils.HasFlag(rooms, RoomSource.Island))
6087
islandList.Add(walletId);
6188

@@ -67,49 +94,57 @@ public void Send<TInput, TMessage>(Action<TInput, TMessage> buildMessage, TInput
6794
sceneList.Add(AUTH_SERVER_IDENTITY);
6895

6996
if (islandList.Count > 0)
70-
BuildMessageAndSend(messagePipesHub.IslandPipe(), islandList);
97+
BuildMessageAndSend(messagePipesHub.IslandPipe(), islandList, buildMessage, args, packetKind, ct);
7198

7299
if (sceneList.Count > 0)
73-
BuildMessageAndSend(messagePipesHub.ScenePipe(), sceneList);
100+
BuildMessageAndSend(messagePipesHub.ScenePipe(), sceneList, buildMessage, args, packetKind, ct);
74101
}
75102
else
76103
{
77104
// Broadcast as before
78-
BuildMessageAndSend(messagePipesHub.IslandPipe(), null);
79-
BuildMessageAndSend(messagePipesHub.ScenePipe(), null);
105+
SendUntargeted(buildMessage, args, packetKind, ct);
80106
}
107+
}
81108

82-
void BuildMessageAndSend(IMessagePipe messagePipe, IReadOnlyList<string>? recipients)
83-
{
84-
MessageWrap<TMessage> message = messagePipe.NewMessage<TMessage>();
85-
buildMessage(args, message.Payload);
109+
private void SendUntargeted<TInput, TMessage>(Action<TInput, TMessage> buildMessage, TInput args,
110+
LKDataPacketKind packetKind, CancellationToken ct) where TMessage: class, IMessage, new()
111+
{
112+
BuildMessageAndSend(messagePipesHub.IslandPipe(), null, buildMessage, args, packetKind, ct);
113+
BuildMessageAndSend(messagePipesHub.ScenePipe(), null, buildMessage, args, packetKind, ct);
114+
}
86115

87-
if (recipients != null)
88-
foreach (string recipient in recipients)
89-
message.AddSpecialRecipient(recipient);
116+
private void BuildMessageAndSend<TInput, TMessage>(IMessagePipe messagePipe, IReadOnlyList<string>? recipients,
117+
Action<TInput, TMessage> buildMessage, TInput args, LKDataPacketKind packetKind, CancellationToken ct) where TMessage: class, IMessage, new()
118+
{
119+
MessageWrap<TMessage> message = messagePipe.NewMessage<TMessage>();
120+
buildMessage(args, message.Payload);
90121

91-
message.SendAndDisposeAsync(ct, packetKind).Forget();
92-
}
122+
if (recipients != null)
123+
foreach (string recipient in recipients)
124+
message.AddSpecialRecipient(recipient);
125+
126+
message.SendAndDisposeAsync(ct, packetKind).Forget();
93127
}
94128

95129
public void Add(string walletId, RoomSource from)
96130
{
97-
if (announcedWallets.TryGetValue(walletId, out RoomSource source))
98-
from |= source;
99-
100-
announcedWallets[walletId] = from;
131+
if (announcedWallets.TryGetValue(walletId, out (RoomSource rooms, Web3Address wallet) entry))
132+
announcedWallets[walletId] = (entry.rooms | from, entry.wallet);
133+
else
134+
announcedWallets[walletId] = (from, new Web3Address(walletId));
101135
}
102136

103137
public void Remove(string walletId, RoomSource roomSource)
104138
{
105-
if (announcedWallets.TryGetValue(walletId, out RoomSource currentSource))
139+
if (announcedWallets.TryGetValue(walletId, out (RoomSource rooms, Web3Address wallet) entry))
106140
{
141+
RoomSource currentSource = entry.rooms;
107142
currentSource.RemoveFlag(roomSource);
108143

109144
if (currentSource == RoomSource.None)
110145
announcedWallets.Remove(walletId);
111146
else
112-
announcedWallets[walletId] = currentSource;
147+
announcedWallets[walletId] = (currentSource, entry.wallet);
113148
}
114149
}
115150
}

Explorer/Assets/DCL/Multiplayer/Profiles/BroadcastProfiles/LiveKitProfileBroadcast.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ async UniTaskVoid GetProfileVersionThenSendAsync(CancellationToken ct)
3333
{
3434
Profile? profile = await selfProfile.ProfileAsync(ct);
3535

36-
broadcaster.Send<Profile?, AnnounceProfileVersion>(static (p, version) => BuildMessage(p, version), profile, LKDataPacketKind.KindReliable, ct);
36+
broadcaster.SendProfileAnnouncement<Profile?, AnnounceProfileVersion>(static (p, version) => BuildMessage(p, version), profile, LKDataPacketKind.KindReliable, ct);
3737
}
3838

3939
GetProfileVersionThenSendAsync(cancellationTokenSource.Token).Forget();

Explorer/Assets/DCL/Multiplayer/Profiles/BroadcastProfiles/Tests.meta

Lines changed: 8 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"reference": "GUID:da80994a355e49d5b84f91c0a84a721f"
3+
}

0 commit comments

Comments
 (0)