feat: livekit cast improvements (#7347)

* feat: enrich getActiveVideoStreams response and auto-resolve audio for livekit user streams

Add participant metadata (name, speaking, trackName, width, height) to
GetActiveVideoStreamsResponse JSON output. Enable automatic audio track
resolution in LivekitPlayer when opening a user stream by pairing video
source to the correct audio source (camera→microphone, screenshare→screenshare audio).

* fix: resolve display name from participant metadata for castV2 support

Parse displayName from participant.Metadata JSON (used by unauthenticated
castV2 users), falling back to participant.Name then participant.Identity.

* feat: auto-recover livekit player when specific track dies

When a specific user stream track (livekit-video://identity/sid) is
unpublished, fallback to current-stream instead of retrying the same
dead address every frame. This lets EnsureVideoIsPlaying and
EnsureAudioIsPlaying self-heal by scanning for any available track.

* feat: play all audio tracks from livekit room in media player

Instead of pairing one audio track to the selected video track,
LivekitPlayer now plays every audio track in the room — so all
participants are heard regardless of which video is displayed.

* feat: auto-switch video to active speaker in CurrentStream mode

When LivekitPlayer plays in CurrentStream mode, the video now follows
the dominant speaker using room.ActiveSpeakers with a 1.5s debounce
to prevent flickering. UserStream mode remains pinned to the specified
participant. Also adds docs/cast.md documenting the full cast feature.

* fix: prevent audio disruption on video recovery, throttle empty-audio rescan, log metadata parse failures

- Add ReopenVideoStream() to recover video without tearing down audio
- Throttle empty-audio rescan to 2s interval to avoid per-frame lock
- Replace silent JsonException catch with ReportHub.LogWarning

* fix: make audio discovery additive to support late-joining participants

OpenAllAudioStreams now skips tracks that already have a source, so it
can be called periodically without disrupting healthy audio. Dead sources
are removed individually. EnsureVideoIsPlaying also drives audio discovery
since PBVideoPlayer entities don't trigger UpdateAudioStream.

* feat: expose LiveKit data messaging to scenes via CommsApi

Add PublishData, SubscribeToTopic, and ConsumeMessages methods to
CommsApiWrap, enabling scenes to send commands and receive state
updates through LiveKit data channels on arbitrary topics.

Includes rate limiting (10 msg/s per topic), 16KB payload cap,
thread-safe buffering for network-thread DataReceived events,
and proper cleanup on dispose.

* feat: expose LiveKit data messaging and metadata to scenes via CommsApi

Add PublishData, SubscribeToTopic, ConsumeMessages, and UpdateMetadata
methods to CommsApiWrap with corresponding JS module bindings, enabling
scenes to send commands, receive state updates, and set participant
roles through LiveKit data channels on arbitrary topics.

Includes rate limiting (10 msg/s per topic), 16KB payload cap,
thread-safe buffering for network-thread DataReceived events,
and proper cleanup on dispose.

Also handle non-protobuf data on the shared DataPipe gracefully in
MessagePipe by catching InvalidProtocolBufferException silently,
since JSON data from CommsApi messaging is now expected on the pipe.

* feat: prioritize presentation bot video track over active speaker

In CurrentStream mode, automatically lock onto a presentation bot's
video track when one is in the room. The bot is identified by the
identity prefix 'presentation-bot:'. Active speaker switching is
suppressed while a presentation bot is active, and resumes when the
bot leaves.

* docs: update cast.md with latest LiveKit features

Document presentation bot priority, audio rescan throttling,
video muted state tracking, resolution capping, data messaging
API, and wallet address identity format.

* refactor: LivekitPlayer review feedback — Weak.Null, hasLiveAudio, flip conditions

- Replace nullable Weak<IVideoStream>? with Weak<T>.Null struct pattern
- Rename tempStreamKeys to streamKeysBuffer
- Replace O(n) .Exists() with tracked hasLiveAudio boolean
- Flip negated if/else to positive condition first
- Clarify video-only audio discovery comment
- Remove unused IsVideoTrackMuted, videoTrackMuted, CheckVideoTrackMuted
- Mark class as main-thread only

* refactor: CommsApi review feedback — ConcurrentDictionary, ITypedArray, ctor subscription

- Subscribe to IDataPipe in constructor, make field readonly
- Replace Dictionary+List+bufferLock with ConcurrentDictionary+ConcurrentQueue
- PublishData accepts ITypedArray<byte> instead of string
- Use IJsOperations.LIVEKIT_MAX_SIZE for message size limit
- Name magic number as RATE_LIMIT_WINDOW_MS, remove unnecessary unchecked
- Simplify DisplayNameOf: remove unused metadata JSON parsing
- Flip ternary, extract computation to separate line
- Update MessagePipe catch comment with IMessagePipe migration TODO

* refactor: CommsApi.js — add type documentation, publishData passes Uint8Array

- Add parameter type comments for all exported functions
- publishData now passes Uint8Array directly to match ITypedArray<byte> C# signature

* fix: clarify EnsureAudioIsPlaying comment in LivekitPlayer

* feat: add gatekeeper mode dropdown to debug settings

Adds a GatekeeperMode selector (Org, Zone, Today, Localhost, Custom)
to DebugSettings, injecting --gatekeeper-url into app parameters.
The custom URL field is only shown when mode is set to Custom.

* feat: raise LiveKit video resolution cap to 2048x2048

* refactor: remove UpdateMetadata from CommsApi

Scene room metadata is managed internally by RemoteMetadata and would be
overwritten, making scene-set metadata unreliable.

* chore: reorder constants and remove dead code per coding standards

- Move private const fields before readonly fields in LivekitPlayer, UpdateMediaPlayerSystem, DebugSettings
- Remove unused FindPairedAudio() from LivekitPlayer (speculative code for hypothetical future)

* refactor: route CommsApi through ISceneCommunicationPipe with MsgType.CommsData

- Replace direct IDataPipe access with ISceneCommunicationPipe routing
- Add MsgType.CommsData = 3 for topic-based pub/sub data
- Wire format: [MsgType=3][topicLen 2B LE][topic UTF-8][data UTF-8]
- Move ISceneCommunicationPipe to CRDT.ECS.Bridge assembly (resolve circular dependency)
- Thread ISceneCommunicationPipe + ISceneData through RegisterAll/RegisterCommsApi
- PublishData now takes string instead of ITypedArray<byte>
- Update CommsApi.js to handle both string and Uint8Array from scenes

* fix: update MessagePipe catch comment and CommsApi.js publishData to string

- Update InvalidProtocolBufferException comment (cast2 webapp, not CommsApi)
- CommsApi.js publishData expects string data directly

* test: add unit tests for CommsApiWrap

- 12 tests covering: handler lifecycle, wire format, round-trip, topic buffering,
  input validation, rate limiting, malformed message handling

* fix: rename stale test method to match MsgType.CommsData

* chore: add missing meta file for CommsApiWrapShould test

* refactor: move gatekeeper URL override from app args to URL source layer

DebugSettings no longer injects --gatekeeper-url into AppParameters.
Instead, GatekeeperMode and CustomGatekeeperUrl are exposed as
properties and resolved in MainSceneLoader into a base URL override
that is passed to DecentralandUrlsSource. All composed gatekeeper
URLs (SceneAdapter, ChatAdapter, Status, BannedUsers, and
LocalGateKeeperSceneAdapter) now inherit the override, ensuring
consistent cross-environment debugging.

* fix: align CommsApiWrap tests with wire format and JSON escaping

- PublishDataWithMsgTypeCommsDataPrefix: decode topic header + data
  instead of treating bytes after MsgType as raw data
- SimulateIncomingMessage: encode wire format [topicLen][topic][data]
  so OnDataReceived can parse incoming messages
- DrainBufferOnConsume: check sender identities instead of inner JSON
  which gets escaped by JsonTextWriter
- PublishAndReceiveRoundTrip: relax assertion to account for
  JSON-escaped inner data string

* fix: address code review — race condition, stackalloc overflow, misleading sentinel

- Wrap ActiveSpeakers iteration in try-catch for thread safety (LivekitPlayer)
- Cap PublishData payload at LIVEKIT_MAX_SIZE before stackalloc (CommsApiWrap)
- Replace unused tuple sentinel with bool currentWritten (CommsApiWrap)
- Add RejectOversizedData test

* refactor: use LKParticipant type alias in GetActiveVideoStreamsResponse

* fix: remove stale #if guard on cross-platform media throttle fields

* fix: address remaining code review items — TOCTOU race, writer recovery, null-conditional

- Snapshot ActiveSpeakers into a local list before iterating to eliminate
  the TOCTOU race between Count and foreach (LivekitPlayer)
- Add ResetWriter() to recover JsonTextWriter state after mid-write
  exceptions corrupt its internal depth stack (CommsApiWrap)
- Simplify confusing !livekitPlayer?.IsVideoOpened ?? false expression
  to explicit null check (UpdateMediaPlayerSystem)

* refactor: address PR review — encapsulate video sourcing, dedup audio sources

- LivekitPlayer: introduce VideoEntity domain object + PresentationBotStream
  variant on LivekitAddress; replace currentVideoIdentity/string-prefix probes
  with exhaustive Match dispatch.
- Swap audioSources to Dictionary<StreamKey, ...> to collapse the O(N^2)
  HasAudioSourceForKey scan and drop the streamKeysBuffer field.
- Drop hasLiveAudio flag; isAudioOpened now reads audioSources.Count directly.
- Simplify TryFollowActiveSpeaker (foreach/break to index access; is { } pattern
  replaces null-forgiving operator).
- Collapse EnsureVideoIsPlaying else branch to a single CurrentStream reopen.
- CommsApiWrap: extract wire-format byte widths into named constants.
- LiveKitMediaExtensions: move PRESENTATION_BOT_IDENTITY_PREFIX out of
  LivekitPlayer; add IsPresentationBotIdentity extension.

* fix: address code review — clarify MessagePipe catch and ResetWriter XML doc

MessagePipe InvalidProtocolBufferException handler now reads as an
explicit defensive fallback for non-DCL participants publishing
non-protobuf data on the shared IDataPipe. DCL clients (cast2 included)
speak protobuf and CommsApi traffic routes via MsgType.CommsData, so
neither reaches this catch.

ResetWriter XML doc expanded to explain JsonTextWriter's internal
depth/token stack: an exception thrown between WriteStart*/WriteEnd*
leaves the stack unbalanced, corrupting subsequent writes. Recreating
on the exception path is cheaper than wrapping each Write* call in
try/catch on the happy path.

No behaviour change.

* refactor: address LiveKit review — AnyExposedAudioSource rename, fold VideoEntity into LivekitAddress

- Rename ExposedAudioSource → AnyExposedAudioSource across LivekitPlayer,
  MultiMediaPlayer and MediaPlayerComponent. The underlying storage moved
  from a single field to a Dictionary so iteration order is unspecified
  (and can reshuffle after audioSources.Remove), making "First" misleading.
  Restore the original inline comments on the method and expand the XMLDoc
  to call out the multi-source / non-deterministic semantics.
- Fix the "referene" typo in MultiMediaPlayer's XMLDoc while there.
- Eliminate VideoEntity: its IsPresentationBot and Identity accessors and
  the FromIdentity factory move directly onto LivekitAddress, which
  already discriminates over the UserStream / PresentationBotStream /
  CurrentStream union. Migrate all 6 call sites in LivekitPlayer.

* clarify #7347 (comment)

* address https://github.com/decentraland/unity-explorer/pull/7347\#discussion_r3132329943

* register new category for reporting

* address https://github.com/decentraland/unity-explorer/pull/7347\#discussion_r3132469058

* allocation free write, address https://github.com/decentraland/unity-explorer/pull/7347\#discussion_r3132270763 and https://github.com/decentraland/unity-explorer/pull/7347\#discussion_r3132314939

* submit todo for futher optimisations

* remove redundant const

* implement message count bouding, add explicit comments to js contract, address https://github.com/decentraland/unity-explorer/pull/7347\#discussion_r3132503328 and https://github.com/decentraland/unity-explorer/pull/7347\#discussion_r3132457159

* implement ListPool

* link livekit package

* mitigate InvalidOperationException and ActiveSpeakers thread-safe access

* remove redundant locks, simplify "switch to current" method

* update package lock

* unify OpenVideoStream

* address LivekitPlayer

* remove redundant code

* remove redundant null check

* update comment

* align livekit deps

* chore: Add id-token permission

Co-authored-by: Copilot <copilot@github.com>

* fix: remove added prop

---------

Co-authored-by: NickKhalow <nickhalow@gmail.com>
Co-authored-by: Copilot <copilot@github.com>

Author: 3 people

…ommunications/ISceneCommunicationPipe.cs …CrdtEcsBridge/ISceneCommunicationPipe.csExplorer/Assets/DCL/Infrastructure/CrdtEcsBridge/JsModulesImplementation/Communications/ISceneCommunicationPipe.cs renamed to Explorer/Assets/DCL/Infrastructure/CrdtEcsBridge/ISceneCommunicationPipe.cs Explorer/Assets/DCL/Infrastructure/CrdtEcsBridge/JsModulesImplementation/Communications/ISceneCommunicationPipe.cs renamed to Explorer/Assets/DCL/Infrastructure/CrdtEcsBridge/ISceneCommunicationPipe.cs

@@ -1,6 +1,4 @@
-using DCL.Multiplayer.Connections.Messaging;
-using Decentraland.Kernel.Comms.Rfc4;
-using System;
+using System;
 using System.Threading;
 
 namespace CrdtEcsBridge.JsModulesImplementation.Communications
@@ -11,6 +9,7 @@ public enum MsgType
         {
             String = 1, // SDK scenes MessageBus messages
             Uint8Array = 2,
+            CommsData = 3, // CommsApi publish/subscribe topic-based data
         }
 
         public enum ConnectivityAssertiveness

…ications/ISceneCommunicationPipe.cs.meta …csBridge/ISceneCommunicationPipe.cs.metaExplorer/Assets/DCL/Infrastructure/CrdtEcsBridge/JsModulesImplementation/Communications/ISceneCommunicationPipe.cs.meta renamed to Explorer/Assets/DCL/Infrastructure/CrdtEcsBridge/ISceneCommunicationPipe.cs.meta Explorer/Assets/DCL/Infrastructure/CrdtEcsBridge/JsModulesImplementation/Communications/ISceneCommunicationPipe.cs.meta renamed to Explorer/Assets/DCL/Infrastructure/CrdtEcsBridge/ISceneCommunicationPipe.cs.meta

@@ -5,6 +5,15 @@
 
 namespace Global.Dynamic.DebugSettings
 {
+    public enum GatekeeperMode
+    {
+        Org = 0,
+        Zone = 1,
+        Today = 2,
+        Localhost = 3,
+        Custom = 4,
+    }
+
     [Serializable]
     public class DebugSettings
     {
@@ -34,6 +43,13 @@ [SerializeField] [Tooltip("Make sure the ENS put here will be loaded as a Global
         private bool overrideConnectionQuality;
         [SerializeField]
         private LKConnectionQuality connectionQuality;
+        [Space]
+        [Header("Comms Gatekeeper")]
+        [SerializeField]
+        private GatekeeperMode gatekeeperMode;
+        [SerializeField] [Tooltip("Base gatekeeper URL used only when Gatekeeper Mode is set to Custom (e.g. http://localhost:3000)")]
+        private string customGatekeeperUrl = string.Empty;
+        [Space]
         [SerializeField]
         private string[] appParameters;
 
@@ -52,6 +68,8 @@ public static DebugSettings Release() =>
                 connectionQuality = LKConnectionQuality.QualityExcellent,
                 enableRemotePortableExperiences = true,
                 emotesToAddToUserProfile = null,
+                gatekeeperMode = GatekeeperMode.Org,
+                customGatekeeperUrl = string.Empty,
                 appParameters = Array.Empty<string>(),
             };
 
@@ -68,6 +86,8 @@ public static DebugSettings Release() =>
         public bool EnableEmulateNoLivekitConnection => Application.isEditor? this.enableEmulateNoLivekitConnection : RELEASE_SETTINGS.enableEmulateNoLivekitConnection;
         public bool OverrideConnectionQuality => Application.isEditor ? this.overrideConnectionQuality : RELEASE_SETTINGS.overrideConnectionQuality;
         public LKConnectionQuality ConnectionQuality => Application.isEditor ? this.connectionQuality : RELEASE_SETTINGS.connectionQuality;
-        public string[] AppParameters => appParameters;
+        public GatekeeperMode GatekeeperMode => Application.isEditor ? gatekeeperMode : RELEASE_SETTINGS.gatekeeperMode;
+        public string CustomGatekeeperUrl => Application.isEditor ? customGatekeeperUrl : RELEASE_SETTINGS.customGatekeeperUrl;
+        public string[] AppParameters => Application.isEditor ? appParameters : RELEASE_SETTINGS.appParameters;
     }
 }

Explorer/Assets/DCL/Infrastructure/Global/Dynamic/DebugSettings/DebugSettings.cs

@@ -45,6 +45,7 @@
 using ECS.StreamableLoading.Common.Components;
 using Newtonsoft.Json.Linq;
 using Global.AppArgs;
+using Global.Dynamic.DebugSettings;
 using Global.Dynamic.RealmUrl;
 using Global.Dynamic.RealmUrl.Names;
 using Global.Versioning;
@@ -187,7 +188,9 @@ private async UniTask InitializeFlowAsync(CancellationToken ct)
             World world = World.Create();
 
             var realmData = new RealmData();
-            var decentralandUrlsSource = new GatewayUrlsSource(decentralandEnvironment, realmData, launchSettings);
+            string? gatekeeperBaseOverride = ResolveGatekeeperBaseOverride(debugSettings.GatekeeperMode, debugSettings.CustomGatekeeperUrl);
+            ReportHub.Log(ReportCategory.STARTUP, $"Gatekeeper mode: {debugSettings.GatekeeperMode}, base override: {gatekeeperBaseOverride ?? "(default)"}");
+            var decentralandUrlsSource = new GatewayUrlsSource(decentralandEnvironment, realmData, launchSettings, gatekeeperBaseOverride);
             DiagnosticInfoUtils.LogEnvironment(decentralandUrlsSource);
 
             var assetsProvisioner = new AddressablesProvisioner();
@@ -802,6 +805,17 @@ public void Dispose()
             }
         }
 
+        private static string? ResolveGatekeeperBaseOverride(GatekeeperMode mode, string customUrl) =>
+            mode switch
+            {
+                GatekeeperMode.Org => null,
+                GatekeeperMode.Zone => "https://comms-gatekeeper.decentraland.zone",
+                GatekeeperMode.Today => "https://comms-gatekeeper.decentraland.today",
+                GatekeeperMode.Localhost => "http://localhost:3000",
+                GatekeeperMode.Custom => string.IsNullOrEmpty(customUrl) ? null : customUrl,
+                _ => throw new ArgumentOutOfRangeException(nameof(mode), mode, null),
+            };
+
         [Serializable]
         public class SplashScreenRef : ComponentReference<SplashScreen>
         {

Explorer/Assets/DCL/Infrastructure/Global/Dynamic/MainSceneLoader.cs

@@ -5,6 +5,7 @@
 using System.IO;
 using UnityEditor;
 using UnityEngine;
+using UnityEngine.Assertions;
 
 namespace Global.Editor
 {
@@ -157,11 +158,15 @@ private static Rect DrawDefaultProperties(Rect position, SerializedProperty prop
             SerializedProperty endProperty = property.GetEndProperty();
 
             bool enterChildren = true;
+            bool isCustomGatekeeper = IsCustomGatekeeperMode(property);
 
             while (iterator.NextVisible(enterChildren) && !SerializedProperty.EqualContents(iterator, endProperty))
             {
                 enterChildren = false;
 
+                if (iterator.name == "customGatekeeperUrl" && !isCustomGatekeeper)
+                    continue;
+
                 float propertyHeight = EditorGUI.GetPropertyHeight(iterator, true);
                 position.height = propertyHeight;
                 EditorGUI.PropertyField(position, iterator, true);
@@ -178,16 +183,28 @@ private static float GetDefaultPropertiesHeight(SerializedProperty property)
             SerializedProperty endProperty = property.GetEndProperty();
 
             bool enterChildren = true;
+            bool isCustomGatekeeper = IsCustomGatekeeperMode(property);
 
             while (iterator.NextVisible(enterChildren) && !SerializedProperty.EqualContents(iterator, endProperty))
             {
                 enterChildren = false;
+
+                if (iterator.name == "customGatekeeperUrl" && !isCustomGatekeeper)
+                    continue;
+
                 height += EditorGUI.GetPropertyHeight(iterator, true) + EditorGUIUtility.standardVerticalSpacing;
             }
 
             return height;
         }
 
+        private static bool IsCustomGatekeeperMode(SerializedProperty property)
+        {
+            SerializedProperty gatekeeperMode = property.FindPropertyRelative("gatekeeperMode");
+            Assert.IsNotNull(gatekeeperMode, "Failed to find 'gatekeeperMode' property.");
+            return gatekeeperMode.intValue == (int)GatekeeperMode.Custom;
+        }
+
         private static string? FindCreatorHubPath()
         {
             // Check default path

Explorer/Assets/DCL/Infrastructure/Global/Editor/DebugSettingsDrawer.cs

@@ -243,7 +243,8 @@ private async UniTask<ISceneFacade> CreateSceneAsync(ISceneData sceneData, IJsAp
                     sceneData,
                     realmData,
                     portableExperiencesController,
-                    remoteMetadata
+                    remoteMetadata,
+                    messagePipesHub
                 );
             }
             else
@@ -271,7 +272,8 @@ private async UniTask<ISceneFacade> CreateSceneAsync(ISceneData sceneData, IJsAp
                     sceneData,
                     realmData,
                     portableExperiencesController,
-                    remoteMetadata
+                    remoteMetadata,
+                    messagePipesHub
                 );
             }