feat: peer clustering over the AoI grid with a NATS feed

.claude/skills/run-test-client/SKILL.md

@@ -3,7 +3,7 @@ name: run-test-client
 description: Launch DCLPulseTestClient bot(s) against a Pulse server. Use when the user wants to run, start, or launch the test client / bot / load test.
 user-invocable: true
 allowed-tools: Bash
-argument-hint: [--account=name] [--bot-count=N] [--ip=address] [--port=port] [--pos-x=X] [--pos-y=Y] [--pos-z=Z] [--rotate-speed=deg]
+argument-hint: [--account=name] [--bot-count=N] [--ip=address] [--port=port] [--pos-x=X] [--pos-y=Y] [--pos-z=Z] [--rotate-speed=deg] [--comms-enabled]
 ---
 
 # Launch DCLPulseTestClient
@@ -39,6 +39,24 @@ If no arguments are provided, use the defaults (account `enetclient-test`, 1 bot
 | `--dispersion-radius=<float>` | `20` | Max wander distance from spawn origin |
 | `--rotate-speed=<deg/s>` | `90` | Idle rotation speed |
 
+### Conn-string harness arguments
+
+Off by default — omit these and the run behaves exactly as before.
+
+| Argument | Default | Description |
+|---|---|---|
+| `--comms-enabled` | off | Each bot also opens a ws-connector session on its own wallet and records the LiveKit conn strings it receives |
+| `--comms-url=<url>` | `ws://127.0.0.1:5000/ws` | ws-connector endpoint. Also accepts a realm's raw adapter string (`archipelago:archipelago:wss://host/ws`); anything that isn't `ws://`/`wss://` after refinement is rejected loudly |
+| `--expect-conn-string-within=<seconds>` | `15` | **Parsed but not yet acted on.** Reserved for the regression scenarios; passing it today changes nothing |
+
+**Argument parsing is `--name=value` only.** `ClientOptions.FromArgs` matches on the `--name=` prefix, so a space-separated `--comms-url ws://…` sets nothing and silently leaves the default in place. `--comms-enabled` is the sole exception — bare or `=true` both work.
+
+## The client does not mint conn strings
+
+The test client is a client: Pulse over ENet/WebTransport, ws-connector over WebSocket, and no broker connection at all. It has **no `--nats-url` and no bridge mode** — an earlier revision had a stub gatekeeper behind `--mode=bridge` and it was removed on purpose.
+
+So `--comms-enabled` on its own produces a bot that connects to ws-connector and then receives nothing, which looks exactly like a healthy idle run. Something has to translate `peer.{addr}.cluster_change` into `engine.peer.{addr}.island_changed`, and that something is **comms-gatekeeper**, run separately against the same broker with `CLUSTER_SUBSCRIBER_ENABLED=true`. It needs Postgres and a LiveKit host/key/secret. See `docs/e2e-livekit.md` section 4.
+
 ## Multi-bot mode
 
 When `--bot-count` > 1:
@@ -69,3 +87,6 @@ When the user mentions a location by name, translate to position flags:
 - **"ENet library failed to initialize"** — the ENet native library is missing. On macOS: `brew install enet`. On Windows: ensure `enet.dll` is in the output directory.
 - **Handshake failed** — the server rejected the auth chain. Check that the server is running and the account's ephemeral key hasn't expired (25h lifetime). Try `metaforge account remove <name>` then re-run to create a fresh account.
 - **Connection timeout** — verify the server IP/port and that UDP traffic is not blocked by a firewall.
+- **`--comms-enabled` set but no `[ws-connector]` lines** — ws-connector isn't up at `--comms-url`, or the flag was passed space-separated. A comms failure is deliberately non-fatal to the Pulse session and reports on the `[comms]` prefix, so the run otherwise looks healthy.
+- **Bot connects to ws-connector but no island ever arrives** — nothing is minting conn strings. Start comms-gatekeeper against the same broker. `docs/e2e-livekit.md` covers the rest of the silent-no-delivery causes.
+- **Second bot on the same account kicks the first** — ws-connector allows one session per wallet and kicks the previous with `KR_NEW_SESSION`. Two bots need two accounts, which `--bot-count` > 1 already gives.

.claude/skills/stop-test-client/SKILL.md

@@ -32,3 +32,9 @@ Gracefully stop running test bots so they disconnect cleanly from the server.
    tasklist | grep -i DCLPulseTestClient && taskkill //F //IM DCLPulseTestClient.exe 2>/dev/null || echo "Bots stopped gracefully."
    ```
    On macOS/Linux fallback: `pkill -9 -f DCLPulseTestClient`
+
+## Scope
+
+The stop file reaches every test-client process, because the binary now only ever runs bots — there is no second entry point. A bot started with `--comms-enabled` also holds a ws-connector session; it closes with the rest of the shutdown and needs no separate step.
+
+Nothing here touches comms-gatekeeper, ws-connector, NATS or Pulse. Those are separate services with their own lifecycles (`docker compose -f docker-compose.e2e.yml down` for the compose stack). Stopping the bots leaves them running, which is usually what you want between runs.

.claude/skills/validate-e2e/SKILL.md

@@ -3,7 +3,7 @@ name: validate-e2e
 description: Validate the full client↔server networking flow end-to-end — QUIC/UDP connect, DCL ECDSA handshake, movement, interest-managed state fan-out, and resync — by running two DCLPulseTestClient bots against a live server and cross-checking server logs, client logs, and per-transport Prometheus metrics. Covers ENet and WebTransport; WebTransport needs the extra cert/bind setup documented here. Use when asked to validate, verify, or smoke-test the end-to-end flow, especially over WebTransport.
 user-invocable: true
 allowed-tools: Bash, PowerShell, Grep, Read
-argument-hint: [--transport=enet|webtransport] [--account=prefix]
+argument-hint: [--transport=enet|webtransport] [--account=prefix] [--comms-enabled]
 ---
 
 # Validate the e2e networking flow
@@ -26,6 +26,14 @@ Run **two** bots that connect, authenticate, move, and observe each other, then
 
 A `Seq gap → resync` line is **expected, not a failure**: unreliable datagrams drop/reorder, the client detects the gap and asks for a `STATE_FULL`. Seeing it means the datagram path *and* the reliable resync path both work (the "client drives resync" model — see CLAUDE.md).
 
+## Scope: this skill does not cover the conn-string path
+
+What is validated here is the Pulse protocol over ENet/WebTransport. The LiveKit **conn-string** path — Pulse clustering → NATS `peer.{addr}.cluster_change` → gatekeeper → NATS `engine.peer.{addr}.island_changed` → ws-connector → client — is a **separate channel with its own harness**, and none of the success criteria below exercise it.
+
+To bring it into a run, add `--comms-enabled` to the bot command **and** run comms-gatekeeper against the same broker — the test client holds no broker connection and cannot mint conn strings itself, so without gatekeeper the bots connect to ws-connector and receive nothing. Each bot then logs `[ws-connector] Island …` lines; failures on that channel report on the `[comms]` prefix and are deliberately **non-fatal** to the Pulse session, so a passing run here says nothing about the conn-string path either way.
+
+That harness has its own prerequisites (NATS, ws-connector, comms-gatekeeper with Postgres and LiveKit credentials, and a `metaforge` new enough to have `account sign`), its own compose file (`docker-compose.e2e.yml`), and its own failure taxonomy — silent no-delivery rather than the visible errors this skill deals in. **See [docs/e2e-livekit.md](../../../docs/e2e-livekit.md).** Do not fold its assertions into the criteria below; the two validations have different prerequisites and should fail independently.
+
 ## Prerequisites
 
 - Build first: `DOTNET_ROOT="$HOME/.dotnet" PATH="$HOME/.dotnet:$PATH" dotnet build src/DCLPulse/DCLPulse.sln -p:GenerateProto=false`

CLAUDE.md

@@ -212,7 +212,7 @@ Standard protobuf `optional` fields provide per-field presence natively — unch
 
 **Protobuf `optional` fields carry delta presence.** The schema expresses intent with `optional`; standard protobuf field presence keeps unchanged fields off the wire. No plugin-generated mask is involved — the plugin only adds the quantized accessors and their step constants.
 
-**Snapshot publishing goes through `PeerSnapshotPublisher`.** Every handler that mutates peer state (`PlayerStateInputHandler`, `EmoteStartHandler`, `TeleportHandler`, the handshake initial-state seed) calls one of two methods on the publisher: `PublishFromPlayerState(from, state, EmoteInput?)` for `PlayerState`-shaped events, or `PublishTeleport(from, teleportRequest)` for teleports (reading the quantized position codes off the request). The publisher owns Seq numbering (`LastSeq + 1`), parcel→global decoding, head-IK lifting from `PlayerState`, the `SnapshotBoard.Publish` + `SpatialGrid.Set` pair, and emote-ledger bookkeeping (`StartSeq` is stamped to the new snapshot's `Seq`, `StartTick` defaults to `ServerTick` when caller leaves it null). Don't reconstruct a `PeerSnapshot` inline in a handler — add it to the publisher.
+**Snapshot publishing goes through `PeerSnapshotPublisher`.** Every handler that mutates peer state (`PlayerStateInputHandler`, `EmoteStartHandler`, `TeleportHandler`, the handshake initial-state seed) calls one of two methods on the publisher: `PublishFromPlayerState(from, state, EmoteInput?)` for `PlayerState`-shaped events, or `PublishTeleport(from, teleportRequest)` for teleports (reading the quantized position codes off the request). The publisher owns Seq numbering (`LastSeq + 1`), parcel→global decoding, head-IK lifting from `PlayerState`, the `SnapshotBoard.Publish` + `RealmSpatialGrids.Set` pair (and, on a realm-changing teleport, the `RealmSpatialGrids.Remove` that must precede the publish so the peer is never indexed in one realm under a snapshot naming another), and emote-ledger bookkeeping (`StartSeq` is stamped to the new snapshot's `Seq`, `StartTick` defaults to `ServerTick` when caller leaves it null). Don't reconstruct a `PeerSnapshot` inline in a handler — add it to the publisher.
 
 `EmoteInput(EmoteId, DurationMs?, StartTick?)` is the caller-facing emote-start descriptor. Callers pass only what's semantically theirs (the emote identity, its duration, optionally a backdated start tick for reconnect resume); ledger fields like `StartSeq` are not part of the API. `EmoteStart` callers omit `StartTick` (defaults to "started right now"); the handshake reconnect path passes a backdated `StartTick` so observers scrub forward by the elapsed-since-real-start delta.
 
@@ -401,6 +401,12 @@ MetaForge/
 - The solution file is `src/DCLPulse/DCLPulse.sln` — always pass it explicitly since it's not in the repo root.
 - Use `-p:GenerateProto=false` unless the user explicitly asks to regenerate proto files.
 - To run tests: `DOTNET_ROOT="$HOME/.dotnet" PATH="$HOME/.dotnet:$PATH" dotnet test src/DCLPulse/DCLPulse.sln -p:GenerateProto=false`
+- **Benchmarks** live in `src/DCLPulseBenchmarks` (BenchmarkDotNet). `Program.cs` uses `BenchmarkSwitcher`, so every `[Benchmark]` class in the assembly is selectable from the command line — never edit it to choose a suite. Always `-c Release`:
+  ```bash
+  dotnet run -c Release --project src/DCLPulseBenchmarks -p:GenerateProto=false -- --list flat
+  dotnet run -c Release --project src/DCLPulseBenchmarks -p:GenerateProto=false -- --filter '*ClusterTracker*'
+  ```
+  `DCLPulse.csproj` grants `InternalsVisibleTo` to the benchmarks project, so `internal` entry points (e.g. `ClusterTracker.RunPass`) are callable. Prefer adding a benchmark class here over a throwaway harness in the test project — a measurement nobody can re-run is a measurement nobody will trust. When a benchmark disproves an optimization, record that in the class docs so it is not retried blind (see `ClusterTrackerBenchmarks`).
 - `dotnet restore` auto-fetches `Decentraland.RustEthereum.<version>.nupkg` into the gitignored `packages/` local NuGet source via `src/Directory.Build.targets`. Bump `RustEthereumVersion` in `src/Directory.Build.props` and the next restore pulls the new version from the GitHub Release. The underlying script is `tools/fetch-rust-eth.{sh,ps1}`.
 - **If you touched anything in the restore/build pipeline** — csprojs (especially `<PackageReference>` or `<ProjectReference>`), `src/Directory.Build.{props,targets}`, `src/NuGet.config`, `tools/fetch-rust-eth.{sh,ps1}`, or the `.gitignore` rules around `packages/` — also build the Docker images to catch layered-COPY misses that `dotnet build` won't surface:
   ```bash

README.md

@@ -93,7 +93,7 @@ Pulse uses ENet over UDP. A couple of non-obvious behaviors worth knowing before
 
 Two knobs control concurrent-peer capacity:
 
-- `Transport.MaxPeers` — size of the `PeerIndex` pool and every per-peer array board (`SnapshotBoard`, `IdentityBoard`, `ProfileBoard`, `SpatialGrid`). Hard ceiling on active + in-grace slots.
+- `Transport.MaxPeers` — size of the `PeerIndex` pool and every per-peer array board (`SnapshotBoard`, `IdentityBoard`, `ProfileBoard`, `RealmSpatialGrids`). Hard ceiling on active + in-grace slots.
 - `Transport.MaxConcurrentConnections` — ENet host capacity. `0` = `MaxPeers`. Set below `MaxPeers` to reserve slots for the allocator's pending-recycle grace window — without headroom, a burst of reconnects can exhaust the `PeerIndex` pool while ENet still has free slots, causing `SERVER_FULL` refusals on otherwise admittable connections.
 
 Rule of thumb: `MaxConcurrentConnections ≈ MaxPeers - ceil(peakDisconnectsPerSecond × Peers.DisconnectionCleanTimeoutMs / 1000)`.
@@ -148,6 +148,16 @@ dotnet run --project src/DCLPulseTestClient -- --account=loadtest --bot-count=10
 
 When `--bot-count=1`, the account name is used as-is. When `--bot-count` > 1, accounts are named `<account>-0`, `<account>-1`, ..., `<account>-N-1` and bots spawn in a circle around the initial position.
 
+### End-to-end conn-string harness
+
+With `--comms-enabled` each bot also opens a ws-connector session on its own wallet, so one process holds both channels and can assert that a LiveKit connection string arrives for the same identity that Pulse clustered. `docker-compose.e2e.yml` brings up the NATS + ws-connector + Pulse stack it runs against; comms-gatekeeper mints the conn strings and is run separately, since it needs Postgres and LiveKit credentials.
+
+The test client stays a client — it holds no broker connection and never publishes on a service's subject.
+
+`--comms-url` accepts either a plain `ws://host/ws` or a realm's raw comms adapter string (`archipelago:archipelago:wss://host/ws`), so a value copied out of `/about` works unchanged.
+
+See [docs/e2e-livekit.md](docs/e2e-livekit.md) for prerequisites, the bridge modes, and how to read a failure.
+
 ### What the bot does
 
 On startup each bot authenticates via MetaForge, connects over ENet, completes the handshake, announces its profile, then enters a 30 fps simulation loop.

docker-compose.debug.yml

@@ -11,10 +11,28 @@ services:
       - ./src:/app/src
     environment:
       DOTNET_ENVIRONMENT: Development
+      # Applies to Pulse's own categories only — the NATS client stays at the Warning floor
+      # appsettings sets for NATS.Client.Core. Below that it re-dumps the server info on every
+      # reconnect, which a flapping broker turns into a stream through the console dashboard.
       Logging__LogLevel__Default: Debug
       Peers__ResyncWithDelta: false
       WebTransport__Enabled: true
+      # Off by default in appsettings; the debug stack turns clustering on and points it at the
+      # sidecar broker. Drop Nats__Url for stats-only mode (tracker runs, nothing published).
+      Clusters__Enabled: true
+      Nats__Url: nats://nats:4222
+    depends_on:
+      - nats
     cap_add:
       - SYS_PTRACE
     security_opt:
       - seccomp:unconfined
+
+  nats:
+    image: nats:2-alpine
+    container_name: dcl-pulse-nats
+    ports:
+      # 4222 client, 8222 HTTP monitoring (/varz, /connz) for confirming the publisher connected.
+      - "4222:4222"
+      - "8222:8222"
+    command: ["--http_port", "8222"]