decentraland
diff --git a/‎.claude/skills/add-metric/SKILL.md‎
Lines changed: 98 additions & 0 deletions b/‎.claude/skills/add-metric/SKILL.md‎
Lines changed: 98 additions & 0 deletions
diff --git a/‎Dockerfile.debug‎
Lines changed: 11 additions & 0 deletions b/‎Dockerfile.debug‎
Lines changed: 11 additions & 0 deletions
diff --git a/‎docker-compose.debug.yml‎
Lines changed: 1 addition & 0 deletions b/‎docker-compose.debug.yml‎
Lines changed: 1 addition & 0 deletions
diff --git a/‎docs/metrics.md‎
Lines changed: 81 additions & 1 deletion b/‎docs/metrics.md‎
Lines changed: 81 additions & 1 deletion
diff --git a/‎src/DCLPulse/DCLPulse.csproj‎
Lines changed: 30 additions & 0 deletions b/‎src/DCLPulse/DCLPulse.csproj‎
Lines changed: 30 additions & 0 deletions
diff --git a/‎src/DCLPulse/Dockerfile‎
Lines changed: 11 additions & 0 deletions b/‎src/DCLPulse/Dockerfile‎
Lines changed: 11 additions & 0 deletions
diff --git a/‎src/DCLPulse/Dockerfile.dev-debug‎
Lines changed: 11 additions & 0 deletions b/‎src/DCLPulse/Dockerfile.dev-debug‎
Lines changed: 11 additions & 0 deletions
@@ -261,6 +261,104 @@ per-type `RateTracker` dictionary and iterate in `TryConsumeSnapshot`.
 
 ---
 
+## Pattern D: Histogram metric
+
+For latency/duration values where the **distribution** matters, not just a count or a rate.
+Buckets are accumulated lock-free on the hot path; the dashboard shows value percentiles
+(window + lifetime) and Prometheus exposes native `_bucket`/`_sum`/`_count` series.
+
+**Examples:** delta staleness per AoI tier, tick duration, outgoing drain-cycle duration.
+
+**Reference implementation:** `delta_staleness` — trace `pulse.sim.delta_staleness_tier0_ms`
+end-to-end across the files below.
+
+### 1. Declare the instrument — `src/DCLPulse/Metrics/PulseMetrics.*.cs`
+
+```csharp
+// PulseMetrics.Simulation.cs
+public static readonly long[] MY_BUCKETS_MS = [1, 2, 4, 8, 16, 32, 64, 128, 256, 512];
+
+public static readonly Histogram<long> MY_LATENCY_MS =
+    METER.CreateHistogram<long>("pulse.sim.my_latency_ms");
+```
+
+Pick bucket bounds dense around the SLO region you care about. `Histogram<long>`, values in
+whole ms or µs (the collector keys on the instrument name, so units live only in the name).
+
+### 2. Record at the source
+
+```csharp
+PulseMetrics.Simulation.MY_LATENCY_MS.Record(elapsedMs);
+```
+
+Record on the measuring thread. Wrap-safe unsigned subtraction for monotonic-clock deltas
+(see `RecordDeltaStaleness` in `PeerSimulation.cs`) — don't cast to signed before subtracting.
+
+### 3. Accumulate in the collector — `src/DCLPulse/Metrics/MeterListenerMetricsCollector.cs`
+
+- Add a `BucketHistogram` field seeded with the bucket bounds:
+  ```csharp
+  private readonly BucketHistogram myLatency = new (PulseMetrics.Simulation.MY_BUCKETS_MS);
+  ```
+- Route the instrument in `OnLongMeasurement`:
+  ```csharp
+  case "pulse.sim.my_latency_ms":
+      myLatency.Record(value);
+      break;
+  ```
+  `BucketHistogram.Record` is lock-free (`Interlocked` per bucket) — no extra locking needed.
+
+### 4. Expose on the snapshot — `src/DCLPulse/Metrics/MetricsSnapshot.cs`
+
+```csharp
+public HistogramSnapshot MyLatencyMs { get; init; }
+```
+`HistogramSnapshot` (not `RateStats`) — an immutable per-bucket copy with a
+`Percentile(p)` helper. Percentiles are computed downstream.
+
+### 5. Populate the snapshot — `MeterListenerMetricsCollector.TakeSnapshot`
+
+```csharp
+MyLatencyMs = myLatency.Snapshot(),
+```
+
+### 6. Emit Prometheus — `PrometheusFormatter.cs`
+
+Native histogram exposition — header once, then one series per label set:
+```csharp
+WriteHistogramHeader(writer, "dcl_pulse_my_latency_ms", "My latency in ms");
+WriteHistogramSeries(writer, "dcl_pulse_my_latency_ms", snap.Simulation.MyLatencyMs, labels: null);
+```
+Emits `_bucket{le="…"}`, `_sum`, `_count`. Consumers run `histogram_quantile()`. Pass a
+`tier="0"`-style label string for per-variant series (see the `delta_staleness` calls).
+
+### 7. Display on the console dashboard — `ConsoleDashboard.cs`
+
+Use `HistogramTracker` (not `RateTracker`) — it adapts the cumulative `HistogramSnapshot`
+into `RateStats` where PerSec = recordings/s, Window = value percentiles over the delta
+since the previous snapshot, Lifetime = value percentiles over the cumulative buckets:
+```csharp
+private readonly HistogramTracker myLatencyTracker = new ();
+private readonly RateStatsView   myLatency = new ();
+private readonly Sparkline       myLatencySparkline = new (Enumerable.Repeat(0.0, SPARKLINE_MAX_SAMPLES));
+```
+In `TryConsumeSnapshot` — note the sparkline plots the **window P99** (the tail), not the rate:
+```csharp
+RateStats myLatencyStats = myLatencyTracker.Update(snap.Simulation.MyLatencyMs, elapsed);
+myLatency.Apply(myLatencyStats, v => v.ToString("N0"));
+ShiftSample(myLatencySparkline.Values, myLatencyStats.Window.P99);
+```
+Add a `RateStatsRow` to the `Latency` group in `BuildVisualTree`. The percentile columns then
+read as value distribution (ms/µs), not rate percentiles.
+
+### 8. Document in `docs/metrics.md`
+
+Add under `## Latency metrics`. Note the value-distribution semantics of the percentile columns,
+the expected range, the Prometheus `histogram_quantile()` guidance, and any exclusions (e.g.
+resync-path deltas are excluded from `delta_staleness`).
+
+---
+
 ## Supporting types reference
 
 | Type | Location | Purpose |
 
@@ -4,6 +4,17 @@ RUN apt-get update && apt-get install -y curl unzip procps \
     && curl -sSL https://aka.ms/getvsdbgsh | bash /dev/stdin -v latest -l /vsdbg \
     && rm -rf /var/lib/apt/lists/*
 COPY . .
+# Geo database — single fetch at image build; the app never downloads at runtime. Refreshes
+# with every image build. IP-range CSVs: geo-whois-asn-country (CC0,
+# https://github.com/sapics/ip-location-db). Country → continent mapping: GeoNames
+# countryInfo.txt (CC-BY 4.0, https://www.geonames.org).
+ADD https://cdn.jsdelivr.net/npm/@ip-location-db/geo-whois-asn-country/geo-whois-asn-country-ipv4-num.csv \
+    https://cdn.jsdelivr.net/npm/@ip-location-db/geo-whois-asn-country/geo-whois-asn-country-ipv6-num.csv \
+    https://download.geonames.org/export/dump/countryInfo.txt \
+    /app/geodb/
+# The image already has fresh geo CSVs from the ADD above; the entrypoint's `dotnet run`
+# would otherwise trigger the csproj predownload at container start, so disable it.
+ENV FetchGeoDb=false
 EXPOSE 7777/udp
 EXPOSE 7743/udp
 ENTRYPOINT ["dotnet", "run", "--project", "src/DCLPulse/DCLPulse.csproj", "--configuration", "Debug", "--property:GenerateProto=false"]
@@ -13,6 +13,7 @@ services:
       DOTNET_ENVIRONMENT: Development
       Logging__LogLevel__Default: Debug
       Peers__ResyncWithDelta: false
+      Transport__GeoDbDirectory: /app/geodb
       WebTransport__Enabled: true
     cap_add:
       - SYS_PTRACE
 
@@ -260,6 +260,85 @@ Counter of post-auth messages rejected for invalid fields (oversized `EmoteId`/`
 
 ---
 
+## Latency metrics
+
+Histogram-backed timing metrics for the simulation and outbound-drain hot paths. The collector holds raw per-bucket counts; the dashboard's percentile columns describe the **value distribution** (ms/µs) — Window over the buckets that filled since the previous 500 ms snapshot, Lifetime over the cumulative buckets — not a rate. The sparkline plots the window P99, the tail we care about.
+
+### Δ Staleness T0 / T1 / T2 (ms)
+
+Publish→fan-out staleness of `STATE_DELTA` per AoI tier — `MonotonicTime − target.ServerTick` measured at `SendDelta`. Each tier gets its own histogram because tiers fan out on different cadences (`tierDivisor`: T0 every tick, T1 every 2nd, T2 every 4th).
+
+**Expected**: bounded by `tierDivisor × BaseTickMs` plus fan-out compute. T0 p99 is the **KR1.1 SLI**. Resync-path deltas are excluded by design — their target can be arbitrarily old when a subject idled after the client lost packets, which would pollute the histogram.
+
+| Signal | Meaning |
+|---|---|
+| P99 within the tier budget | Normal — deltas fan out promptly after publish |
+| Sustained P99 above the tier budget | Tick overrun or input backlog — cross-check Tick Duration and Incoming Queue |
+| T0 climbing while T1/T2 flat | Every-tick fan-out is the bottleneck — AoI set for the hot tier grew |
+
+### Tick Duration (µs)
+
+`SimulateTick` wall time across workers, recorded per tick in `PeersManager.RecordTickDuration`.
+
+**Expected**: well under `BaseTickMs × 1000`.
+
+| Signal | Meaning |
+|---|---|
+| Flat, well under budget | Healthy — plenty of headroom in the tick |
+| Creeping toward `BaseTickMs × 1000` | CPU saturation or AoI fan-out growth — precursor to Tick Overruns |
+
+### Tick Overruns
+
+Count of ticks that exceeded `BaseTickMs`. Rendered as a rate row (per-second), not a histogram.
+
+**Expected**: 0. Any sustained rate is an SLO breach — the simulation is not keeping tick cadence.
+
+### Drain Cycle (µs)
+
+Outbound drain-cycle duration on the ENet thread, non-empty cycles only (empty cycles are not recorded). Upper-bounds how long an outgoing message can wait for the ENet thread to service the queue.
+
+| Signal | Meaning |
+|---|---|
+| Low, stable | ENet thread drains each burst well within a service loop |
+| Growth alongside Outgoing Queue depth | ENet thread saturated — outbound is falling behind; reduce fan-out or serialization cost |
+
+**Prometheus**: these are exposed as native histograms — `dcl_pulse_delta_staleness_ms{tier="0|1|2"}`, `dcl_pulse_tick_duration_us`, `dcl_pulse_outgoing_drain_cycle_us` (Tick Overruns is the `dcl_pulse_tick_overruns_total` counter). Use `histogram_quantile()` over the `_bucket` series for fleet-level percentiles; the dashboard percentile columns show the local value distribution (window / lifetime), not rate percentiles.
+
+### Peer RTT (ms)
+
+Distribution of connected peers' smoothed round-trip time, bucketed by peer continent. ENet maintains `peer->roundTripTime` automatically from the reliable-channel ACK flow — no probe packets, no client cooperation. A sweep on the ENet thread samples **every connected peer every 5 s** (`RTT_SAMPLE_INTERVAL_MS`) and records each peer's current RTT into its region's histogram, so the distribution is **peer-weighted**: a region with more connected peers contributes proportionally more samples.
+
+The `region` label is the continent resolved from the peer's IP against the geo-whois-asn-country (IP-allocation-registry) database described below. `unknown` folds together private/loopback IPs (local dev), addresses outside the loaded ranges or carrying unassigned country codes, and — when the database is absent from the image — every peer.
+
+**500 ms seed caveat**: ENet seeds `roundTripTime` at 500 ms until the first reliable-channel ACK sample lands. A peer connected for less than one ACK round can therefore contribute that 500 ms seed to its first sweep entry — accepted noise at a 5 s cadence rather than a reason to track per-peer connect ages.
+
+The console dashboard shows a single **Peer RTT (ms)** row that merges all seven per-continent histograms (`HistogramSnapshots.Merge`); the per-region breakdown is Grafana-only.
+
+**Prometheus**: exposed as a native histogram `dcl_pulse_peer_rtt_ms` with one `region` label per continent (`af`, `as`, `eu`, `na`, `oc`, `sa`, `unknown`). Per-region percentile:
+
+```promql
+histogram_quantile(0.5, sum by (le) (rate(dcl_pulse_peer_rtt_ms_bucket{region="as"}[5m])))
+```
+
+| Signal | Meaning |
+|---|---|
+| A region's p50 stable and low | Peers there are close to the deployment — healthy |
+| One region's p50 ≫ the others | Distance-dominated latency — a case for a closer regional deployment |
+| A region's p99 ≫ its own p50 | Tail of poorly-connected peers (mobile, congested last mile) in that region |
+| `unknown` dominating with a real player population | Geo database missing from the image, or peers behind private/CGNAT egress the DB can't place |
+| Everything near 500 ms right after a connect burst | The ENet seed showing through before ACK samples arrive — transient, ignore |
+
+**Data source**: two inputs, both fetched fresh at build time. IP-range → country comes from [geo-whois-asn-country](https://github.com/sapics/ip-location-db) (CC0, public domain), the `-num` CSV variants. Country → continent comes from [GeoNames `countryInfo.txt`](https://download.geonames.org/export/dump/countryInfo.txt) (CC-BY 4.0 — *"Contains data from GeoNames (geonames.org), licensed under CC BY 4.0"*), keyed on the ISO 3166-1 alpha-2 code with the continent read from the file's continent column (Antarctica folds to `unknown`). `ContinentResolver` loads all three once at startup from `Transport:GeoDbDirectory` (default `geodb`, resolved against the app base directory; absolute paths are used as-is). A missing mapping file or IPv4 CSV is tolerated — every peer then reports under `region="unknown"`.
+
+The downloads are deliberately unpinned (no checksum, no version tag) so every image build ships current IP-allocation data, which is a stated requirement — geo ranges churn constantly and a stale pin would silently misplace peers. The risk of an unpinned fetch is contained on three fronts: the files are data-only, parsed into a lookup table with no execution path; the parser skips-and-counts malformed rows rather than throwing (see `ContinentResolver.ParseInto`); and an unusable or empty dataset — including a fetch that failed or returned garbage — degrades to `region="unknown"` with a warning rather than crashing startup.
+
+Two ways the files get to that directory:
+
+- **Docker**: each of the three Dockerfiles fetches the fresh IPv4 CSV, IPv6 CSV, and `countryInfo.txt` into the image's `geodb/` directory at build time via a single `ADD` — no runtime download. These freshly-fetched copies are authoritative for images.
+- **Local (non-Docker) builds**: the `DCLPulse.csproj` `FetchGeoDb` target predownloads the three files into the gitignored `packages/geodb/` cache (once) and copies them next to the build output, so local runs resolve regions without a Docker image. Delete `packages/geodb` to force a refresh. Offline builds warn and continue — the app then degrades to `region="unknown"`. The download is skipped in Docker builds and containers (`FetchGeoDb=false`) and on CI (`CI=true`), so it never runs where the `ADD`-provided or in-memory test copies already apply.
+
+---
+
 ## Incoming Messages
 
 Per-message-type rates for `ClientMessage` variants. Shows how many of each message type the server processes per second.
@@ -304,7 +383,8 @@ Per-message-type rates for `ServerMessage` variants. Shows the server's output c
 
 ## Adding new metrics
 
-See the `/add-metric` skill (`/.claude/skills/add-metric/SKILL.md`) for step-by-step instructions covering three patterns:
+See the `/add-metric` skill (`/.claude/skills/add-metric/SKILL.md`) for step-by-step instructions covering four patterns:
 - **Pattern A**: Counter-based (System.Diagnostics.Metrics) — for hot-path values
 - **Pattern B**: Sampled (direct read) — for queue depths and gauges
 - **Pattern C**: Per-enum collection — for counting by message type or enum variant
+- **Pattern D**: Histogram — for latency/duration value distributions (percentiles over buckets)
@@ -50,4 +50,34 @@
             <TargetPath>libenet.dylib</TargetPath>
         </None>
     </ItemGroup>
+
+    <!-- Geo database predownload for local runs. Docker images fetch their own fresh
+         copies via ADD (which take precedence at runtime); Docker builds and CI set
+         FetchGeoDb=false / CI=true so this never runs there. Delete packages/geodb
+         to force a re-download. -->
+    <PropertyGroup>
+        <GeoDbCacheDir>$(MSBuildThisFileDirectory)..\..\packages\geodb\</GeoDbCacheDir>
+    </PropertyGroup>
+
+    <Target Name="FetchGeoDb" BeforeTargets="Build"
+            Condition="'$(FetchGeoDb)' != 'false' and '$(CI)' == ''">
+        <DownloadFile SourceUrl="https://cdn.jsdelivr.net/npm/@ip-location-db/geo-whois-asn-country/geo-whois-asn-country-ipv4-num.csv"
+                      DestinationFolder="$(GeoDbCacheDir)"
+                      Condition="!Exists('$(GeoDbCacheDir)geo-whois-asn-country-ipv4-num.csv')"
+                      ContinueOnError="WarnAndContinue" />
+        <DownloadFile SourceUrl="https://cdn.jsdelivr.net/npm/@ip-location-db/geo-whois-asn-country/geo-whois-asn-country-ipv6-num.csv"
+                      DestinationFolder="$(GeoDbCacheDir)"
+                      Condition="!Exists('$(GeoDbCacheDir)geo-whois-asn-country-ipv6-num.csv')"
+                      ContinueOnError="WarnAndContinue" />
+        <DownloadFile SourceUrl="https://download.geonames.org/export/dump/countryInfo.txt"
+                      DestinationFolder="$(GeoDbCacheDir)"
+                      Condition="!Exists('$(GeoDbCacheDir)countryInfo.txt')"
+                      ContinueOnError="WarnAndContinue" />
+        <ItemGroup>
+            <GeoDbFiles Include="$(GeoDbCacheDir)*.csv" />
+            <GeoDbFiles Include="$(GeoDbCacheDir)countryInfo.txt" />
+        </ItemGroup>
+        <Copy SourceFiles="@(GeoDbFiles)" DestinationFolder="$(OutDir)geodb"
+              SkipUnchangedFiles="true" Condition="'@(GeoDbFiles)' != ''" />
+    </Target>
 </Project>
@@ -1,5 +1,8 @@
 FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
 ARG BUILD_CONFIGURATION=Release
+# The image fetches its own fresh geo CSVs via ADD in the final stage; disable the
+# csproj's local predownload so the build/publish steps never hit the network for them.
+ENV FetchGeoDb=false
 WORKDIR /build
 COPY ["src/DCLPulse/DCLPulse.csproj", "src/DCLPulse/"]
 COPY ["src/DCLPulse/global.json", "src/DCLPulse/"]
@@ -27,6 +30,14 @@ ARG COMMIT_HASH=unknown
 ENV COMMIT_HASH=$COMMIT_HASH
 WORKDIR /app
 COPY --from=publish /app/publish .
+# Geo database — single fetch at image build; the app never downloads at runtime. Refreshes
+# with every image build. IP-range CSVs: geo-whois-asn-country (CC0,
+# https://github.com/sapics/ip-location-db). Country → continent mapping: GeoNames
+# countryInfo.txt (CC-BY 4.0, https://www.geonames.org).
+ADD https://cdn.jsdelivr.net/npm/@ip-location-db/geo-whois-asn-country/geo-whois-asn-country-ipv4-num.csv \
+    https://cdn.jsdelivr.net/npm/@ip-location-db/geo-whois-asn-country/geo-whois-asn-country-ipv6-num.csv \
+    https://download.geonames.org/export/dump/countryInfo.txt \
+    /app/geodb/
 EXPOSE 7777/udp
 EXPOSE 7743/udp
 ENTRYPOINT ["dotnet", "DCLPulse.dll"]
@@ -1,4 +1,7 @@
 FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
+# The image fetches its own fresh geo CSVs via ADD in the final stage; disable the
+# csproj's local predownload so the build/publish steps never hit the network for them.
+ENV FetchGeoDb=false
 WORKDIR /build
 COPY ["src/DCLPulse/DCLPulse.csproj", "src/DCLPulse/"]
 COPY ["src/DCLPulse/global.json", "src/DCLPulse/"]
@@ -34,6 +37,14 @@ RUN apt-get update && apt-get install -y --no-install-recommends curl unzip proc
     && printf 'Port 2222\nPermitRootLogin yes\nPermitEmptyPasswords yes\n' > /etc/ssh/sshd_config.d/debug.conf
 WORKDIR /app
 COPY --from=build /app/publish .
+# Geo database — single fetch at image build; the app never downloads at runtime. Refreshes
+# with every image build. IP-range CSVs: geo-whois-asn-country (CC0,
+# https://github.com/sapics/ip-location-db). Country → continent mapping: GeoNames
+# countryInfo.txt (CC-BY 4.0, https://www.geonames.org).
+ADD https://cdn.jsdelivr.net/npm/@ip-location-db/geo-whois-asn-country/geo-whois-asn-country-ipv4-num.csv \
+    https://cdn.jsdelivr.net/npm/@ip-location-db/geo-whois-asn-country/geo-whois-asn-country-ipv6-num.csv \
+    https://download.geonames.org/export/dump/countryInfo.txt \
+    /app/geodb/
 EXPOSE 7777/udp
 EXPOSE 7743/udp
 EXPOSE 2222/tcp