Skip to content

Commit 69f9051

Browse files
mikhail-dclclaudelorenzo-ranciaffi
authored andcommitted
chore: sync pulse-prd (#441)
* docs: describe protoc-gen-bitwise as it actually works; drop dead BitReader/BitWriter CLAUDE.md still documented the original bit-stream design (BitWriter/ BitReader classes, float fields, "68 bits = 9 bytes on the wire"). The plugin actually emits quantized-accessor partials (*.Bitwise.cs) over plain uint32 fields sent as standard protobuf varints, backed only by Quantize.cs; runtime BitReader.cs/BitWriter.cs were referenced by nothing and are removed. Also sync README with current generator output (QuantizedStep consts, AreQuantizedFieldsInRange, EncodePower/ DecodePower, per-proto-file output naming) and drop the incorrect "cached" accessor wording. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: normalize CRLF when comparing gen:test golden fixtures With core.autocrlf=true (and no .gitattributes) git materializes the golden .cs fixtures with CRLF on Windows while the generator always emits LF, so the strict byte comparison failed on any fresh Windows checkout. Normalize line endings when reading the goldens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: add realm in pulse PlayerJoined message * added realm to teleport performed message --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Lorenzo Ranciaffi <lorenzo.ranciaffi@decentraland.org>
1 parent c2e8777 commit 69f9051

6 files changed

Lines changed: 89 additions & 315 deletions

File tree

CLAUDE.md

Lines changed: 49 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -10,15 +10,23 @@ High-performance MMO-style multiplayer networking stack. Protocol is **open** (U
1010
| Client | Unity (C#) |
1111
| Server | Custom server |
1212
| Schema source of truth | `.proto` files |
13-
| Serialization | Custom protoc plugin (bitwise encoding) |
13+
| Serialization | Standard protobuf wire format + custom protoc plugin (quantized float accessors) |
1414
| Auth | Decentraland ECDSA chain validation (local, on HANDSHAKE channel 0) |
1515

1616
---
1717

1818
## Serialization: Custom Protoc Plugin
1919

2020
### What it does
21-
Reads `.proto` files with custom field options and generates **bitwise encode/decode code** in C#, keeping all client implementations bit-for-bit identical.
21+
Reads `.proto` files with custom field options and generates C# **partial classes** (`*.Bitwise.cs`) that add typed float accessors on top of quantized `uint32` fields, keeping the quantization math bit-for-bit identical across all client implementations.
22+
23+
The wire format is **standard protobuf** — a quantized value lives in a plain `uint32` field and travels as an ordinary varint. There is no custom bit stream; any protobuf-capable client can parse the messages without this plugin. Per annotated field the plugin emits:
24+
25+
- `float {Field}Quantized` — computed accessor (no backing cache): the getter decodes the stored `uint32`, the setter encodes a float back into it, via the static `Quantize` helpers
26+
- `const float {Field}QuantizedStep` — the coarsest quantization step of the field, safe as an equality tolerance
27+
- per message: `bool AreQuantizedFieldsInRange()` — pure-integer check that every stored code fits its declared bit width (`0 .. 2^bits-1`); reject malformed/hostile messages before storing or relaying
28+
29+
Only non-repeated `uint32` fields get accessors; `bit_packed` and unannotated fields pass through with no generated code.
2230

2331
### Custom Field Options (`options.proto`)
2432

@@ -54,25 +62,31 @@ extend google.protobuf.FieldOptions {
5462

5563
### Usage example
5664

65+
Quantized fields are declared **`uint32`** (not `float`) — the float type exists only in the generated accessor:
66+
5767
```protobuf
5868
message PositionDelta {
59-
float dx = 1 [(quantized) = { min: -100.0, max: 100.0, bits: 16 }];
60-
float dy = 2 [(quantized) = { min: -100.0, max: 100.0, bits: 16 }];
61-
float dz = 3 [(quantized) = { min: -100.0, max: 100.0, bits: 16 }];
69+
uint32 dx = 1 [(quantized) = { min: -100.0, max: 100.0, bits: 16 }];
70+
uint32 dy = 2 [(quantized) = { min: -100.0, max: 100.0, bits: 16 }];
71+
uint32 dz = 3 [(quantized) = { min: -100.0, max: 100.0, bits: 16 }];
6272
uint32 entity_id = 4 [(bit_packed) = { bits: 20 }];
73+
uint32 sequence = 5 [(bit_packed) = { bits: 12 }];
6374
}
64-
// Total: 68 bits = 9 bytes on the wire
75+
// Varint wire cost: dx/dy/dz/entity_id ≤ 4 B each (1 B tag + ≤ 3 B varint),
76+
// sequence ≤ 3 B — worst-case 19 B, less when proto3 omits zero-valued fields.
6577
```
6678

79+
`proto/decentraland/common/quantization_example.proto` is the fully worked reference: per-field wire costs for the linear, power-law, and bit-packed annotations.
80+
6781
### Plugin structure
6882

6983
```
7084
protoc-gen-bitwise/
7185
├── plugin.js # stdin -> CodeGeneratorRequest, stdout -> CodeGeneratorResponse (Node)
72-
├── generator_csharp.js # emits C# for Unity
73-
├── options.js # parses the custom quantized / bit_packed field options
86+
├── generator_csharp.js # emits the *.Bitwise.cs accessor partials for Unity
87+
├── options.js # parses the custom quantized / quantized_power / bit_packed field options
7488
├── wire.js # self-contained protobuf wire codec (zero runtime deps)
75-
└── runtime/cs/ # C# runtime; Quantize.cs is copied into the generated output
89+
└── runtime/cs/ # C# runtime: Quantize.cs — consumers copy it next to the generated files
7690
```
7791

7892
Plugin contract: a protoc plugin that reads a serialized `CodeGeneratorRequest` from stdin and writes a serialized `CodeGeneratorResponse` to stdout. It is a plain Node script — **no `npm install` required, only `node` on PATH**. protoc invokes it through a tiny wrapper that runs `node plugin.js` (`.cmd` on Windows, a shell script elsewhere, since protoc cannot exec a `.js` directly).
@@ -89,85 +103,30 @@ Parity is locked down by `npm run gen:test` (compares generator output against g
89103

90104
---
91105

92-
## BitWriter / BitReader
106+
## Quantize Runtime
93107

94-
The C# implementation uses the following bit layout: **big-endian within each byte**, MSB written first. Use **`Round`** (not truncate) when quantizing to minimize error.
108+
Generated accessors call the static `Quantize` class (`protoc-gen-bitwise/runtime/cs/Quantize.cs`, namespace `Decentraland.Networking.Bitwise`) — the only C# runtime file; consumers copy it next to the generated `*.Bitwise.cs` partials. Quantization uses **`Round`** (not truncate) to minimize error; identical rounding on both sides makes encode -> decode a round-trip no-op.
95109

96-
### Core math — WriteQuantizedFloat
110+
### Core math — linear (`Quantize.Encode` / `Quantize.Decode`)
97111

98112
```
99-
normalized = (clamp(value, min, max) - min) / (max - min) // -> [0.0, 1.0]
100-
quantized = Round(normalized * ((1 << bits) - 1)) // -> integer
113+
encoded = Round(clamp01((value - min) / (max - min)) * (2^bits - 1))
114+
decoded = encoded / (2^bits - 1) * (max - min) + min
101115
```
102116

103-
### Core math — ReadQuantizedFloat
117+
### Core math — power-law (`Quantize.EncodePower` / `Quantize.DecodePower`)
118+
119+
For signed fields like velocity that need an exact zero and fine resolution near zero:
104120

105121
```
106-
normalized = quantized / ((1 << bits) - 1)
107-
value = min + normalized * (max - min)
122+
u = clamp01(|value| / max) ^ (1 / pow)
123+
encoded = (Round(u * (2^(bits-1) - 1)) << 1) | sign // magnitude in high bits, sign in LSB
124+
decoded = sign * max * ((encoded >> 1) / (2^(bits-1) - 1)) ^ pow
108125
```
109126

110-
### Implementation
111-
112-
```csharp
113-
public class BitWriter
114-
{
115-
private byte[] _buffer;
116-
private int _bitPos;
117-
118-
public BitWriter(byte[] buffer) { _buffer = buffer; _bitPos = 0; }
119-
120-
public void WriteBits(uint value, int bits)
121-
{
122-
for (int i = bits - 1; i >= 0; i--)
123-
{
124-
int byteIdx = _bitPos / 8;
125-
int bitIdx = 7 - (_bitPos % 8);
126-
if ((value >> i & 1) == 1) _buffer[byteIdx] |= (byte)(1 << bitIdx);
127-
else _buffer[byteIdx] &= (byte)~(1 << bitIdx);
128-
_bitPos++;
129-
}
130-
}
131-
132-
public void WriteQuantizedFloat(float value, float min, float max, int bits)
133-
{
134-
uint maxQ = (1u << bits) - 1;
135-
float clamped = Math.Clamp(value, min, max);
136-
float normalized = (clamped - min) / (max - min);
137-
uint quantized = (uint)Math.Round(normalized * maxQ);
138-
WriteBits(quantized, bits);
139-
}
140-
}
141-
142-
public class BitReader
143-
{
144-
private byte[] _buffer;
145-
private int _bitPos;
146-
147-
public BitReader(byte[] buffer) { _buffer = buffer; _bitPos = 0; }
148-
149-
public uint ReadBits(int bits)
150-
{
151-
uint value = 0;
152-
for (int i = bits - 1; i >= 0; i--)
153-
{
154-
int byteIdx = _bitPos / 8;
155-
int bitIdx = 7 - (_bitPos % 8);
156-
if ((_buffer[byteIdx] >> bitIdx & 1) == 1) value |= 1u << i;
157-
_bitPos++;
158-
}
159-
return value;
160-
}
161-
162-
public float ReadQuantizedFloat(float min, float max, int bits)
163-
{
164-
uint maxQ = (1u << bits) - 1;
165-
uint quantized = ReadBits(bits);
166-
float normalized = (float)quantized / maxQ;
167-
return min + normalized * (max - min);
168-
}
169-
}
170-
```
127+
- Zero encodes exactly to code `0` (a zero magnitude never sets the sign bit), so proto3 omits a stopped field entirely
128+
- `pow > 1` concentrates resolution near zero, coarse near `±max`
129+
- Sign in the LSB makes the varint cost track magnitude, not direction — a small `|value|` of either sign stays in one varint byte
171130

172131
---
173132

@@ -181,13 +140,24 @@ public class BitReader
181140

182141
Sub-centimeter precision is achievable at 12-16 bits for position deltas.
183142

143+
Wire cost is varint-based — 1 tag byte per present field (field numbers ≤ 15) plus:
144+
145+
| Code bits | Worst-case varint | Worst-case field total |
146+
|-----------|-------------------|------------------------|
147+
| ≤ 7 | 1 B | 2 B |
148+
| ≤ 14 | 2 B | 3 B |
149+
| ≤ 21 | 3 B | 4 B |
150+
151+
Proto3 omits fields equal to 0, so typical cost is lower than worst-case.
152+
184153
---
185154

186155
## Key Design Principles
187156

188157
- `.proto` files are the **single source of truth** for all message schemas
189-
- The protoc plugin generates **C#** from the schema — never hand-write serialization
158+
- The protoc plugin generates the **C# quantized accessors** from the schema — never hand-write quantization math; standard protobuf handles the wire encoding
190159
- Encode -> decode is a **no-op** (round-trip safe) due to consistent use of `Round`
160+
- Validate inbound quantized messages with `AreQuantizedFieldsInRange()` before storing or relaying — the server relays raw codes verbatim
191161
- Prefer **client-driven resync** over proactive server corrections
192162
- Push complexity to clients where appropriate; server maintains authority
193163
- Channel 0: reliable messages (STATE_FULL snapshots, ACKs, resync requests, HANDSHAKE)

README.md

Lines changed: 35 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -90,8 +90,10 @@ Rather than a separate binary packing layer, the plugin leverages this:
9090
2. `--csharp_out` generates the standard protobuf class with the raw `uint32`
9191
property (e.g. `PositionX`).
9292
3. `--bitwise_out` (this plugin) generates a `partial class` extension with a
93-
cached float accessor (e.g. `PositionXQuantized`) that encodes/decodes
94-
transparently via `Quantize.Encode` / `Quantize.Decode`.
93+
computed float accessor (e.g. `PositionXQuantized`) that encodes/decodes
94+
transparently via `Quantize.Encode` / `Quantize.Decode` on every access —
95+
the raw `uint32` property remains the single source of truth (no cache to
96+
go stale when the raw field is mutated directly).
9597

9698
The wire representation is a standard protobuf message — any protobuf-capable
9799
client can read it without knowledge of the plugin.
@@ -133,8 +135,8 @@ message PositionDelta {
133135

134136
| Annotation | Target type | Parameters | Effect |
135137
|---|---|---|---|
136-
| `[(decentraland.common.quantized)]` | `uint32` | `min`, `max`, `bits` | Plugin emits a cached `float {Name}Quantized` accessor |
137-
| `[(decentraland.common.quantized_power)]` | `uint32` | `max`, `pow`, `bits` | Power-law quantizer over `[-max, max]`: `(bits-1)`-bit magnitude (high bits) + sign (LSB), decoded as `sign·max·u^pow`. Exact zero; `pow>1` gives fine resolution near zero, coarse near `±max`; sign in the LSB keeps small magnitudes one varint byte. Cached `float {Name}Quantized` accessor (`Quantize.EncodePower`/`DecodePower`) |
138+
| `[(decentraland.common.quantized)]` | `uint32` | `min`, `max`, `bits` | Plugin emits a `float {Name}Quantized` accessor and a `{Name}QuantizedStep` const |
139+
| `[(decentraland.common.quantized_power)]` | `uint32` | `max`, `pow`, `bits` | Power-law quantizer over `[-max, max]`: `(bits-1)`-bit magnitude (high bits) + sign (LSB), decoded as `sign·max·u^pow`. Exact zero; `pow>1` gives fine resolution near zero, coarse near `±max`; sign in the LSB keeps small magnitudes one varint byte. `float {Name}Quantized` accessor (`Quantize.EncodePower`/`DecodePower`) |
138140
| `[(decentraland.common.bit_packed)]` | `uint32` | `bits` | Documents the value range; protobuf handles varint compaction automatically |
139141

140142
### Wire cost at worst-case (all bits set)
@@ -183,13 +185,15 @@ Assets/
183185
```
184186

185187
`Quantize.cs` lives in the `Decentraland.Networking.Bitwise` namespace and
186-
provides two static methods used by the generated accessors:
188+
provides the static encode/decode methods used by the generated accessors:
187189

188190
```csharp
189191
public static class Quantize
190192
{
191193
public static uint Encode(float value, float min, float max, int bits);
192194
public static float Decode(uint encoded, float min, float max, int bits);
195+
public static uint EncodePower(float value, float max, float pow, int bits);
196+
public static float DecodePower(uint encoded, float max, float pow, int bits);
193197
}
194198
```
195199

@@ -213,14 +217,18 @@ SendOnChannel1(bytes);
213217

214218
// --- Receive and read ---
215219
var received = PositionDelta.Parser.ParseFrom(receivedBytes);
220+
if (!received.AreQuantizedFieldsInRange()) return; // reject malformed/hostile codes
216221
float x = received.DxQuantized; // decoded from the stored uint32 on each access
217222
float y = received.DyQuantized;
218223
float z = received.DzQuantized;
219224
```
220225

221226
## Generated file example
222227

223-
For the `PositionDelta` message above the plugin emits `PositionDelta.Bitwise.cs`:
228+
For the `comms.proto` file above the plugin emits `Comms.Bitwise.cs` (one file
229+
per `.proto`, named after the proto file). Each quantized field gets a
230+
`{Name}QuantizedStep` const and a float accessor; each message gets an
231+
`AreQuantizedFieldsInRange()` guard for validating inbound wire codes:
224232

225233
```csharp
226234
// <auto-generated>
@@ -234,23 +242,44 @@ namespace Decentraland.Kernel.Comms.V3
234242
{
235243
public partial class PositionDelta
236244
{
245+
/// <summary>Coarsest quantization step of <see cref="DxQuantized"/>. Safe as an equality tolerance.</summary>
246+
public const float DxQuantizedStep = 0.0030518044f;
247+
/// <summary>Float accessor for <see cref="Dx"/>. Range [-100.0f, 100.0f], 16 bits, step ≈ 0.0030518.</summary>
237248
public float DxQuantized
238249
{
239250
get => Quantize.Decode(Dx, -100.0f, 100.0f, 16);
240251
set => Dx = Quantize.Encode(value, -100.0f, 100.0f, 16);
241252
}
242253

254+
/// <summary>Coarsest quantization step of <see cref="DyQuantized"/>. Safe as an equality tolerance.</summary>
255+
public const float DyQuantizedStep = 0.0030518044f;
256+
/// <summary>Float accessor for <see cref="Dy"/>. Range [-100.0f, 100.0f], 16 bits, step ≈ 0.0030518.</summary>
243257
public float DyQuantized
244258
{
245259
get => Quantize.Decode(Dy, -100.0f, 100.0f, 16);
246260
set => Dy = Quantize.Encode(value, -100.0f, 100.0f, 16);
247261
}
248262

263+
/// <summary>Coarsest quantization step of <see cref="DzQuantized"/>. Safe as an equality tolerance.</summary>
264+
public const float DzQuantizedStep = 0.0030518044f;
265+
/// <summary>Float accessor for <see cref="Dz"/>. Range [-100.0f, 100.0f], 16 bits, step ≈ 0.0030518.</summary>
249266
public float DzQuantized
250267
{
251268
get => Quantize.Decode(Dz, -100.0f, 100.0f, 16);
252269
set => Dz = Quantize.Encode(value, -100.0f, 100.0f, 16);
253270
}
271+
272+
/// <summary>
273+
/// True when every quantized field holds a wire code within its declared bit width
274+
/// (<c>0 .. 2^bits-1</c>). The encoder never emits a code above this bound, so a larger
275+
/// value is a malformed/hostile message: decoding it would land far outside the field's
276+
/// <c>[min, max]</c> and, since the server relays raw codes verbatim, poison every observer.
277+
/// Reject before storing or relaying. Pure integer comparison — no decode.
278+
/// </summary>
279+
public bool AreQuantizedFieldsInRange() =>
280+
Dx <= 65535u
281+
&& Dy <= 65535u
282+
&& Dz <= 65535u;
254283
}
255284

256285
} // namespace Decentraland.Kernel.Comms.V3

proto/decentraland/pulse/pulse_server.proto

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@ message PlayerJoined {
8484
string user_id = 1;
8585
int32 profile_version = 2;
8686
PlayerStateFull state = 3;
87+
string realm = 4;
8788
}
8889

8990
// Notification to the client, that a peer has left, it can mean disconnection or leaving the area of interest
@@ -125,6 +126,7 @@ message TeleportPerformed {
125126
uint32 sequence = 2;
126127
uint32 server_tick = 3;
127128
PlayerState state = 4;
129+
string realm = 5;
128130
}
129131

130132
message ServerMessage {

0 commit comments

Comments
 (0)