feat: improve depth encoding precision and eliminate pipeline-dependent scalar

[mwfarb]

@EdwardLu2018's idea:

Problem

The current depth encoding pipeline uses a linear scalar to map Linear01Depth into the RGB channels of the side-by-side frame:

// Unity shaders (encode)
depth = SCALAR * Linear01Depth(depth, _ZBufferParams);

// Web compositor (decode)
float remoteDepth = remoteColor.r / DEPTH_SCALAR;

This has two problems:

1. Pipeline-dependent scalar mismatch

The correct scalar differs across Unity render pipelines — SRP uses 50×, but URP and HDRP originally used 5×, creating a 10× mismatch with the web compositor's hardcoded DEPTH_SCALAR = 50.0. This was patched in #27, but the root cause (a magic constant that must match across repos) remains fragile.

2. Poor precision distribution

Linear encoding in 8-bit-per-channel RGB distributes precision evenly across the depth range. But compositing artifacts (edge bleeding, depth-test errors) are most visible near the camera. Far-field depth differences are perceptually irrelevant since distant objects occupy fewer pixels.

Option A: Logarithmic Encoding (8-bit, nonlinear)

Replace the linear scalar with a logarithmic encoding that is:

• Pipeline-agnostic — uses Linear01Depth (always [0,1]) as input, no magic scalar needed
• Precision-optimal — concentrates more of the 8-bit range near the camera

Encode (Unity shaders)

// Replace: depth = 50 * Linear01Depth(depth, _ZBufferParams);
// With:
float linear = Linear01Depth(depth, _ZBufferParams);
float encoded = log(1.0 + linear * (LOG_BASE - 1.0)) / log(LOG_BASE);
col.rgb = encoded;

Decode (Web compositor GLSL)

// Replace: float remoteDepth = remoteColor.r / DEPTH_SCALAR;
// With:
float encoded = remoteColor.r;
float remoteDepth = (pow(LOG_BASE, encoded) - 1.0) / (LOG_BASE - 1.0);

Where LOG_BASE is a shared constant (e.g., 1000.0) that controls the precision curve. Higher values push more precision toward near-field.

Precision comparison

Depth Range	Linear 8-bit levels	Log (base=1000) 8-bit levels
0.0 – 0.01 (near)	~2-3	~85
0.01 – 0.1	~23	~85
0.1 – 1.0 (far)	~230	~85

Option B: YUV 16-bit Depth Encoding (via @EdwardLu2018)

Instead of using a single 8-bit channel, split 16-bit depth across the Y and U channels of the video stream's YUV color space. Video codecs (H.264, VP8, VP9) use 4:2:0 chroma subsampling, which preserves luma (Y) at full resolution while downsampling chroma (U, V) by 2× in each dimension. This means:

Channel	Codec treatment	Depth role	Precision
Y (luma)	Full resolution, least compressed	8 MSB of depth	High fidelity
U (chroma)	2× downsampled, more compressed	8 LSB of depth	Lower fidelity (acceptable for LSB)
V (chroma)	2× downsampled, more compressed	Available for other data or redundancy	—

Encode (Unity shaders)

float linear = Linear01Depth(depth, _ZBufferParams);
float depth16 = linear * 65535.0;
float msb = floor(depth16 / 256.0) / 255.0;  // 8 MSB → Y
float lsb = fmod(depth16, 256.0) / 255.0;     // 8 LSB → U

// Pack into YUV, then convert to RGB for the video encoder
// Y = msb, U = lsb, V = 0.5 (neutral)
// YUV → RGB conversion:
col.r = msb + 1.402 * (0.5 - 0.5);
col.g = msb - 0.344136 * (lsb - 0.5) - 0.714136 * (0.5 - 0.5);
col.b = msb + 1.772 * (lsb - 0.5);

Decode (Web compositor GLSL)

// RGB → YUV conversion to recover MSB and LSB
float Y = 0.299 * remoteColor.r + 0.587 * remoteColor.g + 0.114 * remoteColor.b;
float U = -0.169 * remoteColor.r - 0.331 * remoteColor.g + 0.500 * remoteColor.b + 0.5;
float remoteDepth = (Y * 256.0 + U * 255.0) / 65535.0;

Why this works

• 256× more depth resolution than current 8-bit single-channel approach
• MSB in Y is well-preserved — video codecs treat luma as highest priority
• LSB in U tolerates compression — the least significant bits can afford some loss
• V channel is free — could be used for confidence, object IDs, or redundancy
• This is a well-established technique in remote rendering literature

Tradeoff vs Option A

	Option A (Log 8-bit)	Option B (YUV 16-bit)
Precision	~8-bit, redistributed	~16-bit (256× more)
Codec resilience	Good (single channel)	MSB good, LSB degrades under heavy compression
Complexity	Simple log/exp	YUV↔RGB matrix conversion
Pipeline independence	✅	✅
Eliminates scalar	✅	✅

Options A and B could also be combined — apply logarithmic encoding to the 16-bit depth before splitting into MSB/LSB for the best of both worlds.

Files to Change

Unity (encode) — arena-renderfusion

• Runtime/Scripts/Resources/RGBDepth.shader — Standard RP (2 sites: single + dual camera)
• Addons~/io.conix.arena.renderfusion-urp/Runtime/Resources/RGBDepthURP.shader — URP (2 sites)
• Addons~/io.conix.arena.renderfusion-hdrp/Runtime/Resources/RGBDepthHDRP.shader — HDRP (2 sites)

Web (decode) — arena-web-core

• src/systems/postprocessing/shaders/glsl/compositor/frag.glsl — remove DEPTH_SCALAR, add new decode

Notes

• Both sides must agree on encoding constants. Consider making them #define / uniform so they can be tuned without recompiling.
• The frame ID encoding (top-right corner pixels) is unrelated and should not be affected.
• This is a breaking change between Unity and web — both must be updated simultaneously.