Skip to content

DESIGN.md

Latest commit

 

History

History
130 lines (88 loc) · 3.63 KB

DESIGN.md

File metadata and controls

130 lines (88 loc) · 3.63 KB
Raw

Design Document

token-streaming-latency-bench

Author: Joao Felipe De Souza Year: 2026

1. Objective

This benchmark measures user-perceived streaming quality in LLM token delivery, separating the server-side metrics (TTFT, TPOT) from the client-side experience (TTI, TTRC, SPIS, SPIS-R).

2. Pipeline Model

The full streaming pipeline modeled:

token generation
→ SSE frame construction (per-flush overhead)
→ network transport (base latency + jitter + per-KB cost)
→ client render frame coalescing (60 FPS)
→ user perception

3. Metrics

TTI - Time to Interactive

Time from request start until the user receives MIN_INTERACTIVE_TOKENS tokens after render frame coalescing.

TTRC - Time to Readable Chunk

Time until the first chunk where the last token is a word boundary. This is when the user first sees a complete word.

SPIS - Streaming Perceived Interactivity Score

Composite metric 0-100. Penalizes:

  • High TTI
  • Large inter-chunk gaps (p95)
  • Large average chunk size
  • High coefficient of variation in gaps

Rewards:

  • High word-boundary flush fraction

SPIS-R - Readable-Aware SPIS

Extends SPIS by:

  • Using TTRC instead of TTI as the latency baseline
  • Adding explicit penalty for midword chunk fraction
  • Rewarding policies that flush at readable boundaries

SPIS-R better reflects perceived quality in subword-heavy workloads where SPIS would assign high scores to policies that deliver unreadable partial-word chunks with good timing.

4. Flush Policies

Token-count policies

Flush after N tokens regardless of content. Simple, predictable, high overhead for small N.

Timer policies

Flush on a fixed interval. Consistent cadence, moderate overhead.

Hybrid

Flush on whichever comes first: N tokens or T milliseconds. Balances responsiveness and overhead.

Word-boundary (semantic)

Flush when the current token ends a word, with a timer fallback. Lowest midword chunk fraction, highest TTI.

Punctuation (semantic)

Flush on sentence-ending punctuation with a timer fallback. High word-boundary alignment, lower overhead than per-token flushing.

Render-frame-aware

Flush at 60 FPS boundaries (every 16.7ms). Minimizes TTI but creates irregular inter-frame gaps.

5. Token Properties

Each token is assigned properties sampled from workload-specific distributions:

  • is_word_boundary: prob varies from 0.20 (subword_heavy) to 0.65 (natural_text)
  • is_punctuation: prob varies from 0.04 (subword_heavy) to 0.18 (code_heavy)

This allows semantic policies to behave differently across workload types and exposes the tokenization alignment problem in code/subword workloads.

6. Key Design Insight: The Three-Way Trade-off

No flush policy dominates all three user-perceived dimensions:

TTI (first visible text):  render_frame_aware < timer_50ms < word_boundary
SPIS (cadence fluidity):   timer_50ms > hybrid > semantic
SPIS-R (readability):      timer_50ms > hybrid > word_boundary

The trade-off is fundamental. A policy that minimizes TTI (render-frame) creates irregular gaps. A policy that maximizes readability (word_boundary) increases TTI. Timer policies achieve the best composite score by balancing all three.

7. Limitations

Known simplifications:

  • token generation times are deterministic given batch size
  • network jitter is Gaussian (real networks have heavier tails)
  • render coalescing assumes 60 FPS stable frame rate
  • token properties are stationary within a workload
  • no modeling of backpressure or flow control
  • no multi-user contention

These simplifications isolate the streaming policy comparison from confounding factors.