WIP feat: added video support for approximate prefix match

[capri-xiyue]

What type of PR is this?

/kind feature
What this PR does / why we need it:
dded video support for approximate prefix match.

Which issue(s) this PR fixes:

Part of #1746

Release note (write NONE if no user-facing change):

example for qwen3 vl
{
    "estimate": {
      "video": {
        "tokensPerFrame": { "mode": "dynamic", "factor": 1024 },
        "frames": { "mode": "sampled", "sampleFPS": 2, "minFrames": 4, "temporalPatchSize": 2 },
        "defaultResolution": { "width": 640, "height": 360 },
        "defaultDuration": 10,
        "maxVideoTokens": 12288
      }
    }
  }

example for gemma4
  {
    "estimate": {
      "video": {
        "tokensPerFrame": { "mode": "static", "staticToken": 256 },
        "frames": { "mode": "strided", "defaultSourceFPS": 24, "frameStride": 4, "maxFrames": 16 },
        "defaultDuration": 10,
        "maxVideoTokens": 12288
      }
    }
  }
   

Customers can provide the video metadata via headers

- x-llm-d-video-fps for video fps
- x-llm-d-video-resolution for video resulution
- x-llm-d-video-duration-seconds for video duration

[capri-xiyue]

@namgyu-youn
I drafted this to set up a framework for video estimation. A few thoughts based on experimenting your PR:

• Extracting metadata directly from the video looks complex, so let's start by grabbing it from the request headers for now.
• I simplified the token estimation logic in this PR to prioritize a clean user interface.

Let's keep iterating in your PR to figure out the best balance between accurate token estimation and a simple user experience. And extracting metadata from video directly could be a follow up PR if we think it is feasible.

And let me know your thoughts about this PR

[capri-xiyue]

Here is the estimated comparison with qwen3 vl
with setting
cfg: &estimateConfig{Video: &videoEstimateConfig{
Frames: &framesConfig{Mode: videoFramesModeSampled, SampleFPS: 2, MinFrames: 4, TemporalPatchSize: 2},
TokensPerFrame: &tokensPerFrameConfig{Mode: videoTPFModeDynamic, Factor: 32 * 32},
MaxVideoTokens: 12288,
}},

=== RUN TestVideoEstimator_PlaceholderCount_Live
estimate_live_test.go:72: resolution dur estimate actual err%
=== RUN TestVideoEstimator_PlaceholderCount_Live/360p/1s
estimate_live_test.go:96: 360p 1 s 450 456 -1.3%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/360p/1s (0.90s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/360p/10s
estimate_live_test.go:96: 360p 10 s 2250 2280 -1.3%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/360p/10s (1.10s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/360p/20s
estimate_live_test.go:96: 360p 20 s 4500 4570 -1.5%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/360p/20s (1.51s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/360p/30s
estimate_live_test.go:96: 360p 30 s 6750 6860 -1.6%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/360p/30s (3.14s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/360p/60s
estimate_live_test.go:96: 360p 60 s 12288 11930 3.0%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/360p/60s (3.02s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/360p/90s
estimate_live_test.go:96: 360p 90 s 12288 11600 5.9%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/360p/90s (4.43s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/720p/1s
estimate_live_test.go:96: 720p 1 s 1800 1776 1.4%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/720p/1s (0.63s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/720p/10s
estimate_live_test.go:96: 720p 10 s 9000 8880 1.4%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/720p/10s (1.46s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/720p/20s
estimate_live_test.go:96: 720p 20 s 12288 12050 2.0%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/720p/20s (2.64s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/720p/30s
estimate_live_test.go:96: 720p 30 s 12288 12410 -1.0%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/720p/30s (3.48s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/720p/60s
estimate_live_test.go:96: 720p 60 s 12288 11930 3.0%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/720p/60s (5.53s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/720p/90s
estimate_live_test.go:96: 720p 90 s 12288 11600 5.9%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/720p/90s (7.58s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/1080p/1s
estimate_live_test.go:96: 1080p 1 s 4050 4096 -1.1%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/1080p/1s (0.77s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/1080p/10s
estimate_live_test.go:96: 1080p 10 s 12288 12300 -0.1%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/1080p/10s (2.15s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/1080p/20s
estimate_live_test.go:96: 1080p 20 s 12288 12050 2.0%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/1080p/20s (3.63s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/1080p/30s
estimate_live_test.go:96: 1080p 30 s 12288 12410 -1.0%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/1080p/30s (5.12s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/1080p/60s
estimate_live_test.go:96: 1080p 60 s 12288 11930 3.0%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/1080p/60s (10.03s)
=== RUN TestVideoEstimator_PlaceholderCount_Live/1080p/90s
estimate_live_test.go:96: 1080p 90 s 12288 11600 5.9%
--- PASS: TestVideoEstimator_PlaceholderCount_Live/1080p/90s (13.31s)
--- PASS: TestVideoEstimator_PlaceholderCount_Live (70.44s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live
estimate_live_test.go:120: resolution dur estimate actual err%
=== RUN TestEstimateBackend_ProduceTokenCount_Live/360p/1s
estimate_live_test.go:160: 360p 1 s 456 467 -2.4%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/360p/1s (0.41s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/360p/10s
estimate_live_test.go:160: 360p 10 s 2256 2291 -1.5%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/360p/10s (0.81s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/360p/20s
estimate_live_test.go:160: 360p 20 s 4506 4581 -1.6%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/360p/20s (1.04s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/360p/30s
estimate_live_test.go:160: 360p 30 s 6756 6871 -1.7%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/360p/30s (1.44s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/360p/60s
estimate_live_test.go:160: 360p 60 s 12294 11941 3.0%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/360p/60s (2.68s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/360p/90s
estimate_live_test.go:160: 360p 90 s 12294 11611 5.9%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/360p/90s (3.87s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/720p/1s
estimate_live_test.go:160: 720p 1 s 1806 1787 1.1%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/720p/1s (0.62s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/720p/10s
estimate_live_test.go:160: 720p 10 s 9006 8891 1.3%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/720p/10s (1.64s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/720p/20s
estimate_live_test.go:160: 720p 20 s 12294 12061 1.9%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/720p/20s (2.67s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/720p/30s
estimate_live_test.go:160: 720p 30 s 12294 12421 -1.0%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/720p/30s (3.48s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/720p/60s
estimate_live_test.go:160: 720p 60 s 12294 11941 3.0%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/720p/60s (5.53s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/720p/90s
estimate_live_test.go:160: 720p 90 s 12294 11611 5.9%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/720p/90s (7.58s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/1080p/1s
estimate_live_test.go:160: 1080p 1 s 4056 4107 -1.2%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/1080p/1s (0.76s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/1080p/10s
estimate_live_test.go:160: 1080p 10 s 12294 12311 -0.1%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/1080p/10s (2.19s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/1080p/20s
estimate_live_test.go:160: 1080p 20 s 12294 12061 1.9%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/1080p/20s (3.93s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/1080p/30s
estimate_live_test.go:160: 1080p 30 s 12294 12421 -1.0%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/1080p/30s (5.60s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/1080p/60s
estimate_live_test.go:160: 1080p 60 s 12294 11941 3.0%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/1080p/60s (10.24s)
=== RUN TestEstimateBackend_ProduceTokenCount_Live/1080p/90s
estimate_live_test.go:160: 1080p 90 s 12294 11611 5.9%
--- PASS: TestEstimateBackend_ProduceTokenCount_Live/1080p/90s (13.93s)
--- PASS: TestEstimateBackend_ProduceTokenCount_Live (68.41s)

for gemma4, I use setting as follows
cfg: &estimateConfig{Video: &videoEstimateConfig{
Frames: &framesConfig{Mode: videoFramesModeStrided, FrameStride: 4, MaxFrames: 8},
TokensPerFrame: &tokensPerFrameConfig{Mode: videoTPFModeStatic, StaticToken: 296},
}},
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live
estimate_live_test.go:131: resolution dur estimate actual err%
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/360p/1s
estimate_live_test.go:170: 360p 1 s 1782 2265 -21.3%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/360p/1s (2.41s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/360p/10s
estimate_live_test.go:170: 360p 10 s 2374 2340 1.5%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/360p/10s (2.93s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/360p/20s
estimate_live_test.go:170: 360p 20 s 2374 2415 -1.7%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/360p/20s (4.94s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/360p/30s
estimate_live_test.go:170: 360p 30 s 2374 2340 1.5%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/360p/30s (6.43s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/360p/60s
estimate_live_test.go:170: 360p 60 s 2374 2340 1.5%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/360p/60s (10.78s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/360p/90s
estimate_live_test.go:170: 360p 90 s 2374 2340 1.5%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/360p/90s (15.25s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/720p/1s
estimate_live_test.go:170: 720p 1 s 1782 2265 -21.3%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/720p/1s (2.40s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/720p/10s
estimate_live_test.go:170: 720p 10 s 2374 2415 -1.7%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/720p/10s (4.20s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/720p/20s
estimate_live_test.go:170: 720p 20 s 2374 2415 -1.7%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/720p/20s (9.23s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/720p/30s
estimate_live_test.go:170: 720p 30 s 2374 2340 1.5%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/720p/30s (11.66s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/720p/60s
estimate_live_test.go:170: 720p 60 s 2374 2340 1.5%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/720p/60s (24.01s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/720p/90s
estimate_live_test.go:170: 720p 90 s 2374 2340 1.5%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/720p/90s (35.89s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/1080p/1s
estimate_live_test.go:170: 1080p 1 s 1782 2265 -21.3%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/1080p/1s (1.97s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/1080p/10s
estimate_live_test.go:170: 1080p 10 s 2374 2340 1.5%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/1080p/10s (6.34s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/1080p/20s
estimate_live_test.go:170: 1080p 20 s 2374 2415 -1.7%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/1080p/20s (18.59s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/1080p/30s
estimate_live_test.go:170: 1080p 30 s 2374 2340 1.5%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/1080p/30s (29.19s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/1080p/60s
estimate_live_test.go:170: 1080p 60 s 2374 2340 1.5%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/1080p/60s (47.46s)
=== RUN TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/1080p/90s
estimate_live_test.go:170: 1080p 90 s 2374 2340 1.5%
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live/1080p/90s (94.36s)
--- PASS: TestEstimateBackend_GEMMA4_ProduceTokenCount_Live (328.02s)

[capri-xiyue]

cc @ahg-g

[namgyu-youn]

Thanks for drafting this — the header-based metadata idea seems a nice simplification over parsing MP4 containers directly. Before we converge on an interface, I want to flag an accuracy concern with the formula itself.

frames.mode=sampled (clamp(duration × sampleFPS, minFrames, maxFrames) / temporalPatchSize) and tokensPerFrame.mode=dynamic (width × height / factor) treat frame count and per-frame tokens as independent knobs. For Qwen3-VL they aren't:

1. Frame count is a hard cap, not a smooth function of duration. vLLM caps decoded frames at exactly 32 once duration × sourceFPS > 32; below that it resamples to totalFrames/srcFPS × sampleFPS. That's a branch on total frames (depends on source fps), and the sampled-mode formula has no fps term at all. See https://github.com/vllm-project/vllm/blob/e040899a006ffd07c51151b32d29649f30cb21e9/vllm/model_executor/models/qwen3_vl.py#L1012-L1030
2. Tokens-per-frame is coupled to frame count — smart_resize divides one shared pixel budget, across however many frames were sampled, so per-frame resolution shrinks as frame count grows. A fixed $w × h / factor$ doesn't capture that.
3. Timestamp token overhead scales with the temporal-patch-grouped frame count, which depends on the cap in (1).

Your benchmark shows durations 1–30s tracking linearly within ~1.5%, then 60s/90s both plateauing at exactly maxVideoTokens: 12288 — which looks like factor and maxVideoTokens were tuned to fit this specific grid rather than the formula reproducing the cap independently. Could you reproduce different source like 60fps, non-24/30 capture rate?

#1408's formula mirrors the real vLLM mechanism rather than fitting a curve to it, which is why it holds up on Video-MME clips it wasn't tuned against. Separately, could we land #1408 first? It'd give us a merged, validated baseline for Qwen3-VL to iterate the interface question on top of, rather than deciding the interface before the underlying formula is settled.

[capri-xiyue]

reproduce different source like 60fps, non-24/30 capture rate?

I ran it with the fps you suggested

estimate_live_test.go:217: res      dur   fps        estimate     actual     err%
estimate_live_test.go:256: 360p     10  s 24             2256       2291    -1.5%
estimate_live_test.go:256: 360p     10  s 25             2256       2291    -1.5%
estimate_live_test.go:256: 360p     10  s 29.97          2256       2291    -1.5%
estimate_live_test.go:256: 360p     10  s 30             2256       2291    -1.5%
estimate_live_test.go:256: 360p     10  s 50             2256       2291    -1.5%
estimate_live_test.go:256: 360p     10  s 60             2256       2291    -1.5%
estimate_live_test.go:256: 720p     10  s 24             9006       8891     1.3%
estimate_live_test.go:256: 720p     10  s 25             9006       8891     1.3%
estimate_live_test.go:256: 720p     10  s 29.97          9006       8891     1.3%
estimate_live_test.go:256: 720p     10  s 30             9006       8891     1.3%
estimate_live_test.go:256: 720p     10  s 50             9006       8891     1.3%
estimate_live_test.go:256: 720p     10  s 60             9006       8891     1.3%

The estimation is still relatively accurate. The source video's FPS actually doesn't matter, since Qwen 3VL extracts frames using its own sampling rate(sample fps).

For context, I adapted the formula from your PR and the Qwen VL source code, just simplifying it slightly. To clarify, our goal isn't 100% perfect token estimation, but rather finding the right balance between UI simplicity, processing speed, and token estimation accuracy.