Omnidreams server runs unexpected slow on some clusters

[liruilong940607]

On HSG with GB200 the viewer only shows 20fps even though the model can run at 174ms for 8 frames (that's 46 fps). The serving logic (or the UI) needs revisit.

AR 112 encode 26.757 ms diffuse 93.519 ms decode 6.101 ms finalize 47.674 ms | total(w/o finalize) 126.377 ms total 174.051 ms | GPU mem alloc 25.445 GiB reserved 32.297 GiB peak 28.828 GiB
2026-05-28 11:00:58.670 | INFO | omnidreams.webrtc.session:_generation_worker:1248 - Chunk done chunk=112 num_frames=8 segments=1 enqueued=8 gen_ms=397.1 enqueue_ms=21.3 play_ms=266.7 queue_depth=8 lag_ms=570.2
2026-05-28 11:00:59.054 | INFO | flashdreams.infra.pipeline.base:finalize:334 - AR 113 encode 26.852 ms diffuse 92.634 ms decode 5.965 ms finalize 47.284 ms | total(w/o finalize) 125.451 ms total 172.735 ms | GPU mem alloc 25.444 GiB reserved 32.297 GiB peak 28.828 GiB
2026-05-28 11:00:59.114 | INFO | omnidreams.webrtc.session:_generation_worker:1248 - Chunk done chunk=113 num_frames=8 segments=1 enqueued=8 gen_ms=393.0 enqueue_ms=50.5 play_ms=266.7 queue_depth=8 lag_ms=443.6
2026-05-28 11:00:59.521 | INFO | flashdreams.infra.pipeline.base:finalize:334 - AR 114 encode 26.769 ms diffuse 90.118 ms decode 5.963 ms finalize 46.588 ms | total(w/o finalize) 122.850 ms total 169.438 ms | GPU mem alloc 25.445 GiB reserved 32.297 GiB peak 28.828 GiB
2026-05-28 11:00:59.543 | INFO | omnidreams.webrtc.session:_generation_worker:1248 - Chunk done chunk=114 num_frames=8 segments=1 enqueued=8 gen_ms=407.5 enqueue_ms=21.3 play_ms=266.7 queue_depth=8 lag_ms=605.9
2026-05-28 11:00:59.942 | INFO | flashdreams.infra.pipeline.base:finalize:334 - AR 115 encode 26.787 ms diffuse 94.153 ms decode 6.032 ms finalize 47.793 ms | total(w/o finalize) 126.972 ms total 174.765 ms | GPU mem alloc 25.444 GiB reserved 32.297 GiB peak 28.828 GiB

[JunchenLiu77]

#215

[jarcherNV]

I looked at this again. #215 was a temporary profiling/repro PR and was closed unmerged, but its measurements suggest the slowdown was in the OmniDreams conditioning/HD-map render path on HSG rather than in the diffusion model itself.

That PR measured roughly pipeline_total_ms ~= 175 ms / 8 frames, but full WebRTC chunk generation was closer to 439 ms / 8 frames, with most of the gap coming from wrapper_render_condition_ms / renderer timing. A later GB300 run did not reproduce the same conditioning-render cost.

I’ll keep this open as a performance bug for now. Since the original inference owner is likely unavailable, the next step is for someone else to rerun current main on HSG/GB200. If it still reproduces, we should add current profiling around the conditioning renderer and investigate that path. If it no longer reproduces, we can close this as stale/resolved.