Generated video quality degrades progressively（omnidreams）

[RookieHuang]

Description

When running the omnidreams-sv-2steps-chunk2-loc6-lightvae-lighttae-perf recipe,
the first frame (conditioned on first_frame.png) is visually sharp, but the
generated video becomes progressively blurrier with each autoregressive step.
By the later AR steps the output is noticeably degraded across the full frame,
not localized to edges or fast-moving regions.

Steps to Reproduce

Run the demo script with any single-view example:

torchrun --nproc_per_node=1 --no-python \
    flashdreams-run \
    omnidreams-sv-2steps-chunk2-loc6-lightvae-lighttae-perf \
    --hdmap-video-paths <uuid>_hdmap.mp4 \
    --first-frame-paths first_frame.png \
    --camera-names camera_front_wide_120fov \
    --total-blocks 20 \
    --output-dir ./outputs

Tested with two clips:

• 23599139-948f-4681-b7f4-74794113086d
• 2375ecb8-c232-11ee-8685-00044baf694d

Observed Behavior

• AR step 0 (first_frame-conditioned): output quality looks correct.
• AR steps 1–19: quality degrades monotonically; by step ~10 the video is clearly
  blurry end-to-end.
• The degradation appears global (not limited to motion or occluded areas).

Expected Behavior

Consistent visual sharpness across all 20 AR steps.

Environment

• Recipe: omnidreams-sv-2steps-chunk2-loc6-lightvae-lighttae-perf
  (compile + CUDA graphs on image encoder, encoder, decoder)
• Output resolution: 704 × 1280, 157 frames
• GPU memory: ~33 GiB alloc / ~39 GiB reserved
• CUDA 13.2.1

[RookieHuang]

Anyone can reproduce the omnidreams demo?

[csyonghe]

@RookieHuang thank you for reporting this issue! We don't expect quality issues like this with the checkpoint, and we are looking into this. It is a short week for many of us on the team due to US holidays/company free days, but we will get back with details next week.

[jmccaffrey-nv]

Hi @RookieHuang , I don't reproduce the severe blurriness shown in the example image within five seconds . Inference videos attached.
I do measure decrease in sharpness (laplacian variance) over the rollout , but not so severe.

The NVIDIA OmniDreams model initial version does degrade faster for inference where there is water drops/dirt on the windshield at the first frame image.

https://github.com/user-attachments/assets/b4b63b3a-ce8a-43ed-a55b-872e4f57db4a
https://github.com/user-attachments/assets/1e9b8485-1337-413e-ae60-6ddb74da0f50

[RookieHuang]

@jmccaffrey-nv Thanks for your reply.I will check again to see where the problem lies, and share any progress with you

[RookieHuang]

hi @jmccaffrey-nv , Just to confirm with you, are these models you used?
cae4a40708e2ec9124752f5f9457d702 lightvaew2_1.pth
65738d6fbd434ce877a8c1f2137f75c5 lighttaew2_1.pth
2dc32b81e9a9621ce023d8b7148213b0 2b_res720p_30fps_i2v_hdmap_distilled.pt

[jmccaffrey-nv]

Checkpoint trace for omnidreams-sv-2steps-chunk2-loc6-lightvae-lighttae-perf:

• The runner is registered in integrations/omnidreams/pyproject.toml and maps to omnidreams.config:RUNNER_SV_2STEPS_CHUNK2_LOC6_LIGHTVAE_LIGHTTAE_PERF.
• The perf pipeline is derived from SV_2STEPS_CHUNK2_LOC6_LIGHTVAE_LIGHTTAE; the perf override only enables use_compile=True and use_cuda_graph=True on image_encoder, encoder, and decoder.
• Therefore the checkpoint paths are inherited from the base LightVAE/LightTAE chunk2 config:
  • image encoder + per-AR-step encoder: https://huggingface.co/lightx2v/Autoencoders/resolve/main/lightvaew2_1.pth
  • decoder: https://huggingface.co/lightx2v/Autoencoders/resolve/main/lighttaew2_1.pth
  • transformer: https://huggingface.co/nvidia/omni-dreams-models/resolve/main/single_view/2b_res720p_30fps_i2v_hdmap_distilled.pt

This matches the expected artifact filenames:

cae4a40708e2ec9124752f5f9457d702  lightvaew2_1.pth
65738d6fbd434ce877a8c1f2137f75c5  lighttaew2_1.pth
2dc32b81e9a9621ce023d8b7148213b0  2b_res720p_30fps_i2v_hdmap_distilled.pt

Caveat: source currently pins these by Hugging Face URL/revision (resolve/main), not by the SHA1 values above. I did not find the files in the local cache to recompute the digests, so this confirms the configured filenames/URLs, but not the exact downloaded bytes. Also note the transformer artifact is .pt, not .pth.

So the perf preset does appear wired to the intended three artifacts; the progressive blur is probably not explained by the config accidentally selecting the full Wan VAE or a different transformer checkpoint path. The remaining thing to verify is the actual HF-cached file hashes on the machine where the repro was run.

[jarcherNV]

Thanks for the details so far. I’m taking over the follow-up on this issue.

The configured runner paths look correct for this preset, but we still need to rule out an environment or cached-artifact mismatch. Could you please confirm the actual hashes of the files downloaded on the machine where you reproduced the blur, and whether this still reproduces on current main?

The expected artifacts are:

cae4a40708e2ec9124752f5f9457d702  lightvaew2_1.pth
65738d6fbd434ce877a8c1f2137f75c5  lighttaew2_1.pth
2dc32b81e9a9621ce023d8b7148213b0  2b_res720p_30fps_i2v_hdmap_distilled.pt