Temporal inconsistency when decoding chunk-wise ODE latents

[rqq394]

Hi, thank you for your great work!

I’m generating ODE data with the chunk-wise model:

torchrun --nproc_per_node=8 \
  get_causal_ode_data_chunkwise.py \
  --generator_ckpt checkpoints/chunkwise/ar_diffusion.pt \
  --rawdata_path dataset/clean_data \
  --output_folder dataset/ODE6KCausal_chunkwise_latents

During generation, I decoded the latents into RGB for visualization and observed noticeable frame-to-frame jumps.

00000.mp4

00001.mp4

My understanding is that this may stem from a mismatch between conditioning on ground-truth frames (teacher forcing) versus model-generated frames during autoregressive rollout.

I would like to kindly ask:

1. Is this behavior expected?
2. Would it affect the subsequent ODE training?

Thank you very much for your help!

[zhuhz22]

Yes it's expected, and it's exactly the correct way to train the ODE distillation, since the ODE distillation is performed via teacher forcing.

The inconsistency arises from the fact that, in teacher forcing training each frame is generated by attending to the GROUND TRUTH history, rather than the self-generated history. Thus, each generated frame is only dependent to the GT video, and is independent each other.