Question bout Autoregressive Diffusion Training

[LiHaoy-ux]

I am working on an action-controlled Wan2.2 TI2V (Image-to-Video) model. When converting the model from bidirectional flow to an autoregressive teacher for distillation, I encountered the following issue:
Training predictions: Look normal and reasonable.
Inference results: Are significantly worse, with poor quality.
I have attached the following for reference:

Training prediction samples

train_pred_step_001300.mp4

Inference output samples:

ar_VID_20251229_164739_147_slice_295_back_180_step_001300.mp4

Training loss curves

What could be causing this discrepancy between training and inference quality?

[zhuhz22]

I think there must be a bug.

Would you like to share details that what training predictions and inference results refer to? What's implemental difference between them?

[LiHaoy-ux]

Model & weights
Base model switched from Wan2.1-T2V to Wan2.2-TI2V-5B
VAE switched from 16-channel Wan2.1 VAE to 48-channel Wan2.2 VAE (VAE-only conditioning, no CLIP)
LoRA and camera action-control modules loaded on top of the base checkpoint.

Latent / sequence geometry
Latent shape: [B, 21, 16, 60, 104] → [B, 21, 48, 30, 60]
Tokens per frame: 1560 → 450; total sequence length: 32760 → 9450

Stage-1 TF training (I2V + chunkwise)
I2V: frame 0 kept clean (no noise, no loss); frames 1–20 trained in 4-frame chunks with pattern [1] + [4×5]
Teacher forcing: full 21-frame GT passed as clean_x; camera poses duplicated for the clean+noisy token halves
New I2V-specific TF attention mask: _prepare_teacher_forcing_mask_i2v()
New I2V causal mask for non-TF path: _prepare_blockwise_causal_attn_mask_i2v()
Inference

I2V pipeline: first frame encoded online via Wan2.2 VAE → cached into KV at t=0; remaining 20 frames generated as 5 autoregressive 4-frame chunks
frame_seq_length and seq_len set explicitly to 450 / 9450 at inference time
Camera control wired into inference with separate action KV caches (cond / uncond)

Unchanged (still same as upstream)
Stage-1 loss: single-step flow-matching MSE; no CFG during training
Inference: 50-step CFG (scale 3.0) + KV-cache autoregression
initial_latent is passed from the trainer but not used inside generator_loss

[zhuhz22]

What training prediction samples and inference output samples refer to? What's implemental difference between them?