Understanding the training

[Urheen]

Hi:

I have a question regarding to your training script.

In your paper, it seems that you need to add the $\Delta \epsilon$ to the noise during training, to calculate the L_reg, however, it seems that this part doesn't show in train_smooth_diffusion.py, since the noise_offset is set to 0.

If we want to replicate your work, what value should we set up?

Thanks for help!

[JiayiGuo821]

Hi @Urheen

In Eq. 9 of our paper, you can find that the computation of L_reg does not depend on sampling a delta epsilon but a unit-length delta x_0.
The noise at https://github.com/SHI-Labs/Smooth-Diffusion/blob/main/train_smooth_diffusion.py#L308 is exactly the delta x_0 in Eq. 9.

Simply put, in our implementation, we observe the input change (delta epsilon) given a unit output change (delta x_0). We expect their variation ratio can be stable.

[Urheen]

That makes a lot of sense. Thanks for clarifying!!!

[wtliao]

Hi, I am still confused about how to normally sample pixel intensities normalized to unit length, i.e. delta x_0 in Eq. 9. In the codenoise = torch.randn_like(fake_img) / math.sqrt( fake_img.shape[1] * fake_img.shape[2] * fake_img.shape[3] ), it has no business to pixel intensities. It is just simpley sampled from white noise torch.randn_like(fake_img). Please help me to understand, how it is called as "normally sample pixel intensities". Thanks.

[JiayiGuo821]

Random Gaussian noise with a fixed shape has an approximately fixed length, which can be theoretically proven or experimentally verified via torch.norm().