Learning Curves and Gradient Behavior

[HadiHammoud44]

Hello, thanks you for your work.
I have tried to run JEPA as is, with the same configuration, but using 2K samples from Kinetics dataset as a starting point. Subsequently, I plotted the losses (jepa and regression, although regression coeff was set to 0) along the global encoder and predictor norms. Below are the plots.

Are such plots familiar or normal?

[HadiHammoud44]

@MidoAssran

[erow]

Did you reproduce the results on the paper? I trained v-jepa on 4 nodes with 8 GPUs and got an inferior model. The interesting thing is that my training curves are different with yours.

ViT-L/16 K710	note	content	K400	ssv2	IN1K	note
v-jepa	official		81.25	68.9	74.9	bs=3072
V-jepa (n4)	impl.		78.3	66.39	70.25	accum=2, ipe=600, loss=0.578

[HadiHammoud44]

Hey, my intention was not to reproduce JEPA as I was using it on a custom dataset (and I dont have the compute power to do so). Rather, I wanted to understand the optimization dynamics (loss and gradients) of JEPA in general. However, given the small data(sub)set, it wouldnt be accurate to compare my plots with the original JEPA.

What I observed though, is that using smaller learning rates helped stabilizing the gradients and loss, ie, the loss did not have this sudden decrease and then increase anymore, and the gradient stopped having these weird spikes. In my special case, I attributed the original unfamiliar plots to overfitting on the relatively small dataset and also using high learning rate (which leads the optimizer to take a big step and miss the minima, hence resulting in the high increase in the loss).

Not sure if this is relevant to your case, but hope something will help!