Activation Checkpointing for Llama3 branch#773

Open

ademeure wants to merge 3 commits intokarpathy:llama3from

ademeure:llama3_arun_new

Commits on Oct 2, 2024

activation checkpointing on llama3 branch: use "-ac 1 -r 0" for lowest possible memory, "-ac 16 -r 2" for big savings at a smaller perf cost, etc...
ademeure
committed
remove redundant swiglu recompute & add explanation
ademeure
committed
Added error when number of layers is not divisible by number of layers per activation checkpoints
ademeure
committed