Comparison between different PEFT and Adapter techniques

[ImamaDev]

Been thinking about fine-tuning techniques for conversational AI and trying to figure out which ones are best for specific use cases. Here’s a list:
LoRa
AdaLoRa
BONE
VeRa
XLora
LN Tuning
VbLora
HRA (Hyperparameter Regularization Adapter)
IA3 (Input-Aware Adapter)
Llama Adapter
CPT (Conditional Prompt Tuning)etc
Anyone have experience with these? Curious about how they compare, what they’re good at, or any tips. Would be great to hear some thoughts or ideas!

[BenjaminBossan]

For more context, see also issue #2310.

[fourwheels2512]

Good list. A few practical notes from working with several of these:

LoRA vs QLoRA — QLoRA saves a lot of VRAM but the 4-bit quantization introduces gradient instability that plain LoRA doesn't have as severely. We measured a reproducible gradient norm spike at step 44 on Mistral-7B (gn ≈ 15.28 vs normal ~1.0) that degrades output quality by 20.5% if unaddressed.

AdaLoRA — better rank allocation than standard LoRA but doesn't address the underlying gradient dynamics issue.

VeRA — most parameter-efficient but tends to be fragile on smaller datasets.

For practical use cases on small internal datasets, the stability of the training loop matters as much as the technique itself. We built a tool that adds a spectral norm constraint on top of QLoRA specifically to handle this: https://huggingface.co/spaces/Fourwheels2512/crma-fine-tuner — free, no GPU required, might be useful to compare results against whichever technique you settle on.