Beyond LoRA: Can you beat the most popular fine-tuning technique?

Parameter-efficient fine-tuning (PEFT) significantly reduces the memory required to adapt large models, with Low Rank Adaptation (LoRA) being the most dominant technique. Despite its popularity, it is questionable whether LoRA is truly the best-performing method or if its usage is self-reinforcing due to early adoption and better tooling. Choosing an alternative based on academic papers is difficult due to biased comparisons and a lack of standardized benchmarks. To address this, the Hugging Face PEFT library provides a unified API and is developing its own benchmarks for both language and image generation to help users make more informed decisions.