arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.10993cs.CLcs.AI
收藏

LoRA-Squeeze:简单且有效的LoRA模块后调优和调优压缩

LoRA-Squeeze: Simple and Effective Post-Tuning and In-Tuning Compression of LoRA Modules

Google(谷歌)

Ivan Vulić, Adam Grycner, Quentin de Laroussilhe, Jonas Pfeiffer

AI总结:

LoRA-Squeeze通过动态调整LoRA模块秩,提升参数高效微调效果,实验表明事后压缩和渐进式秩退火方法在性能与效率之间取得最佳平衡。

摘要:

尽管存在大量的变种,标准低秩适应(LoRA)仍然是参数高效微调(PEFT)的主导技术。然而,它仍然面临持续的挑战,包括最优秩的预选择以及特定秩的超参数设置,以及异构秩模块的部署复杂性和更复杂的LoRA衍生技术。在本文中,我们介绍了LoRA-Squeeze,一种简单且高效的方法,旨在通过在事后或训练过程中动态改变LoRA模块的秩来改进标准LoRA的学习。我们的方法认为,先学习一个表达能力强、高秩的解决方案,然后进行压缩,比直接学习一个受限的低秩解决方案更好。该方法涉及使用故意较高的源秩进行微调,重建或高效近似完整权重更新矩阵的重建,并随后使用随机奇异值分解(RSVD)创建一个在较低目标秩下的新压缩LoRA模块。在13个文本和10个视觉-语言任务上进行的广泛实验表明,事后压缩通常会产生优于在目标秩上直接训练的低秩适配器,尤其是如果允许少量在目标秩上的微调步骤。此外,LoRA-Squeeze的渐进式、在调优中秩退火变体始终实现了最佳的LoRA大小-性能权衡。

英文摘要:

Despite its huge number of variants, standard Low-Rank Adaptation (LoRA) is still a dominant technique for parameter-efficient fine-tuning (PEFT). Nonetheless, it faces persistent challenges, including the pre-selection of an optimal rank and rank-specific hyper-parameters, as well as the deployment complexity of heterogeneous-rank modules and more sophisticated LoRA derivatives. In this work, we introduce LoRA-Squeeze, a simple and efficient methodology that aims to improve standard LoRA learning by changing LoRA module ranks either post-hoc or dynamically during training}. Our approach posits that it is better to first learn an expressive, higher-rank solution and then compress it, rather than learning a constrained, low-rank solution directly. The method involves fine-tuning with a deliberately high(er) source rank, reconstructing or efficiently approximating the reconstruction of the full weight update matrix, and then using Randomized Singular Value Decomposition (RSVD) to create a new, compressed LoRA module at a lower target rank. Extensive experiments across 13 text and 10 vision-language tasks show that post-hoc compression often produces lower-rank adapters that outperform those trained directly at the target rank, especially if a small number of fine-tuning steps at the target rank is allowed. Moreover, a gradual, in-tuning rank annealing variant of LoRA-Squeeze consistently achieves the best LoRA size-performance trade-off.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑