arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.03044cs.LG
收藏

RepLoRA:从混合专家视角重新参数化低秩适配

RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts

The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Tuan Truong, Chau Nguyen, Huy Nguyen, Minh Le, Trung Le, Nhat Ho

AI总结:

本文从混合专家视角分析LoRA,提出用轻量级MLP重新参数化适配矩阵的RepLoRA,理论上将数据需求由指数降至多项式量级,并在有限数据下显著优于LoRA。

摘要:

低秩适配(LoRA)已成为微调大规模基础模型的一种强大方法。尽管其广受欢迎,对LoRA的理论理解仍然有限。本文通过考察LoRA与混合专家模型之间的联系,对其进行理论分析。在该框架下,我们表明对LoRA矩阵进行简单的重新参数化可以显著加速低秩矩阵估计过程。特别地,我们证明重新参数化可将达到期望估计误差所需的数据从指数量级减少到多项式量级。受此洞见启发,我们提出重新参数化低秩适配(RepLoRA),它结合轻量级MLP来重新参数化LoRA矩阵。跨多个领域的大量实验表明,RepLoRA始终优于原始LoRA。值得注意的是,在数据有限的情况下,RepLoRA以最高40.0%的幅度超过LoRA,并且仅用30.0%的训练数据即可达到LoRA的性能,突显了我们这种PEFT方法在理论和实证上的稳健性。

英文摘要:

Low-rank Adaptation (LoRA) has emerged as a powerful method for fine-tuning large-scale foundation models. Despite its popularity, the theoretical understanding of LoRA has remained limited. This paper presents a theoretical analysis of LoRA by examining its connection to the Mixture of Experts models. Under this framework, we show that simple reparameterizations of the LoRA matrices can notably accelerate the low-rank matrix estimation process. In particular, we prove that reparameterization can reduce the data needed to achieve a desired estimation error from an exponential to a polynomial scale. Motivated by this insight, we propose Reparameterized Low-Rank Adaptation (RepLoRA), which incorporates lightweight MLPs to reparameterize the LoRA matrices. Extensive experiments across multiple domains demonstrate that RepLoRA consistently outperforms vanilla LoRA. Notably, with limited data, RepLoRA surpasses LoRA by a margin of up to 40.0% and achieves LoRA's performance with only 30.0% of the training data, highlighting both the theoretical and empirical robustness of our PEFT method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑