arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.03290cs.LGcs.CL
收藏

SARA:基于奇异值的自适应低秩适配

SARA: Singular-Value Based Adaptive Low-Rank Adaption

Beijing University of Posts and Telecommunications(北京邮电大学)

Jihao Gu, Shuai Chen, Zelin Wang, Yibo Zhang, Ping Gong

AI总结:

本文提出SARA方法,在初始化时通过对预训练权重做SVD自适应确定各层低秩适配的秩,并进一步提出Mo-SARA通过路由器仅微调多组并行奇异值以大幅减少参数量,实验验证了其简洁性与参数效率。

摘要:

随着大型预训练模型参数数量的不断增加,LoRA作为一种参数高效微调(PEFT)方法,因不增加推理开销而被广泛使用。LoRA方法假设微调过程中的权重变化可以用低秩矩阵来近似。然而,秩值需要人工验证以匹配不同的下游任务,并且它们无法适应模型中不同层的重要性差异。在本工作中,我们首先利用SVD分析了不同层的性能与其秩之间的关系。在此基础上,我们设计了基于奇异值的自适应低秩适配(SARA),该方法在初始化阶段通过对预训练权重执行SVD来自适应地确定秩。此外,我们探索了Mixture-of-SARA(Mo-SARA),通过仅微调由路由器控制的多组并行奇异值,显著减少了参数数量。在各种复杂任务上的大量实验证明了我们方法的简洁性和参数效率。它们能够有效且自适应地为每个模型的每一层找到最合适的秩。

英文摘要:

With the increasing number of parameters in large pre-trained models, LoRA as a parameter-efficient fine-tuning(PEFT) method is widely used for not adding inference overhead. The LoRA method assumes that weight changes during fine-tuning can be approximated by low-rank matrices. However, the rank values need to be manually verified to match different downstream tasks, and they cannot accommodate the varying importance of different layers in the model. In this work, we first analyze the relationship between the performance of different layers and their ranks using SVD. Based on this, we design the Singular-Value Based Adaptive Low-Rank Adaption(SARA), which adaptively finds the rank during initialization by performing SVD on the pre-trained weights. Additionally, we explore the Mixture-of-SARA(Mo-SARA), which significantly reduces the number of parameters by fine-tuning only multiple parallel sets of singular values controlled by a router. Extensive experiments on various complex tasks demonstrate the simplicity and parameter efficiency of our methods. They can effectively and adaptively find the most suitable rank for each layer of each model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑