arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.13604cs.CL
收藏

BeamLoRA:波束约束低秩适配

BeamLoRA: Beam-Constraint Low-Rank Adaptation

Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Baidu Inc.(百度公司)

Naibin Gu, Zhenyu Zhang, Xiyu Liu, Peng Fu, Zheng Lin, Shuohuan Wang, Yu Sun, Hua Wu, Weiping Wang, Haifeng Wang

AI总结:

针对LoRA微调准确率仍有提升空间的问题,提出BeamLoRA方法,将LoRA模块视为波束、秩对应子解,动态调整子解参数空间,在固定秩下提升性能,经多模型多数据集验证优于现有基线。

摘要:

鉴于大语言模型高效微调的需求,低秩适配(Low-Rank Adaptation, LoRA)作为最有效的参数高效微调方法之一已被广泛采用。然而,LoRA在提升效率的同时,准确率仍有提升空间。本文采用全新视角评估LoRA秩的特性,结果显示LoRA模块内的不同秩不仅重要程度各异,还会在微调过程中动态演化,这可能限制了LoRA的性能。基于上述发现,我们提出BeamLoRA,该方法将每个LoRA模块概念化为一个波束,其中每个秩自然对应一个潜在子解,微调过程则变为寻找最优子解组合的过程。BeamLoRA会动态剔除表现不佳的子解,同时为有潜力的子解扩展参数空间,在固定秩的条件下提升性能。我们在3个基座模型、12个涵盖数学推理、代码生成和常识推理的数据集上开展了大量实验,结果表明BeamLoRA持续提升了LoRA的性能,超越了其他基线方法。

英文摘要:

Due to the demand for efficient fine-tuning of large language models, Low-Rank Adaptation (LoRA) has been widely adopted as one of the most effective parameter-efficient fine-tuning methods. Nevertheless, while LoRA improves efficiency, there remains room for improvement in accuracy. Herein, we adopt a novel perspective to assess the characteristics of LoRA ranks. The results reveal that different ranks within the LoRA modules not only exhibit varying levels of importance but also evolve dynamically throughout the fine-tuning process, which may limit the performance of LoRA. Based on these findings, we propose BeamLoRA, which conceptualizes each LoRA module as a beam where each rank naturally corresponds to a potential sub-solution, and the fine-tuning process becomes a search for the optimal sub-solution combination. BeamLoRA dynamically eliminates underperforming sub-solutions while expanding the parameter space for promising ones, enhancing performance with a fixed rank. Extensive experiments across three base models and 12 datasets spanning math reasoning, code generation, and commonsense reasoning demonstrate that BeamLoRA consistently enhances the performance of LoRA, surpassing the other baseline methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑