arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.09353cs.CLcs.CV
收藏

DoRA:权重分解低秩适配

DoRA: Weight-Decomposed Low-Rank Adaptation

NVIDIA(英伟达);HKUST(香港科技大学)

Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen

AI总结:

针对LoRA等参数高效微调方法与全量微调存在精度差距的问题,提出权重分解低秩适配DoRA,通过分解预训练权重的幅值与方向、用LoRA更新方向分量,在无额外推理开销下提升学习能力与稳定性,多任务多模型上性能优于LoRA。

摘要:

在广泛使用的参数高效微调(PEFT)方法中,LoRA及其变体因无需额外推理成本而广受欢迎。然而,这些方法与全量微调(FT)之间往往仍存在精度差距。在本研究中,我们首先引入一种新颖的权重分解分析,以探究FT与LoRA之间的内在差异。基于研究发现,为了模拟FT的学习能力,我们提出了权重分解低秩适配(Weight-Decomposed Low-Rank Adaptation, DoRA)。DoRA将预训练权重分解为幅值和方向两个分量进行微调,特别采用LoRA进行方向更新,以高效减少可训练参数数量。通过使用DoRA,我们在避免任何额外推理开销的同时,提升了LoRA的学习能力和训练稳定性。在常识推理、视觉指令调优、图像/视频-文本理解等多种下游任务上,对LLaMA、LLaVA和VL-BART进行微调时,DoRA的性能始终优于LoRA。代码可在https://github.com/NVlabs/DoRA获取。

英文摘要:

Among the widely used parameter-efficient fine-tuning (PEFT) methods, LoRA and its variants have gained considerable popularity because of avoiding additional inference costs. However, there still often exists an accuracy gap between these methods and full fine-tuning (FT). In this work, we first introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed Low-Rank Adaptation (DoRA). DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters. By employing \ours, we enhance both the learning capacity and training stability of LoRA while avoiding any additional inference overhead. \ours~consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding. Code is available at https://github.com/NVlabs/DoRA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑