arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.09044cs.CL
收藏

MiLoRA:利用次要奇异成分实现参数高效的LLM微调

MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning

Shanghai University of Finance and Economics(上海财经大学);Southern University of Science and Technology(南方科技大学);Tsinghua University(清华大学)

Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, Yun Chen

AI总结:

提出MiLoRA方法,通过仅更新权重矩阵的次要奇异成分并保持主要成分冻结,在正交子空间中学习以保留预训练知识,在多项推理和指令跟随基准上表现优越。

摘要:

大型语言模型(LLMs)的高效微调旨在以降低的计算和内存成本来适配LLMs。先前基于LoRA的方法使用高斯分布和零值初始化低秩矩阵,同时保持原始权重矩阵冻结。然而,在无引导子空间中优化的可训练模型参数可能会干扰预训练权重矩阵中已充分学习的子空间。在本文中,我们提出MiLoRA,这是一种简单而有效的LLM微调方法,它仅更新权重矩阵的次要奇异成分,同时保持主要奇异成分冻结。据观察,次要矩阵对应于噪声或长尾信息,而主要矩阵包含重要知识。MiLoRA在与主要矩阵正交的子空间内初始化低秩矩阵,因此预训练知识有望得到良好保留。在微调期间,MiLoRA充分利用较少优化的子空间来学习带标签的数据集。在常识推理、数学推理、指令跟随和视觉指令跟随基准上的广泛实验展示了我们方法的优越性能。

英文摘要:

Efficient finetuning of large language models (LLMs) aims to adapt the LLMs with reduced computational and memory cost. Previous LoRA-based approaches initialize the low-rank matrices with Gaussian distribution and zero values while keeping the original weight matrices frozen. However, the trainable model parameters optimized in an unguided subspace might interfere with the well-learned subspace of the pretrained weight matrices. In this paper, we propose MiLoRA, a simple yet effective LLM finetuning approach that only updates the minor singular components of the weight matrix while keeping the principal singular components frozen. It is observed that the minor matrix corresponds to the noisy or long-tail information, while the principal matrix contains important knowledge. The MiLoRA initializes the low-rank matrices within a subspace that is orthogonal to the principal matrix, thus the pretrained knowledge is expected to be well preserved. During finetuning, MiLoRA makes the most use of the less-optimized subspace for learning the labeled dataset. Extensive experiments on commonsense reasoning, math reasoning, instruction following and visual instruction following benchmarks present the superior performance of our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑