arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.14482cs.CL
收藏

NLoRA:面向大语言模型的 Nyström 初始化低秩适配

NLoRA: Nyström-Initiated Low-Rank Adaptation for Large Language Models

School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);International Center of Future Science, Jilin University(吉林大学未来科学国际中心)

Chenlu Guo, Yuan Wu, Yi Chang

AI总结:

针对 LoRA 收敛慢且 SVD 初始化昂贵的问题,提出 SLoRA、Nyström 初始化的 NLoRA 及仅调中间矩阵的 IntTune,在 NLG/NLU 任务上以极少参数显著提升性能。

摘要:

参数高效微调(PEFT)对于适配大语言模型(LLM)至关重要,其中低秩适配(LoRA)是最常用的方法。然而,LoRA 存在收敛缓慢的问题,而 PiSSA 等近期的 LoRA 变体主要依赖奇异值分解(SVD)进行初始化,导致计算成本高昂。为缓解这些问题,我们采用遵循三矩阵操作的 Nyström 方法。首先,我们提出 StructuredLoRA(SLoRA),研究在低秩矩阵 A 和 B 之间加入一个小型中间矩阵。其次,我们提出 NyströmLoRA(NLoRA),利用基于 Nyström 的初始化来初始化 SLoRA,以提升其有效性和效率。最后,我们提出 IntermediateTune(IntTune),探索仅微调 NLoRA 的中间矩阵,以进一步提高 LLM 效率。我们在五个自然语言生成(NLG)任务和八个自然语言理解(NLU)任务上评估了所提方法。在 GSM8K 上,SLoRA 和 NLoRA 分别取得 56.48% 和 57.70% 的准确率,比 LoRA 高出 33.52% 和 36.41%,且仅增加 367 万个可训练参数。IntTune 仅使用 LoRA 1.25% 的参数,就使平均 NLG 性能较 LoRA 提升 7.45%。这些结果证明了我们的方法能够以极小的参数开销提升模型性能的效率和有效性。

英文摘要:

Parameter-efficient fine-tuning (PEFT) is essential for adapting large language models (LLMs), with low-rank adaptation (LoRA) being the most popular approach. However, LoRA suffers from slow convergence, and some recent LoRA variants, such as PiSSA, primarily rely on Singular Value Decomposition (SVD) for initialization, leading to expensive computation. To mitigate these problems, we use the Nyström method, which follows a three-matrix manipulation. We first introduce StructuredLoRA (SLoRA), which investigates adding a small intermediate matrix between the low-rank matrices A and B. Secondly, we propose NyströmLoRA (NLoRA), which leverages Nyström-based initialization for SLoRA to improve its effectiveness and efficiency. Finally, we propose IntermediateTune (IntTune), which explores fine-tuning exclusively on the intermediate matrix of NLoRA to further boost LLM efficiency. We evaluate our methods on five natural language generation (NLG) tasks and eight natural language understanding (NLU) tasks. On GSM8K, SLoRA and NLoRA achieve accuracies of 56.48% and 57.70%, surpassing LoRA by 33.52% and 36.41%, with only 3.67 million additional trainable parameters. IntTune improves average NLG performance over LoRA by 7.45% while using only 1.25% of its parameters. These results demonstrate the efficiency and effectiveness of our approach in enhancing model performance with minimal parameter overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑