arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.17263cs.CL
收藏

MELoRA:用于参数高效微调的迷你集成低秩适配器

MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning

Shandong University(山东大学);Leiden University(莱顿大学);University of Amsterdam(阿姆斯特丹大学);Centrum Wiskunde & Informatica(数学与信息学研究中心)

Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, Jiahuan Pei

AI总结:

MELoRA通过冻结预训练权重并训练一组迷你低秩适配器,以更少参数保持高秩,在自然语言理解和指令跟随任务上分别以8倍和36倍更少的可训练参数超越LoRA,实现参数高效微调。

摘要:

参数高效微调(PEFT)是一种流行的针对预训练大语言模型(LLMs)进行定制的方法,尤其是在模型规模和任务多样性增加的情况下。低秩适配(LoRA)基于这样的思想:适配过程本质上是低维的,即显著的模型变化可以用相对较少的参数来表示。然而,与全参数微调相比,降低秩会遇到特定任务的泛化误差挑战。我们提出了MELoRA,一种迷你集成低秩适配器,它使用更少的可训练参数同时保持更高的秩,从而提供更好的性能潜力。核心思想是冻结原始预训练权重,并训练一组仅包含少量参数的迷你LoRA。这可以捕获迷你LoRA之间的显著多样性,从而促进更好的泛化能力。我们在各种NLP任务上进行了理论分析和实证研究。我们的实验结果表明,与LoRA相比,MELoRA在自然语言理解任务上以少8倍的可训练参数取得了更好的性能,在指令跟随任务上以少36倍的可训练参数取得了更好的性能,这证明了MELoRA的有效性。

英文摘要:

Parameter-efficient fine-tuning (PEFT) is a popular method for tailoring pre-trained large language models (LLMs), especially as the models' scale and the diversity of tasks increase. Low-rank adaptation (LoRA) is based on the idea that the adaptation process is intrinsically low-dimensional, i.e., significant model changes can be represented with relatively few parameters. However, decreasing the rank encounters challenges with generalization errors for specific tasks when compared to full-parameter fine-tuning. We present MELoRA, a mini-ensemble low-rank adapters that uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential. The core idea is to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters. This can capture a significant degree of diversity among mini LoRAs, thus promoting better generalization ability. We conduct a theoretical analysis and empirical studies on various NLP tasks. Our experimental results show that, compared to LoRA, MELoRA achieves better performance with 8 times fewer trainable parameters on natural language understanding tasks and 36 times fewer trainable parameters on instruction following tasks, which demonstrates the effectiveness of MELoRA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑