arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.07558cs.CLcs.LG
收藏

DyLoRA:使用动态免搜索低秩适配对预训练模型进行参数高效微调

DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation

University of Waterloo(滑铁卢大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)

Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, Ali Ghodsi

AI总结:

本文提出DyLoRA方法,通过在一个秩范围内训练LoRA块,解决固定秩和穷举搜索问题,在GLUE及生成任务上实现比LoRA快4-7倍且性能稳定的参数高效微调。

摘要:

随着预训练模型(PMs)规模不断增大,对其进行微调已变得更加昂贵且耗费资源。作为一种补救方法,低秩适配器(LoRA)保持模型的主要预训练权重冻结,仅向模型中引入一些可学习的截断SVD模块(即所谓的LoRA块)。尽管LoRA块具有参数高效性,但它们存在两个主要问题:第一,这些块的大小是固定的,训练后无法修改(例如,如果需要改变LoRA块的秩,则必须从头重新训练它们);第二,优化其秩需要进行穷举搜索并耗费大量精力。在本工作中,我们引入一种动态低秩适配(DyLoRA)技术来同时解决这两个问题。我们的DyLoRA方法通过在训练期间对不同秩下适配器模块学习到的表示进行排序,从而针对一个秩范围而非单一秩来训练LoRA块。我们使用不同规模的预训练模型(如RoBERTa和GPT),在不同的自然语言理解(GLUE基准)和语言生成任务(E2E、DART和WebNLG)上评估了我们的解决方案。结果表明,借助DyLoRA,我们训练动态免搜索模型的速度至少比LoRA快4到7倍(具体取决于任务),且不会显著牺牲性能。此外,与LoRA相比,我们的模型能够在更大的秩范围内持续表现良好。

英文摘要:

With the ever-growing size of pretrained models (PMs), fine-tuning them has become more expensive and resource-hungry. As a remedy, low-rank adapters (LoRA) keep the main pretrained weights of the model frozen and just introduce some learnable truncated SVD modules (so-called LoRA blocks) to the model. While LoRA blocks are parameter-efficient, they suffer from two major problems: first, the size of these blocks is fixed and cannot be modified after training (for example, if we need to change the rank of LoRA blocks, then we need to re-train them from scratch); second, optimizing their rank requires an exhaustive search and effort. In this work, we introduce a dynamic low-rank adaptation (DyLoRA) technique to address these two problems together. Our DyLoRA method trains LoRA blocks for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training. We evaluate our solution on different natural language understanding (GLUE benchmark) and language generation tasks (E2E, DART and WebNLG) using different pretrained models such as RoBERTa and GPT with different sizes. Our results show that we can train dynamic search-free models with DyLoRA at least 4 to 7 times (depending to the task) faster than LoRA without significantly compromising performance. Moreover, our models can perform consistently well on a much larger range of ranks compared to LoRA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑