arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.11316cs.CLcs.AI
收藏

PRILoRA:剪枝与秩递增的低秩适配

PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation

Tel Aviv University(特拉维夫大学)

Nadav Benedek, Lior Wolf

AI总结:

PRILoRA通过为每层递增分配秩并在训练中剪枝,解决了LoRA忽略层间重要性差异的问题,在八个GLUE基准上达到新最优。

摘要:

随着大型预训练语言模型(PLMs)的激增,微调所有模型参数变得日益低效,尤其是在处理众多下游任务时,这会带来大量的训练和存储成本。已有多种旨在实现参数高效微调(PEFT)的方法被提出。其中,低秩适配(LoRA)作为一种典型方法脱颖而出,它将可训练的秩分解矩阵融入每个目标模块。然而,LoRA并未考虑各层重要性的差异。为解决这些问题,我们提出了PRILoRA,它按递增方式为每一层线性分配不同的秩,并在整个训练过程中进行剪枝,同时考虑权重的临时幅度和任意给定层输入的累积统计信息。我们通过在八个GLUE基准上的大量实验验证了PRILoRA的有效性,并取得了新的最先进成果。

英文摘要:

With the proliferation of large pre-trained language models (PLMs), fine-tuning all model parameters becomes increasingly inefficient, particularly when dealing with numerous downstream tasks that entail substantial training and storage costs. Several approaches aimed at achieving parameter-efficient fine-tuning (PEFT) have been proposed. Among them, Low-Rank Adaptation (LoRA) stands out as an archetypal method, incorporating trainable rank decomposition matrices into each target module. Nevertheless, LoRA does not consider the varying importance of each layer. To address these challenges, we introduce PRILoRA, which linearly allocates a different rank for each layer, in an increasing manner, and performs pruning throughout the training process, considering both the temporary magnitude of weights and the accumulated statistics of the input to any given layer. We validate the effectiveness of PRILoRA through extensive experiments on eight GLUE benchmarks, setting a new state of the art.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑