稀疏性或许已足够:稀疏随机参数适配
Sparsity May Be All You Need: Sparse Random Parameter Adaptation
AI总结:
本文提出稀疏随机参数适配方法,仅随机训练少量模型参数,并证明其性能可与LoRA媲美,表明PEFT效果关键在于可训练参数量而非适配器结构。
摘要:
随着模型规模不断增大,针对对齐和任务适配对大语言模型进行全参数微调已变得极其昂贵。参数高效微调(PEFT)方法旨在通过仅训练少量参数而非全部模型参数,显著降低微调所需的计算和内存资源。目前最流行的PEFT方法是低秩适配(LoRA),它冻结模型参数,并以低秩矩阵的形式引入少量可训练参数。我们提出一种简单方法:随机选择一小部分模型参数进行训练,同时固定所有其他参数,不附加任何诸如低秩结构之类的先验假设。在本文中,我们将所提方法的效率和性能与其他PEFT方法以及全参数微调进行比较。我们发现,在使用相近数量的可训练参数时,我们的方法与LoRA具有竞争力。我们的研究结果表明,对于PEFT技术而言,真正重要的不一定是特定的适配器结构,而是所使用的可训练参数数量。
英文摘要:
Full fine-tuning of large language models for alignment and task adaptation has become prohibitively expensive as models have grown in size. Parameter-Efficient Fine-Tuning (PEFT) methods aim at significantly reducing the computational and memory resources needed for fine-tuning these models by only training on a small number of parameters instead of all model parameters. Currently, the most popular PEFT method is the Low-Rank Adaptation (LoRA), which freezes the parameters of the model and introduces a small set of trainable parameters in the form of low-rank matrices. We propose simply reducing the number of trainable parameters by randomly selecting a small proportion of the model parameters to train on, while fixing all other parameters, without any additional prior assumptions such as low-rank structures. In this paper, we compare the efficiency and performance of our proposed approach to other PEFT methods as well as full parameter fine-tuning. We find our method to be competitive with LoRA when using a similar number of trainable parameters. Our findings suggest that what truly matters for a PEFT technique to perform well is not necessarily the specific adapter structure, but rather the number of trainable parameters being used.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。