arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.05926cs.LGcs.CL
收藏

SVFit:利用奇异值对大型预训练模型进行参数高效微调

SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Values

Chengwei Sun, Jiwei Wei, Yujia Wu, Yiming Shi, Shiyuan He, Zeyu Ma, Ning Xie, Yang Yang

AI总结:

针对LoRA等PEFT方法随机初始化低秩矩阵导致优化低效的问题,本文提出SVFit,利用SVD和关键奇异值初始化并训练低秩适配参数,在多类任务上以少16倍参数超越LoRA。

摘要:

大型预训练模型(LPM)已在多种自然语言处理和计算机视觉任务中展现出卓越性能。然而,对这些模型进行全量微调会带来巨大的内存挑战,尤其是在资源受限的环境中。LoRA等参数高效微调(PEFT)方法通过仅调整一小部分参数来缓解这一问题。尽管如此,这些方法通常对低秩矩阵采用随机初始化,这会因次优起点而导致梯度下降效率低下,并降低泛化能力。为解决这些局限,我们提出SVFit,这是一种新颖的PEFT方法,利用奇异值分解(SVD)初始化低秩矩阵,并将关键奇异值作为可训练参数。具体而言,SVFit对预训练权重矩阵执行SVD以获得最佳秩-r近似矩阵,重点关注捕捉矩阵99%以上信息的最关键奇异值。随后,这些排名靠前的r个奇异值被用作可训练参数,以缩放矩阵的基本子空间,从而促进快速领域适应。在自然语言理解、文本到图像生成和图像分类任务中,针对多种预训练模型开展的大量实验表明,SVFit优于LoRA,同时所需可训练参数减少16倍。

英文摘要:

Large pre-trained models (LPMs) have demonstrated exceptional performance in diverse natural language processing and computer vision tasks. However, fully fine-tuning these models poses substantial memory challenges, particularly in resource-constrained environments. Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, mitigate this issue by adjusting only a small subset of parameters. Nevertheless, these methods typically employ random initialization for low-rank matrices, which can lead to inefficiencies in gradient descent and diminished generalizability due to suboptimal starting points. To address these limitations, we propose SVFit, a novel PEFT approach that leverages singular value decomposition (SVD) to initialize low-rank matrices using critical singular values as trainable parameters. Specifically, SVFit performs SVD on the pre-trained weight matrix to obtain the best rank-r approximation matrix, emphasizing the most critical singular values that capture over 99% of the matrix's information. These top-r singular values are then used as trainable parameters to scale the fundamental subspaces of the matrix, facilitating rapid domain adaptation. Extensive experiments across various pre-trained models in natural language understanding, text-to-image generation, and image classification tasks reveal that SVFit outperforms LoRA while requiring 16 times fewer trainable parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑