Large pre-trained models (LPMs) have demonstrated exceptional performance in diverse natural language processing and computer vision tasks. However, fully fine-tuning these models poses substantial memory challenges, particularly in resource-constrained environments. Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, mitigate this issue by adjusting only a small subset of parameters. Nevertheless, these methods typically employ random initialization for low-rank matrices, which can lead to inefficiencies in gradient descent and diminished generalizability due to suboptimal starting points. To address these limitations, we propose SVFit, a novel PEFT approach that leverages singular value decomposition (SVD) to initialize low-rank matrices using critical singular values as trainable parameters. Specifically, SVFit performs SVD on the pre-trained weight matrix to obtain the best rank-r approximation matrix, emphasizing the most critical singular values that capture over 99% of the matrix's information. These top-r singular values are then used as trainable parameters to scale the fundamental subspaces of the matrix, facilitating rapid domain adaptation. Extensive experiments across various pre-trained models in natural language understanding, text-to-image generation, and image classification tasks reveal that SVFit outperforms LoRA while requiring 16 times fewer trainable parameters.
HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance
Harvard University(哈佛大学);University of Chinese Academy of Sciences(中国科学院大学);Fudan University(复旦大学);University of Chicago(芝加哥大学);University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);South China Normal University(华南师范大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai University of Engineering Science(上海工程技术大学)
Parameter Efficient Fine-tuning via Explained Variance Adaptation
ELLIS Unit, LIT AI Lab, Institute for Machine Learning, JKU Linz(ELLIS单位、LIT人工智能实验室、机器学习研究所、JKU林茨);University College London(伦敦大学学院);EMMI AI, Linz(EMMI AI、林茨);NXAI GmbH, Linz, Austria(NXAI GmbH、林茨、奥地利)
本文提出 EVA 初始化方法,以激活方差最大方向构造 LoRA 子空间并支持自适应秩,在多类任务上实现更快收敛、更高平均得分及更少可训练参数。
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室)
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning
King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));University of Sydney(悉尼大学)
Australian Centre for Robotics, School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney(澳大利亚机器人中心,航空航天、机械与机电工程学院,悉尼大学);Google DeepMind(谷歌DeepMind)
SSVD: Structured SVD for Parameter-Efficient Fine-Tuning and Benchmarking under Domain Shift in ASR
Department of Electrical Engineering KU Leuven Leuven, Belgium(电气工程系 卡尔·冯·特里尔大学 布鲁塞尔,比利时);Language Technologies Institute Carnegie Mellon University Pittsburgh, PA, USA(语言技术研究所 哈佛大学 贝尔蒙特,美国)
Two Stage Wireless Federated LoRA Fine-Tuning with Sparsified Orthogonal Updates
Department of Electrical and Computer Engineering, and the Institute of New Media and Communications, Seoul National University (SNU)(电气与计算机工程系,以及新媒体与通讯研究所,首尔国立大学)
Mila - Quebec AI Institute(魁北克AI研究所);Polytechnique Montréal(蒙特利尔理工学院);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);CIFAR AI Chair(CIFAR人工智能主席);Google DeepMind(谷歌DeepMind)
Mila - Quebec AI Institute(魁北克人工智能研究所);Polytechnique Montréal(蒙特利尔理工学院);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);CIFAR AI Chair(CIFAR人工智能主席);Google DeepMind(谷歌DeepMind)
CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室)
Robust Layerwise Scaling Rules by Proper Weight Decay Tuning
Department of Electrical Engineering and Computer Science, MIT, Cambridge, MA, USA(麻省理工学院电子工程与计算机科学系);Department of Computer Science, UCLA, CA, USA(加州大学洛杉矶分校计算机科学系)