QLoRA reduces the memory-cost of fine-tuning a large language model (LLM) with LoRA by quantizing the base LLM. However, quantization introduces quantization errors that negatively impact model performance after fine-tuning. In this paper we introduce QuAILoRA, a quantization-aware initialization for LoRA that mitigates this negative impact by decreasing quantization errors at initialization. Our method spends a small amount of computational overhead to compute this quantization-aware initialization, without increasing the memory-cost of fine-tuning. We evaluate our method on several causal language modeling and downstream evaluation tasks using several different model sizes and families. We observe that almost all LLMs fined-tuned with QuAILoRA achieve better validation perplexity. When evaluated on downstream tasks, we find that QuAILoRA yields improvements proportional to the negative effect of quantization error. On average, applying QuAILoRA to 4-bit QLoRA models yields 75% of the validation perplexity decrease and 86% of the downstream task accuracy increase as doubling the quantization precision to 8-bit, without increasing GPU memory utilization during fine-tuning.
Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning
Department of Computer Science(计算机科学系);Information Engineering, National Cheng Kung University(信息工程系,国立成功大学);Department of Artificial Intelligence, Chang Gung University(人工智能系,长庚大学);Department of Computer Science, National Tsing Hua University(计算机科学系,国立清华大学)
Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Georgia Institute of Technology(佐治亚理工学院);Massachusetts Institute of Technology(麻省理工学院)
SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRA
Shanghai Qi Zhi Institute(上海启智研究院);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Xiongan AI Institute(雄安人工智能研究院)
Task-Specific Directions: Definition, Exploration, and Utilization in Parameter Efficient Fine-Tuning
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University(人工智能基础理论实验室,人工智能研究院,上海交通大学);School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(电子信息与电气工程学院,上海交通大学);School of Engineering and Applied Sciences, Harvard University(工程与应用科学学院,哈佛大学)
University of Science and Technology of China(中国科学技术大学);Institute of Intelligent Machines, HFIPS, Chinese Academy of Sciences(中国科学院智能机械研究所);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学);University of Sydney(悉尼大学);Anhui Disaster Warning & Agrometeorological Information Center(安徽省灾害预警与农业气象信息中心)
Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China(软件研究所智能软件研究中心,中国科学院,北京);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京)
A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Parameter Efficient Fine-tuning via Explained Variance Adaptation
ELLIS Unit, LIT AI Lab, Institute for Machine Learning, JKU Linz(ELLIS单位、LIT人工智能实验室、机器学习研究所、JKU林茨);University College London(伦敦大学学院);EMMI AI, Linz(EMMI AI、林茨);NXAI GmbH, Linz, Austria(NXAI GmbH、林茨、奥地利)
本文提出 EVA 初始化方法,以激活方差最大方向构造 LoRA 子空间并支持自适应秩,在多类任务上实现更快收敛、更高平均得分及更少可训练参数。
R-LoRA: Randomized Multi-Head LoRA for Efficient Multi-Task Learning
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(知识驱动人机智能工程研究中心,教育部,中国);International Center of Future Science, Jilin University(未来科学国际中心,吉林大学)
DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);University of International Business and Economics, China(对外经济贸易大学);XiaoMi AI Lab, China(小米AI实验室)
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(教育部知识驱动的人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际合作中心)
PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning
Peking University(北京大学);XiaoMi AI Lab(小米AI实验室);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);University of Electronic Science and Technology of China(电子科技大学)