Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?
Beijing Institute of Technology(北京理工大学);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);MMLab, The Chinese University of Hong Kong(香港中文大学MMLab);Hebei Province Key Laboratory of Big Data Science and Intelligent Technology(河北省大数据科学与智能技术重点实验室)
Xi Chen, Kaituo Feng, Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, Guoren Wang
Low-rank training has emerged as a promising approach for reducing memory usage in training Large Language Models (LLMs). Previous methods either rely on decomposing weight matrices (e.g., LoRA), or seek to decompose gradient matrices (e.g., GaLore) to ensure reduced memory consumption. However, both of them constrain the training in a low-rank subspace, thus inevitably leading to sub-optimal performance. This raises a question: whether it is possible to consistently preserve the low-rank constraint for memory efficiency, while achieving full-rank training (i.e., training with full-rank gradients of full-rank weights) to avoid inferior outcomes? In this paper, we propose a new plug-and-play training framework for LLMs called Fira, as the first attempt to achieve this goal. First, we observe an interesting phenomenon during LLM training: the scaling impact of adaptive optimizers (e.g., Adam) on the gradient norm remains similar from low-rank to full-rank training. Based on this observation, we propose a norm-based scaling method, which utilizes the scaling impact of low-rank optimizers as substitutes for that of original full-rank optimizers to enable full-rank training. In this way, we can preserve the low-rank constraint in the optimizer while achieving full-rank training for better performance. Moreover, we find that there are sudden gradient rises during the optimization process, potentially causing loss spikes. To address this, we further put forward a norm-growth limiter to smooth the gradient via regulating the relative increase of gradient norms. Extensive experiments on the pre-training and fine-tuning of LLMs show that Fira outperforms both LoRA and GaLore, achieving performance that is comparable to or even better than full-rank training.
Memory-Efficient LLM Pretraining via Minimalist Optimizer Design
Department of Electrical and Computer Engineering, University of Minnesota, USA(电气与计算机工程系,明尼苏达大学,美国);School of Mathematics and Statistics, University of Sydney, Australia(数学与统计学学院,悉尼大学,澳大利亚)
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Chinese Academy of Sciences(中国科学院大学);CAS Key Laboratory of AI Safety, Institute of Computing Technology, CAS(中国科学院人工智能安全重点实验室);University of Science and Technology of China(中国科学技术大学);Harbin Institute of Technology(哈尔滨工业大学);University of Oxford(牛津大学);Tsinghua University(清华大学)
Continual Gradient Low-Rank Projection Fine-Tuning for LLMs
Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence(北京交通大数据挖掘与具身智能重点实验室);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院);State Key Laboratory of Advanced Rail Autonomous Operation(先进轨道交通自主运行国家重点实验室)
Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMs
Theoretical Division, Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室理论分部);Advanced Research in Cyber Systems, Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室网络系统高级研究分部);University of Texas at Arlington(德克萨斯大学阿灵顿分校)
MGAA: Multi-Granular Adaptive Allocation fof Low-Rank Compression of LLMs
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);School of Computer Science and Cyber Engineering, Guangzhou University(广州大学计算机科学与网络工程学院)
Enabling Weak LLMs to Judge Response Reliability via Meta Ranking
Tsinghua University(清华大学);Weiyang College, Tsinghua University(清华大学未央书院);Institute of Intelligent Computing, Alibaba Group(阿里巴巴集团智能计算研究所);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能国际研究院)
Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs
Nanyang Technological University(南洋理工大学);The Chinese University of Hong Kong(香港中文大学);Data61;CSIRO(联邦科学与工业研究组织);Tsinghua University(清华大学);Australian National University(澳大利亚国立大学)
AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning
Department of Electrical Engineering-Systems at Tel Aviv University(特拉维夫大学电子工程系统系);Faculty of Engineering, Bar Ilan University(巴伊兰大学工程学院);Mathematical Institute, University of Oxford(牛津大学数学学院)
RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty
Lehigh University(莱维大学);University of Notre Dame(诺特大学);Zhejiang Wanli University(浙江万里大学);Squirrel Ai Learning;City University of Hong Kong(香港城市大学);Duke University(杜克大学)
Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking
Department of Electrical and Computer Engineering, University of Minnesota(明尼苏达大学电气与计算机工程系);Peking University(北京大学);School of Mathematics and Statistics, University of Sydney(悉尼大学数学与统计学学院)
QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation
NSF AI Institute for Artificial Intelligence and Fundamental Interactions(NSF人工智能与基础交互AI研究所);Massachusetts Institute of Technology(麻省理工学院);Harvard University(哈佛大学)