Training Large Language Models (LLMs) is memory-intensive due to the large number of parameters and associated optimization states. GaLore, a recent method, reduces memory usage by projecting weight gradients into a low-rank subspace without compromising performance. However, GaLore relies on time-consuming Singular Value Decomposition (SVD) operations to identify the subspace, and the frequent subspace updates lead to significant training time overhead. Moreover, GaLore offers minimal improvements in accuracy and efficiency compared to LoRA in more accessible fine-tuning scenarios. To address these limitations, we introduce Q-Galore, a novel approach that substantially reduces memory usage by combining quantization and low-rank projection, surpassing the benefits of GaLore. Our method is based on two key observations: (i) the gradient subspace exhibits diverse properties, with some layers converging early in training while others are subject to frequent changes; (ii) the projection matrices are highly resilient to low-bit quantization. Leveraging these insights, Q-GaLore adaptively updates the gradient subspace based on its convergence statistics, achieving comparable performance while significantly reducing the number of SVD operations. We maintain the projection matrices in INT4 format and weights in INT8 format, incorporating stochastic rounding to capture accumulated gradient information. This approach enables a high-precision training trajectory using only low-precision weights. We demonstrate that Q-GaLore achieves highly competitive performance with exceptional memory efficiency. At pre-training, Q-GaLore facilitates training a LLaMA-7B model from scratch on a single NVIDIA RTX 4060 Ti with only 16 GB memory. At fine-tuning, it reduces memory consumption by up to 50% compared to LoRA and GaLore, while consistently outperforming QLoRA at the same memory cost.
FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training
Moscow Institute of Physics and Technology, Russia（莫斯科物理技术学院）；Ivannikov Institute for System Programming RAS, Russia（伊万诺夫系统编程研究所）；Skolkovo Institute of Science and Technology, Russia（斯克尔罗沃科学与技术研究所）；Mohamed bin Zayed University of Artificial Intelligence, UAE（穆罕默德·本·扎耶德人工智能大学）
本文提出 FRUGAL 框架，通过梯度拆分实现全秩更新，在低维方向使用 Adam 等高内存优化器、其余方向使用 SGD 等无状态方法，从而在固定内存预算下提升预训练和微调性能。
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Sichuan University（四川大学）；Beijing Zhongguancun Academy（北京中关村科学院）；Zhejiang University（浙江大学）；Beijing Key Laboratory of Brain-Inspired General Intelligence Large Model（北京脑科学与类脑研究中心通用人工智能大模型北京市重点实验室）；Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology（脑认知与脑机智能技术重点实验室）
Subspace Optimization for Large Language Models with Convergence Guarantees
Peking University（北京大学）；Zhongguancun Academy（中关村学院）；Beihang University（北航大学）；AI for Science Institute, Beijing, China（人工智能科学研究院，北京，中国）；National Engineering Laboratory for Big Data Analytics（大数据分析与应用国家工程实验室）
Lotus: Efficient LLM Training by Randomized Low-Rank Gradient Projection with Adaptive Subspace Switching
Hong Kong Baptist University, School of Science, Hong Kong, China（香港 Baptist 大学，科学学院，香港，中国）；Fudan University, School of Data Science, Shanghai, China（复旦大学，数据科学学院，上海，中国）；New York University, Tandon School of Engineering, New York, USA（纽约大学，Tandon 工程学院，纽约，美国）
Forget by Uncertainty: Orthogonal Entropy Unlearning for Quantized Neural Networks
Hubei Key Laboratory of Transportation Internet of Things, School of Computer Science and Artificial Intelligence, Wuhan University of Technology, China（交通物联网湖北重点实验室，计算机科学与人工智能学院，武汉理工大学，中国）；College of Computing and Data Science, Nanyang Technological University, Singapore（计算与数据科学学院，南洋理工大学，新加坡）
On Orthogonal Recurrent Neural Networks Quantization
Institut de Mathématiques de Toulouse（图卢兹数学研究所）；Université de Toulouse（图卢兹大学）；CNRS（法国国家科学研究中心）；UPS IMT；Institut de Recherche Technologique Saint Exupéry（圣埃克苏佩里技术研究院）
Decentralized Optimization on Compact Submanifolds by Quantized Riemannian Gradient Tracking
National Special Education Resource Center for Children with Autism, Zhejiang Normal University（浙江师范大学生自闭症教育资源中心）；School of Computer Science and Technology, Zhejiang Normal University（浙江师范大学计算机科学与技术学院）；China Mobile Research Institute（中国移动研究院）；Institute of Cyber-Systems and Control, Zhejiang University（浙江大学控制系统研究所）；SGIT AI Lab, State Grid Corporation of China（国网SGIT人工智能实验室）；Zhejiang Key Laboratory of Intelligent Education Technology and Application, Zhejiang Normal University（浙江智能教育技术与应用重点实验室）；School of Information Engineering, Huzhou University（湖州大学信息工程学院）；Centre for Frontier Artificial Intelligence Research, A*STAR（A*STAR前沿人工智能研究中心）
Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation
Hong Kong University of Science and Technology（香港科技大学）；Carnegie Mellon University（卡内基梅隆大学）；Reality Labs, Meta Inc.（Meta公司现实实验室）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
