Large language models (LLMs) are omnipresent, however their practical deployment is challenging due to their ever increasing computational and memory demands. Quantization is one of the most effective ways to make them more compute and memory efficient. Quantization-aware training (QAT) methods, generally produce the best quantized performance, however it comes at the cost of potentially long training time and excessive memory usage, making it impractical when applying for LLMs. Inspired by parameter-efficient fine-tuning (PEFT) and low-rank adaptation (LoRA) literature, we propose LR-QAT -- a lightweight and memory-efficient QAT algorithm for LLMs. LR-QAT employs several components to save memory without sacrificing predictive performance: (a) low-rank auxiliary weights that are aware of the quantization grid; (b) a downcasting operator using fixed-point or double-packed integers and (c) checkpointing. Unlike most related work, our method (i) is inference-efficient, leading to no additional overhead compared to traditional PTQ; (ii) can be seen as a general extended pretraining framework, meaning that the resulting model can still be utilized for any downstream task afterwards; (iii) can be applied across a wide range of quantization settings, such as different choices quantization granularity, activation quantization, and seamlessly combined with many PTQ techniques. We apply LR-QAT to LLaMA-1/2/3 and Mistral model families and validate its effectiveness on several downstream tasks. Our method outperforms common post-training quantization (PTQ) approaches and reaches the same model performance as full-model QAT at the fraction of its memory usage. Specifically, we can train a 7B LLM on a single consumer grade GPU with 24GB of memory. Our source code is available at https://github.com/qualcomm-ai-research/LR-QAT
SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University（应用信息学与应用数学博士学院，奥布达大学）；John von Neumann Faculty of Informatics, Obuda University（约翰·冯·诺依曼信息学院，奥布达大学）
AFPQ: Asymmetric Floating Point Quantization for LLMs
Shanghai Jiao Tong University（上海交通大学）；Microsoft Research Asia（微软亚洲研究院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Science and Technology of China（中国科学技术大学）
Achieving binary weight and activation for LLMs using Post-Training Quantization
MAIS, Institute of Automation, Chinese Academy of Sciences（自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Central Media Technology Institute, Huawei Technologies Ltd.（华为技术有限公司媒体技术研究所）
Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition
Department of Computer Science, Yonsei University（yonsei大学计算机科学系）；Department of Artificial Intelligence, Yonsei University（yonsei大学人工智能系）；Department of Electronic & Electrical Convergence Engineering, Hongik University（hongik大学电子与电气融合工程系）
GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
University of Pittsburgh（匹兹堡大学）；University of Central Florida（佛罗里达州立大学）；University of Arizona（亚利桑那大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
The Impact of Quantization on Retrieval-Augmented Generation: An Analysis of Small LLMs
Amsterdam University of Applied Sciences（阿姆斯特丹应用科学大学）；Leiden University（莱顿大学）；Peoples' Friendship University of Russia (RUDN University)（俄罗斯人民友谊大学（鲁登大学））；Joint Institute for Nuclear Research（联合核子研究所）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；University of Skövde（舍夫德大学）
Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
Institute for Artificial Intelligence, Peking University, Beijing, China（北京大学人工智能研究院）；Yuanpei College, Peking University, Beijing, China（北京大学元培学院）；State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China（通用人工智能国家重点实验室）；Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））
