Post-training quantization of Large Language Models (LLMs) is challenging. In this work, we introduce Low-rank Quantization Error Reduction (LQER), which combines quantization and low-rank approximation to recover the model capability. LQER leverages an activation-induced scale matrix to drive the singular value distribution of quantization error towards a desirable distribution, which enables nearly-lossless W4A8 quantization on various LLMs and downstream tasks without the need for knowledge distillation, grid search, or gradient-base iterative optimization. Unlike existing methods, the computation pattern of LQER eliminates the need for specialized Scatter and Gather processes to collect high-precision weights from irregular memory locations. Our W4A8 LLMs achieve near-lossless performance on six popular downstream tasks, while using 1.36$\times$ fewer hardware resources than the leading state-of-the-art method. We open-source our framework at https://github.com/ChengZhang-98/lqer
Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition
Department of Computer Science, Yonsei University（yonsei大学计算机科学系）；Department of Artificial Intelligence, Yonsei University（yonsei大学人工智能系）；Department of Electronic & Electrical Convergence Engineering, Hongik University（hongik大学电子与电气融合工程系）
VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization
LUMIA Lab（LUMIA实验室）；School of Artificial Intelligence（人工智能学院）；Shanghai Jiao Tong University（上海交通大学）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Cambridge（剑桥大学）；National University of Singapore（新加坡国立大学）
A Comprehensive Evaluation on Quantization Techniques for Large Language Models
School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；School of Engineering Math and Technology, University of Bristol（布里斯托大学工程数学与技术学院）
LoPRo: Enhancing Low-Rank Quantization via Permuted Block-Wise Rotation
Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing University of Aeronautics and Astronautics（北京航空航天大学）
Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
Institute for Artificial Intelligence, Peking University, Beijing, China（北京大学人工智能研究院）；Yuanpei College, Peking University, Beijing, China（北京大学元培学院）；State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China（通用人工智能国家重点实验室）；Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））
LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning
Carnegie Mellon University（卡内基梅隆大学）；Columbia University（哥伦比亚大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Petuum Inc.（Petuum公司）；Massachusetts Institute of Technology（麻省理工学院）
GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation
Department of Mathematics, University of California San Diego（数学系，加州大学圣地亚哥分校）；Department of Mathematics and Halıcıoğlu Data Science Institute, University of California San Diego（数学系和Halıcıoğlu数据科学研究所，加州大学圣地亚哥分校）
D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank Approximation
Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））；TeleAI of China Telecom（中国电信TeleAI）；The Hong Kong Polytechnic University（香港理工大学）；The University of Hong Kong（香港大学）
Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization
Nanjing University of Posts and Telecommunications（南京邮电大学）；Tylogi AI Lab / TAIL（Tylogi AI实验室 / TAIL）；Anhui University of Technology（安徽工业大学）；Shanghai Jiao Tong University（上海交通大学）；Southeast University（东南大学）；Microsoft（微软）；University of Cambridge（剑桥大学）
Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data
