Post-training quantization (PTQ) has played a key role in compressing large language models (LLMs) with ultra-low costs. However, existing PTQ methods only focus on handling the outliers within one layer or one block, which ignores the dependency of blocks and leads to severe performance degradation in low-bit settings. In this paper, we propose CBQ, a cross-block reconstruction-based PTQ method for LLMs. CBQ employs a cross-block dependency using a homologous reconstruction scheme, establishing long-range dependencies across multiple blocks to minimize error accumulation. Furthermore, CBQ incorporates a coarse-to-fine preprocessing (CFP) strategy for suppressing weight and activation outliers, coupled with an adaptive LoRA-Rounding technique for precise weight quantization. These innovations enable CBQ to not only handle extreme outliers effectively but also improve overall quantization accuracy. Extensive experiments show that CBQ achieves superior low-bit quantization (W4A4, W4A8, W2A16) and outperforms existing state-of-the-art methods across various LLMs and datasets. Notably, CBQ quantizes the 4-bit LLAMA1-65B model within only 4.3 hours on a single GPU, achieving a commendable tradeoff between performance and quantization efficiency.
Department of Computer Science and Technology, University of Cambridge(计算机科学与技术系,剑桥大学);Department of Electrical & Electronic Engineering, Imperial College London(电气与电子工程系,伦敦帝国学院);Department of Computer Science, University of Hong Kong(计算机科学系,香港大学);School of Informatics, University of Edinburgh(信息学院,爱丁堡大学)
Block Rotation is All You Need for MXFP4 Quantization
Nanjing University of Science and Technology(南京理工大学);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Zhongguancun Academy(中关村学院);School of Computer Science, University of Sydney(悉尼大学计算机科学学院)
Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
Institute for Artificial Intelligence, Peking University, Beijing, China(北京大学人工智能研究院);Yuanpei College, Peking University, Beijing, China(北京大学元培学院);State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China(通用人工智能国家重点实验室);Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学(深圳))
Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inference
KlingAI Research(可灵AI研究院);Fudan University(复旦大学);Shanghai AI Incubation and Innovation Center(上海人工智能孵化与创新中心);Shanghai Academy of AI for Science(上海人工智能科学研究院);University of Science and Technology of China(中国科学技术大学)
G$^2$PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation
ZTE Corporation(中兴通讯);The Chinese University of Hong Kong(香港中文大学);Northwestern Polytechnical University(西北工业大学);Peking University(北京大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学)
DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs
Institute of Automation, CAS(中国科学院自动化研究所);City University of Hong Kong(香港城市大学);Tsinghua University(清华大学);Zhejiang University(浙江大学);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
HAFLQ: Heterogeneous Adaptive Federated LoRA Fine-tuned LLM with Quantization
Department of Engineering, King’s College London(伦敦国王学院工程系);Advanced Technology Group, Ericsson Inc.(爱立信公司先进科技组);Institute for Digital Communication, Friedrich-Alexander-Universität Erlangen-Nürnberg(埃尔兰根-纽伦堡弗里德里希-亚历山大大学数字通信研究所)
LoPRo: Enhancing Low-Rank Quantization via Permuted Block-Wise Rotation
Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing University of Aeronautics and Astronautics(北京航空航天大学)
Improving Quantization-aware Training of Low-Precision Network via Block Replacement on Full-Precision Counterpart
College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院);ZJU-UIUC Institute, Zhejiang University(浙江大学-伊利诺伊大学厄巴纳-香槟分校联合研究院)
Pushing the Limits of Block Rotations in Post-Training Quantization
Advanced Micro Devices, Inc. (AMD)(Advanced Micro Devices公司);State University of New York at Buffalo(纽约州立大学布法罗分校);Norwegian University of Science(挪威科学与技术大学)