In this paper, we address post-training quantization (PTQ) for large language models (LLMs) from an overlooked perspective: given a pre-trained high-precision LLM, the predominant sequential quantization framework treats different layers equally, but this may be not optimal in challenging bit-width settings. We empirically study the quantization impact of different layers on model accuracy, and observe that: (1) shallow/deep layers are usually more sensitive to quantization than intermediate layers; (2) among shallow/deep layers, the most sensitive one is the first/last layer, which exhibits significantly larger quantization error than others. These empirical observations imply that the quantization design for different layers of LLMs is required on multiple levels instead of a single level shared to all layers. Motivated by this, we propose a new PTQ framework termed Sliding-layer Quantization (SliderQuant) that relies on a simple adaptive sliding quantization concept facilitated by few learnable parameters. The base component of SliderQuant is called inter-layer sliding quantization, which incorporates three types of novel sliding window designs tailored for addressing the varying quantization sensitivity of shallow, intermediate and deep layers. The other component is called intra-layer sliding quantization that leverages an incremental strategy to quantize each window. As a result, SliderQuant has a strong ability to reduce quantization errors across layers. Extensive experiments on basic language generation, zero-shot commonsense reasoning and challenging math and code tasks with various LLMs, including Llama/Llama2/Llama3/Qwen2.5 model families, DeepSeek-R1 distilled models and large MoE models, show that our method outperforms existing PTQ methods (including the latest PTQ methods using rotation transformations) for both weight-only quantization and weight-activation quantization.
ReQuant: Fixed-Grid Discrete Refinement for Post-Training Quantization
School of Computer Science, Peking University(北京大学计算机学院);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);School of Physics, Peking University(北京大学物理学院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Central Research Institute, ZTE Corporation(中兴公司中央研究院)
AWEQ: Post-Training Quantization with Activation-Weight Equalization for Large Language Models
School of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University(吉林大学符号计算与知识工程教育部重点实验室)
Achieving binary weight and activation for LLMs using Post-Training Quantization
MAIS, Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Central Media Technology Institute, Huawei Technologies Ltd.(华为技术有限公司媒体技术研究所)
RaanA: A Fast, Flexible, and Data-Efficient Post-Training Quantization Algorithm
Computer Science and Engineering, University of Michigan(密歇根大学计算机科学与工程系);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算机科学与数据科学学院);Physics of Artificial Intelligence Group, NTT Research, Inc.(NTT研究所人工智能物理组)
Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(多模态人工智能系统国家重点实验室,中国科学院自动化所);School of Artificial Intelligence, UCAS(人工智能学院,中国科学院大学);Beijing National Research Center for Information Science and Technology(北京信息科学研究中心);Institute of Artificial Intelligence, USTB(信息科学技术大学人工智能学院);School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);Zhongguancun Academy(中关村学院)
PTQ4ARVG: Post-Training Quantization for AutoRegressive Visual Generation Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)