arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.25284cs.AI
收藏

SliderQuant: 针对LLMs的准确后训练量化

SliderQuant: Accurate Post-Training Quantization for LLMs

Intel Labs China(英特尔中国研究院);BUPT(北京邮电大学)

Shigeng Wang, Chao Li, Yangyuxuan Kang, Jiawei Fan, Zhonghong Ou, Anbang Yao

AI总结:

本文提出SliderQuant框架,通过分层滑动量化方法优化不同层的量化敏感性,提升LLM在不同位宽下的精度。

摘要:

本文针对大语言模型(LLMs)的后训练量化(PTQ)问题,从被忽视的角度出发,发现传统逐层量化方法在挑战性位宽设置下可能不优。通过实证研究不同层的量化影响,发现浅层和深层通常比中间层更敏感,且最敏感层为第一或最后一层。基于此,提出SliderQuant框架,采用适应性滑动量化概念,通过三种新型滑动窗口设计应对不同层的量化敏感性差异,有效降低量化误差,实验表明其在多种LLM任务中优于现有PTQ方法。

英文摘要:

In this paper, we address post-training quantization (PTQ) for large language models (LLMs) from an overlooked perspective: given a pre-trained high-precision LLM, the predominant sequential quantization framework treats different layers equally, but this may be not optimal in challenging bit-width settings. We empirically study the quantization impact of different layers on model accuracy, and observe that: (1) shallow/deep layers are usually more sensitive to quantization than intermediate layers; (2) among shallow/deep layers, the most sensitive one is the first/last layer, which exhibits significantly larger quantization error than others. These empirical observations imply that the quantization design for different layers of LLMs is required on multiple levels instead of a single level shared to all layers. Motivated by this, we propose a new PTQ framework termed Sliding-layer Quantization (SliderQuant) that relies on a simple adaptive sliding quantization concept facilitated by few learnable parameters. The base component of SliderQuant is called inter-layer sliding quantization, which incorporates three types of novel sliding window designs tailored for addressing the varying quantization sensitivity of shallow, intermediate and deep layers. The other component is called intra-layer sliding quantization that leverages an incremental strategy to quantize each window. As a result, SliderQuant has a strong ability to reduce quantization errors across layers. Extensive experiments on basic language generation, zero-shot commonsense reasoning and challenging math and code tasks with various LLMs, including Llama/Llama2/Llama3/Qwen2.5 model families, DeepSeek-R1 distilled models and large MoE models, show that our method outperforms existing PTQ methods (including the latest PTQ methods using rotation transformations) for both weight-only quantization and weight-activation quantization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑