arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.18154cs.CL
收藏

面向低比特大语言模型的混合精度图神经网络量化

Mixed-Precision Graph Neural Quantization for Low Bit Large Language Models

School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院)

Wanlong Liu, Yichen Xiao, Dingyi Zeng, Hongyang Zhao, Wenyu Chen, Malu Zhang

AI总结:

针对大语言模型在<3比特低比特量化下性能不佳的问题,提出混合精度图神经PTQ方法(MG-PTQ),利用GNN模块捕获权重依赖并自适应分配比特宽度,显著超越GPTQ并创下低比特量化新基准。

摘要:

训练后量化(PTQ)对于在资源受限环境中部署大语言模型(LLMs)至关重要,它能显著降低资源需求。然而,由于量化权重与原始权重之间存在显著差异,现有PTQ策略在<3比特的低比特级别下表现不佳。为提升低比特宽度下的量化性能,我们提出了一种混合精度图神经PTQ(MG-PTQ)方法,采用图神经网络(GNN)模块来捕获权重间的依赖关系,并自适应地分配量化比特宽度。通过GNN模块的信息传播,我们的方法能更有效地捕获目标权重间的依赖关系,从而更准确地评估权重重要性并优化量化策略的分配。在WikiText2和C4数据集上的大量实验表明,我们的MG-PTQ方法优于先前最先进的PTQ方法GPTQ,在低比特条件下创下了量化性能的新基准。

英文摘要:

Post-Training Quantization (PTQ) is pivotal for deploying large language models (LLMs) within resource-limited settings by significantly reducing resource demands. However, existing PTQ strategies underperform at low bit levels < 3 bits due to the significant difference between the quantized and original weights. To enhance the quantization performance at low bit widths, we introduce a Mixed-precision Graph Neural PTQ (MG-PTQ) approach, employing a graph neural network (GNN) module to capture dependencies among weights and adaptively assign quantization bit-widths. Through the information propagation of the GNN module, our method more effectively captures dependencies among target weights, leading to a more accurate assessment of weight importance and optimized allocation of quantization strategies. Extensive experiments on the WikiText2 and C4 datasets demonstrate that our MG-PTQ method outperforms previous state-of-the-art PTQ method GPTQ, setting new benchmarks for quantization performance under low-bit conditions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑