ADMM-Q: 一种改进的基于Hessian的权重量化器用于大语言模型的后训练量化
ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Models
AI总结:
ADMM-Q是一种新的权重量化算法,通过交替方向乘子法的组合变体,优化层间重建误差并逐步施加量化约束,提升大语言模型的压缩性能。
摘要:
量化是减少大语言模型(LLMs)存储和计算足迹的有效策略。后训练量化(PTQ)是压缩LLMs的主流方法。流行的权重量化过程,包括GPTQ和RTN,在模型效用上表现不佳,特别是在极端量化级别(低于4位)时。我们提出了ADMM-Q,一种新的权重量化算法,考虑了层间量化问题。我们的算法基于交替方向乘子法(ADMM)的组合变体。我们的操作分裂过程连续更新权重以最小化层间重建误差,同时逐步施加量化约束并保证收敛性。我们还提出了额外的算法改进(例如,惩罚调度、预条件和局部搜索后处理步骤),使ADMM-Q在LLM规模上更高效。ADMM-Q是模块化的,可以作为现有量化流程中任何权重量化器的直接替换:ADMM-Q完全可组合于现有技术,包括范围截断、学习或随机旋转以及激活缩放。在Qwen3-8B上用ADMM-Q替代GPTQ,我们降低了WikiText-2的困惑度:(i)在W3A16权重-only设置中(12.85→10.06);(ii)在W4A8 SmoothQuant过程中(9.29→8.68);(iii)在W2A4KV4 SpinQuant过程中(66.11→19.42)
英文摘要:
Quantization is an effective strategy to reduce the storage and computation footprint of large language models (LLMs). Post-training quantization (PTQ) is a leading approach for compressing LLMs. Popular weight quantization procedures, including GPTQ and RTN, suffer in model utility, especially at aggressive quantization levels (sub-4-bit). We propose ADMM-Q, a novel weight quantization algorithm that considers the layer-wise quantization problem. Our algorithm is based on a combinatorial variant of the Alternating Direction Method of Multipliers (ADMM). Our operator-splitting procedure updates weights continuously to minimize the layer-wise reconstruction error, while gradually enforcing the quantization constraints with convergence guarantees. We propose additional algorithmic enhancements (e.g., penalty scheduling, preconditioning, and a local search post-processing step) to make ADMM-Q efficient at LLM scale. ADMM-Q is modular and can be used as a drop-in replacement for any weight quantizer within existing quantization pipelines: ADMM-Q is fully composable with existing techniques including range clipping, learned or random rotations, and activation scaling. Using ADMM-Q in place of GPTQ on Qwen3-8B, we decrease WikiText-2 perplexity in: (i) the W3A16 weight-only setting (12.85 $\rightarrow$ 10.06); (ii) the W4A8 SmoothQuant procedure (9.29 $\rightarrow$ 8.68); and (iii) the W2A4KV4 SpinQuant procedure (66.11 $\rightarrow$ 19.42).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。