分解量化误差:一种集中对齐视角
Dissecting Quantization Error: A Concentration-Alignment Perspective
AI总结:
本文提出块集中对齐变换(CAT),通过提升权重与激活的对齐度以减少量化误差,验证了在4位精度下优于现有方法。
摘要:
量化可以显著提高大语言和视觉模型的效率,但通常会带来精度下降。最近,保持函数的变换(例如旋转、哈达玛变换、通道级缩放)已被成功应用于减少后训练量化误差,但尚缺乏系统性的解释。我们通过信号与量化噪声比(SQNR)分析线性层量化,显示在固定位宽下均匀整数量化时,SQNR分解为(i)权重和激活的集中度(捕捉分布和异常值),以及(ii)其主导变化方向的对齐程度。这揭示了可操作的见解:在集中度(大多数先前变换如旋转或哈达玛的关注点)之外,改进权重和激活之间的对齐可以进一步减少量化误差。受此启发,我们引入了块集中对齐变换(CAT),一种轻量的线性变换,利用小校准集的协方差估计,共同提高集中度和对齐度,近似最大化SQNR。在多个LLM上的实验表明,CAT在4位精度下始终与先前基于变换的量化方法匹配或表现更好,证实了我们框架中获得的见解。
英文摘要:
Quantization can drastically increase the efficiency of large language and vision models, but typically incurs an accuracy drop. Recently, function-preserving transforms (e.g. rotations, Hadamard transform, channel-wise scaling) have been successfully applied to reduce post-training quantization error, yet a principled explanation remains elusive. We analyze linear-layer quantization via the signal-to-quantization-noise ratio (SQNR), showing that for uniform integer quantization at a fixed bit width, SQNR decomposes into (i) the concentration of weights and activations (capturing spread and outliers), and (ii) the alignment of their dominant variation directions. This reveals an actionable insight: beyond concentration - the focus of most prior transforms (e.g. rotations or Hadamard) - improving alignment between weight and activation can further reduce quantization error. Motivated by this, we introduce block Concentration-Alignment Transforms (CAT), a lightweight linear transformation that uses a covariance estimate from a small calibration set to jointly improve concentration and alignment, approximately maximizing SQNR. Experiments across several LLMs show that CAT consistently matches or outperforms prior transform-based quantization methods at 4-bit precision, confirming the insights gained in our framework.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。