AFPQ：面向大语言模型的非对称浮点量化
AFPQ: Asymmetric Floating Point Quantization for LLMs
AI总结：
针对现有浮点量化在小组尺寸或4位以下场景中性能不佳的问题，提出为正负值分别设置缩放因子的AFPQ方法，可提升精度、兼容现有量化方案且无额外存储开销。
摘要：
大语言模型（LLM）在各类任务中表现优异，但受限于内存容量与带宽，部署面临挑战。低位权重量化可节省内存并加速推理。尽管浮点（FP）格式在LLM量化中表现良好，但在小组尺寸或4位以下的场景下性能往往不佳。我们发现，原因在于以往浮点量化缺乏非对称性，无法适配LLM权重张量的非对称数值分布。本文提出非对称浮点量化（AFPQ），为正值和负值分别设置缩放因子。该方法可大幅提升精度，且能轻松集成到GPTQ、AWQ等其他量化方法中以获得更优性能。此外，与非对称整数（INT）量化相比，该方法无需额外存储空间。代码已发布于https://github.com/zhangsichengsjtu/AFPQ。
英文摘要：
Large language models (LLMs) show great performance in various tasks, but face deployment challenges from limited memory capacity and bandwidth. Low-bit weight quantization can save memory and accelerate inference. Although floating-point (FP) formats show good performance in LLM quantization, they tend to perform poorly with small group sizes or sub-4 bits. We find the reason is that the absence of asymmetry in previous FP quantization makes it unsuitable for handling asymmetric value distribution of LLM weight tensors. In this work, we propose asymmetric FP quantization (AFPQ), which sets separate scales for positive and negative values. Our method leads to large accuracy improvements and can be easily plugged into other quantization methods, including GPTQ and AWQ, for better performance. Besides, no additional storage is needed compared with asymmetric integer (INT) quantization. The code is available at https://github.com/zhangsichengsjtu/AFPQ.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
