自适应块缩放数据类型
Adaptive Block-Scaled Data Types
AI总结:
本文提出自适应块缩放数据类型,通过动态选择整数和浮点表示以减少量化误差,在4位量化中实现更低的训练损失和更高的任务准确性。
摘要:
NVFP4作为一种4位量化格式因其硬件支持和保留有用信息的能力而日益流行。然而,其误差分布导致每组16个值中近最大值的量化误差较大。本文基于此见解设计了新的自适应块缩放数据类型,通过动态选择FP4或INT4表示并使用E4M3缩放因子进行缩放。所选数据类型通过缩放因子的符号位表示,该位在NVFP4中未被使用。我们还为其他位宽设计了IF3和IF6格式。在语言模型量化中,IF4在量化训练中损失更低,在后训练量化中准确性更高。此外,我们设计并评估了IF4乘加单元以证明其在下一代硬件加速器中的高效实现。代码可在https://github.com/mit-han-lab/fouroversix获取。
英文摘要:
NVFP4 has grown increasingly popular as a 4-bit format for quantizing large language models due to its hardware support and its ability to retain useful information with relatively few bits per parameter. However, the format is not without limitations: recent work has shown that NVFP4 suffers from its error distribution, resulting in large amounts of quantization error on near-maximal values in each group of 16 values. In this work, we leverage this insight to design new Adaptive Block-Scaled Data Types that can adapt to the distribution of their input values. For four-bit quantization, our proposed IF4 (Int/Float 4) data type selects between FP4 and INT4 representations for each group of 16 values, which are then scaled by an E4M3 scale factor as is done with NVFP4. The selected data type is denoted using the scale factor's sign bit, which is currently unused in NVFP4, and we apply the same insight to design formats for other bit-widths, including IF3 and IF6. When used to quantize language models, we find that IF4 outperforms existing 4-bit block-scaled formats, achieving lower loss during quantized training and achieving higher accuracy on many tasks in post-training quantization. We additionally design and evaluate an IF4 Multiply-Accumulate (MAC) unit to demonstrate that IF4 can be implemented efficiently in next-generation hardware accelerators. Our code is available at https://github.com/mit-han-lab/fouroversix.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。