Qrazor：通过显著数据剃削实现可靠且轻松的4位LLM量化
Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring
AI总结：
QRazor提出一种两阶段4位量化方案，先以8/16位整数加绝对最大值缩放保精度，再用显著数据剃削技术压缩至4位，无需微调即在LLaMA2-7B零样本推理上超越现有方法，并配套免解压整数算术单元。
摘要：
大规模语言模型（LLMs）在语言处理任务中表现出色，但由于高内存和计算需求而面临部署挑战。虽然低位量化（如4位技术）提供了一种潜在的解决方案，但这些方法通常会导致显著的精度损失，或需要大量的实现工作，例如重排序、旋转等。为了应对这些挑战，我们提出了QRazor，这是一种简单而有效的量化方案，可在基于Transformer的LLMs中实现权重、激活和KV缓存的4位量化。QRazor分两个阶段运行：首先，使用8位或16位整数作为基础，采用绝对最大值缩放对数据进行量化，以保持接近全精度模型的精度；其次，使用我们的显著数据剃削（SDR）技术将量化数据压缩到4位，该技术仅保留四个最显著的比特位。无需任何微调或额外训练，QRazor在4位量化方法中实现了与最先进方法相似或更好的性能，在LLaMA2-7B模型的零样本推理任务准确率上，比Smoothquant和QLLM高出超过12个百分点，比Quarot(RTN)高出超过2.9个百分点。此外，我们引入了一个针对QRazor优化的基于整数的算术单元，允许直接对SDR数据进行低精度运算而无需解压缩。
英文摘要：
Large-scale language models (LLMs) excel in language processing tasks but face deployment challenges due to high memory and computational demands. While low-bit quantization, such as 4-bit techniques, offers a potential solution, these methods often suffer from significant accuracy loss or require considerable effort for implementation such as reordering, rotation, etc. To address these challenges, we propose QRazor, a simple yet effective quantization scheme that enables 4-bit quantization of weights, activations, and KV cache in transformer-based LLMs. QRazor operates in two stages: first, quantizing data using 8 or 16-bit integers as a basis with absolute max scaling to preserve accuracy close to full-precision models, and second, compressing the quantized data to 4-bit using our significant data razoring (SDR) technique, which retains only the four most salient bits. Without any additional requirment of fine-tuning or additional training, QRazor achieves performance similar or better compared to state-of-the-art in 4-bit quantization method, surpassing Smoothquant and QLLM by over 12 points and Quarot(RTN) by more than 2.9 points in zero-shot reasoning task accuracy on the LLaMA2-7B model. Additionally, we introduce an integer-based arithmetic unit optimized for QRazor, allowing direct low-precision operations on SDR data without decompression.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
