GPTVQ：维度对LLM量化的祝福
GPTVQ: The Blessing of Dimensionality for LLM Quantization
AI总结：
本文提出GPTVQ方法，通过提高量化维度并利用Hessian信息交错更新权重与码本，在LLM上实现了更优的规模-精度权衡，且解压延迟优于4位整数格式。
摘要：
在这项工作中，我们表明，通过提高量化维度，可以显著改善神经网络量化在规模与精度之间的权衡。我们提出了GPTVQ方法，这是一种新的快速训练后向量量化（VQ）方法，能够很好地扩展到大型语言模型（LLMs）。我们的方法将一列或多列的量化与对剩余未量化权重的更新交错进行，利用每层输出重构均方误差（MSE）的Hessian矩阵信息。量化码本使用一种高效的数据感知版EM算法进行初始化。随后码本被更新，并通过整数量化和基于SVD的压缩进一步压缩。GPTVQ在广泛的LLMs（如Llama-v2和Mistral）上建立了规模与精度权衡的新最优水平。此外，我们的方法效率很高：在单个H100上，处理一个Llamav2-70B模型需要3到11小时，具体取决于量化设置。最后，通过在移动CPU上进行VQ解压缩的设备端计时，我们表明与使用4位整数格式相比，VQ能够带来更低的延迟。
英文摘要：
In this work we show that the size versus accuracy trade-off of neural network quantization can be significantly improved by increasing the quantization dimensionality. We propose the GPTVQ method, a new fast method for post-training vector quantization (VQ) that scales well to Large Language Models (LLMs). Our method interleaves quantization of one or more columns with updates to the remaining unquantized weights, using information from the Hessian of the per-layer output reconstruction MSE. Quantization codebooks are initialized using an efficient data-aware version of the EM algorithm. The codebooks are then updated, and further compressed by using integer quantization and SVD-based compression. GPTVQ establishes a new state-of-the art in the size vs accuracy trade-offs on a wide range of LLMs such as Llama-v2 and Mistral. Furthermore, our method is efficient: on a single H100 it takes between 3 and 11 hours to process a Llamav2-70B model, depending on quantization setting. Lastly, with on-device timings for VQ decompression on a mobile CPU we show that VQ leads to improved latency compared to using a 4-bit integer format.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
