量化大语言模型的低秩校正
Low-Rank Correction for Quantized LLMs
AI总结：
针对大语言模型训练后量化压缩的激活误差问题，提出添加全精度低秩权重矩阵的联合优化校正方法，在W4A4量化下可缩小甚至消除与原模型的准确率差距，已在多款主流LLM上验证。
摘要：
我们研究大语言模型（LLM）训练后阶段的模型压缩问题，该任务的目标是仅使用少量校准输入数据压缩已训练好的模型。本研究提出一种新的低秩方法，用于校正大语言模型中激活值的量化误差：我们建议添加作用于未量化激活值的全精度低秩权重矩阵。随后，我们针对权重的量化表示与额外的低秩权重矩阵求解联合优化问题，以同时量化权重和激活值。我们重点研究4位权重-激活量化（W4A4）的场景。当使用的秩相当于原始权重矩阵规模的10%时，我们的方法将与原始模型的准确率差距缩小了50%以上；当秩相当于原始权重矩阵的30%时，准确率差距被完全消除。我们在四款最新大语言模型上验证了结果，分别是Llama-2、Llama-3、Phi-3和Mixtral模型。
英文摘要：
We consider the problem of model compression for Large Language Models (LLMs) at post-training time, where the task is to compress a well-trained model using only a small set of calibration input data. In this work, we introduce a new low-rank approach to correct for quantization errors of \emph{activations} in LLMs: we propose to add low-rank weight matrices in full precision that act on the \emph{unquantized} activations. We then solve a joint optimization problem over the quantized representation of the weights and additional low-rank weight matrices to quantize both weights and activations. We focus on the case of 4-bit weight-and-activation quantization (W4A4). Using ranks equivalent to 10\% of the original weight matrix size, our approach reduces the accuracy gap with the original model by more than 50\%. Using ranks equivalent to 30\% of the original weight matrix, the accuracy gap is closed completely. We demonstrate our results on four recent LLMs, namely Llama-2, Llama-3, Phi-3 and Mixtral models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
