NestQuant：用于矩阵乘积与大语言模型的嵌套格量化
NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
AI总结：
本文提出基于自相似嵌套格与 Gosset 格的 NestQuant 训练后量化方法，可即插即用地量化 LLM 权重、激活和 KV-cache，并在 4 bit 设置及最高 70B 模型上显著降低困惑度损失。
摘要：
训练后量化（PTQ）已成为高效部署大语言模型（LLM）的关键技术。本文提出 NestQuant，这是一种基于自相似嵌套格的新型权重与激活 PTQ 方案。近期工作已从数学上证明，此类量化器在低精度矩阵乘法中具有信息论最优性。我们基于 Gosset 格实现了一个实用、低复杂度的 NestQuant 版本，使其可作为任意矩阵乘法步骤（例如 self-attention、MLP 等）的即插即用量化器。例如，NestQuant 将 Llama-3-8B 的权重、KV-cache 和激活量化至 4 bit，在 wikitext2 上达到 6.6 的困惑度。相对于未量化模型（困惑度为 6.14），与最先进的 Meta SpinQuant（困惑度 7.3）、OstQuant（7.3）和 QuaRot（8.2）相比，这表示困惑度差距缩小超过 55%。在更大模型（最高达 70B）以及多种 LLM 评估基准上的比较，证实了 NestQuant 的一致优越性。
英文摘要：
Post-training quantization (PTQ) has emerged as a critical technique for efficient deployment of large language models (LLMs). This work proposes NestQuant, a novel PTQ scheme for weights and activations that is based on self-similar nested lattices. Recent works have mathematically shown such quantizers to be information-theoretically optimal for low-precision matrix multiplication. We implement a practical low-complexity version of NestQuant based on Gosset lattice, making it a drop-in quantizer for any matrix multiplication step (e.g., in self-attention, MLP etc). For example, NestQuant quantizes weights, KV-cache, and activations of Llama-3-8B to 4 bits, achieving perplexity of 6.6 on wikitext2. This represents more than 55% reduction in perplexity gap with respect to unquantized model (perplexity of 6.14) compared to state-of-the-art Metas SpinQuant (perplexity 7.3), OstQuant (7.3) and QuaRot (8.2). Comparisons on bigger models (up to 70B) and on various LLM evaluation benchmarks confirm uniform superiority of NestQuant.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
