使用加性码本的大语言模型多比特宽度量化
Multi-Bitwidth Quantization for LLMs Using Additive Codebooks
AI总结：
提出Drop-by-Drop框架，基于信息论和逐次细化理论，利用加性码本和Matryoshka监督实现单个模型在推理时支持多精度权重控制，降低存储开销并保持性能。
摘要：
随着大语言模型（LLM）在具有不同资源约束的异构硬件上部署越来越广泛，无需重新训练即可自适应管理性能与效率之间权衡的能力变得至关重要。我们提出Drop-by-Drop，一种新颖的多比特宽度训练后量化框架，能够从单个训练模型实现对LLM权重的推理时精度控制。我们的方法在理论上基于信息论和逐次细化。我们证明，通常服从高斯分布的LLM权重，在由LLM损失函数驱动的加权均方误差失真下，随着额外比特的加入可以以递增的保真度最优重建。为了在实践中实现这一点，Drop-by-Drop将Matryoshka风格的监督纳入损失函数，利用了加性码本的结构。Drop-by-Drop生成单个模型，其中有序的码本子集在每个精度级别产生精确的部分重建。这种方法通过允许单个检查点服务于多个比特宽度，显著减少了存储和内存开销，同时在主要架构（如Qwen、LLaMA、Gemma和Mistral）上保持了有竞争力的困惑度和准确度。
英文摘要：
As large language models (LLMs) are increasingly deployed across heterogeneous hardware with varying resource constraints, the ability to adaptively manage the performance-efficiency trade-off without retraining is critical. We propose Drop-by-Drop, a novel multi-bitwidth post-training quantization framework enabling inference-time precision control over LLM weights from a single quantized model. As theoretical motivation, we prove that Gaussian sources are successively refinable under a weighted mean squared error distortion motivated by LLM loss functions: successive refinement across distortion levels incurs no rate penalty relative to encoding at each level separately. Drop-by-Drop approximates this hierarchical structure in practice by incorporating Matryoshka-style supervision into additive codebook training, inducing an ordering in which codebook prefixes yield accurate partial reconstructions at each precision level. Furthermore, a block-Hadamard rotation brings the weight distribution closer to our Gaussian source assumption. The result is a single model that serves multiple bitwidths by dropping codebooks, reducing storage and quantization cost relative to the static per-bitwidth models. Across Qwen, LLaMA, Gemma, and Mistral, Drop-by-Drop achieves lower perplexity than state-of-the-art multi-bitwidth methods with competitive zero-shot accuracy, while our specialized kernels further reduce decoding latency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
