MatGPTQ：准确且高效的后训练Matryoshka量化
MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models
AI总结：
MatGPTQ通过单次训练生成多精度模型，提升低比特精度下的性能，实现高效且准确的后训练量化。
摘要：
Matryoshka量化（MatQuant）是一种近期的量化方法，表明通过在推理时切分最显著的二进制位（MSB），一个整数量化模型可以跨多种精度服务。这使单个检查点能够覆盖广泛的内存和延迟预算，但使量化变得更加具有挑战性。特别是，初始的MatQuant依赖于昂贵的量化感知训练（QAT）变体，而不是快速的一次性后训练量化（PTQ），并且缺乏开源和内核支持。我们通过引入后训练Matryoshka量化（MatGPTQ）来解决所有这些限制，这是一种新的PTQ流水线，能够通过一个小的校准集，在一次训练中生成一个联合优化多种目标精度的父模型。MatGPTQ将Matryoshka量化视为具有位切分和跨位误差补偿的多精度目标，从而产生一个单次通过的多比特宽度、'可切分'模型。我们还结合了一种新的预算感知搜索，用于异构层的比特宽度，并提供了高效的内核来实现切分和混合精度执行。在标准LLM和基准测试中，MatGPTQ在低比特宽度设置下显著提高了性能，同时保持高比特精度。总体而言，我们为Matryoshka风格的后训练量化建立了新的状态，使单检查点、多精度部署变得开放和实用。代码可在https://github.com/IST-DASLab/MatGPTQ上获得。
英文摘要：
Matryoshka Quantization (MatQuant), Any-Precision-LLM (AP) and AnyBCQ (AB) are recent quantization approaches showing that a single integer-quantized model can be served across multiple precisions. In this paradigm, lower-precision models are extracted from a higher-precision model by simply reading fewer bits of the weights. This enables a single checkpoint to cover a wide range of memory and latency budgets, but makes both quantization and efficient execution substantially harder. Existing methods rely on expensive quantization-aware training (QAT) or gradient-based post-training quantization (PTQ) rather than fast one-shot PTQ, and offer limited system support: dedicated kernels are either missing or restricted to single- or small-batch decoding. We address these limitations with Post-Training Matryoshka Quantization (MatGPTQ), an end-to-end pipeline for nested-model quantization and inference. MatGPTQ casts Matryoshka quantization as multi-precision error compensation, producing a single "sliceable" parent model jointly optimized for multiple target precisions in one pass over a small calibration set. We further refine the MatQuant representation so that an $r$-bit model reads exactly $r$ bits, and introduce the first dedicated inference kernels for this format, supporting batch sizes beyond one and integrated into vLLM. Across standard LLMs and benchmarks, MatGPTQ outperforms MatQuant while remaining competitive with AP and AB at the smallest checkpoint size, and our kernels achieve end-to-end speedups of up to 3.5$\times$ over BF16 at the low-bit regime. Overall, MatGPTQ makes nested quantized models practical to serve from a single, compact checkpoint. Code is available at https://github.com/IST-DASLab/MatGPTQ.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
