SAGE：推动2比特向量量化大语言模型实现实际应用
SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability
AI总结：
针对2比特向量量化大语言模型的敏感通道保护与比特分配挑战，提出SAGE方法，在多基准测试中实现最优性能，保留Qwen3-30B-A3B模型95.3%的BF16精度，端到端加速最高达2.55倍。
摘要：
极低比特仅权重量化是降低大语言模型部署成本的有效方法。低于3比特时，标量量化常出现严重精度损失，而向量量化（VQ）通过更具表达力的联合编码能更好地保留精度。不过，现有VQ方法在实际任务上仍存在显著性能下降，且有两个设计挑战未得到充分探索：（1）保护联合编码向量内的敏感权重通道；（2）在表示分量（如方向和半径）间分配固定比特预算，以适配不同权重矩阵。为解决这些挑战，我们提出SAGE：基于能量的拆分适配与分组，用于仅权向量量化。首先，能量分层分组（ESG）将通道分组与激活加权拟合相结合，在每个子向量内对不同能量层级的通道进行分组，以优先保护敏感通道。其次，自适应比特拆分（ABS）在固定预算下，为每个矩阵平衡方向码本容量与半径精度，选择激活加权重构误差最低的拆分方式。在涵盖知识、数学、代码生成、指令跟随和工具使用的基准测试中，SAGE在2比特量化设置下实现了最优性能。在Qwen3-30B-A3B模型上，它保留了BF16模型95.3%的平均基准分数。通过我们定制设计的CUDA内核，SAGE实现了最高达FP16的2.55倍非流式端到端加速。代码可在指定URL获取。
英文摘要：
Extreme low-bit weight-only quantization is an effective approach to reducing the deployment costs of large language models. Below 3 bits, scalar quantization often suffers severe accuracy loss, while vector quantization (VQ) better preserves accuracy through more expressive joint encoding. However, existing VQ methods still exhibit substantial degradation on real-world tasks, and two design challenges remain underexplored: (1) protecting sensitive weight channels within jointly encoded vectors; and (2) allocating a fixed bit budget among representation components, such as direction and radius, to suit different weight matrices. To address these challenges, we propose SAGE: Split Adaptation and Grouping by Energy for weight-only vector quantization. First, Energy-Stratified Grouping (ESG) couples channel grouping with activation-weighted fitting, grouping channels from different energy tiers within each subvector to prioritize sensitive channels. Second, Adaptive Bit Split (ABS) balances direction-codebook capacity and radius precision per matrix, selecting the split with the lowest activation-weighted reconstruction error under a fixed budget. Across benchmarks covering knowledge, mathematics, code generation, instruction following, and tool use, SAGE achieves state-of-the-art performance in the 2-bit quantization setting. On Qwen3-30B-A3B, it retains 95.3% of the BF16 model's average benchmark score. With our custom-designed CUDA kernel, SAGE delivers up to a 2.55-fold non-streaming end-to-end speedup over FP16. The code is available at https://github.com/Zincwolf/sage.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
