QUASAR：通过损失感知重构降低量化感知训练的损失下限
QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction
浏览论文内容
中文总结 AI 辅助
QUASAR是一种QAT方法，通过训练循环中轻量级损失感知重构降低大语言模型量化损失下限，在2-4比特下优于现有方法，提升了低比特模型准确率。
中文摘要 AI 辅助
随着大语言模型推理转向更低精度，后训练量化（PTQ）变得越来越脆弱，这使得量化感知训练（QAT）对于保持模型质量至关重要。然而，QAT使用潜在全精度权重的有损重构来计算损失和代理梯度，同时对潜在权重本身进行更新，这种不匹配会导致次优的训练轨迹和更高的损失下限。二阶PTQ方法通过最小化损失感知重构误差来缓解类似的差距，但对于冻结模型仅执行一次该过程就需要数小时；随着权重演变在整个QAT中重复此过程是不切实际的。我们提出了QUASAR，这是一种QAT方法，它在训练循环中持续执行轻量级的损失感知重构，以降低损失下限并改进生成的低比特模型。在每个训练步骤中，QUASAR使用梯度平方的指数移动平均作为在线显著性估计，在一小部分裁剪范围内搜索，并通过显著性加权最小二乘拟合仿射反量化器。我们的分析表明，损失感知重构误差是QAT收敛边界中唯一依赖于重构的项，并且控制最终量化模型的损失，从而确立QUASAR的目标是一个有原则的优化目标。QUASAR仅修改训练过程，支持标准部署格式，包括整数量化和NVFP4，无推理时更改或开销。在Qwen3和Llama-3.1上，QUASAR在2、3和4比特的竞争性QAT方法中实现了最低的保留KL散度，在3和4比特时将KL散度至少降低10%，在2比特时降低29%；在2比特时，它在八个任务上的平均准确率比强大的QAT和PTQ基线提高了3.5至4.3个百分点。
英文摘要
As large language model inference shifts toward lower precision, post-training quantization (PTQ) becomes increasingly brittle, making quantization-aware training (QAT) essential for preserving model quality. However, QAT has a structural mismatch: gradient updates are applied to latent full-precision weights, while the loss and gradients are computed on lossy reconstructions of those weights. This mismatch can lead to suboptimal training trajectories and a higher loss floor. Second-order PTQ methods address a similar problem by minimizing loss-aware reconstruction error, but applying such expensive reconstruction repeatedly during QAT as the weights evolve is impractical. We introduce QUASAR, a QAT method that brings lightweight, loss-aware reconstruction into the training loop. At each training step, QUASAR reconstructs the latent weights by searching over a small set of clipping ranges and fitting dequantization parameters through saliency-weighted least squares. We use an exponential moving average of squared gradients as the per-parameter saliency signal. Our theoretical analysis shows that optimizing QUASAR's reconstruction objective tightens both the convergence and final-loss bounds of QAT. We evaluate QUASAR across four model families and across INT4, INT3, INT2, and NVFP4 quantization formats. QUASAR consistently achieves lower training and evaluation loss than competitive QAT methods and outperforms QAT and PTQ baselines on downstream benchmarks. At INT2, QUASAR improves average accuracy over the best QAT baseline by 13.3 points with quantization-aware distillation and by 10.9 points with QAT on mathematical reasoning data. Notably, after distillation on only about 600M tokens, QUASAR's INT4 Gemma-4 E4B checkpoint outperforms the corresponding QAT checkpoint released by Google, with 66% lower KL divergence and 1.8 points higher average accuracy.
发表机构
- Together AI
- Cornell University（康奈尔大学）
机构由 AI 辅助整理，请以论文原文为准。
