超越后训练量化中的重建损失：大型视觉-语言模型的平衡拟合

中文摘要 AI 辅助

后训练量化（PTQ）使得大型视觉-语言模型（LVLMs）的高效部署成为可能，但通常在小型数据集上进行校准，却期望在多样化的下游任务中泛化。尽管近期针对LVLMs的PTQ方法融入了敏感性信号，它们仍然相对于全精度模型最小化重建损失，可能过度保留FP行为和校准特定偏差。我们不将量化仅仅视为需要最小化的误差，而是观察到量化也可以为某些层和模态提供有益的正则化。受此观察启发，我们提出平衡拟合（Balanced Fitting），一种基于量化效果的框架，在基于重建的优化之外平衡精度与正则化。通过分别衡量权重、视觉激活和文本激活的逐层和逐组件量化效果，平衡拟合对敏感组件采用细粒度拟合，而对其他组件采用较粗拟合以利用潜在的正则化收益。在多个LVLMs上的实验表明，我们的方法在仅权重量化和权重-激活量化下均持续优于先前的PTQ方法，而较低的重建损失并不能可靠地转化为更好的下游性能。源代码可从此https URL公开获取。

英文摘要

Post-training quantization (PTQ) enables efficient deployment of large vision-language models (LVLMs), but is typically calibrated on a small set while expected to generalize across diverse downstream tasks. Although recent PTQ methods for LVLMs incorporate sensitivity signals, they still minimize reconstruction loss with respect to the full-precision model, potentially over-preserving FP behavior and calibration-specific bias. Rather than treating quantization solely as an error to be minimized, we observe that it can also provide beneficial regularization for certain layers and modalities. Motivated by this observation, we propose Balanced Fitting, a quantization effect-based framework that balances precision and regularization beyond reconstruction-based optimization. By measuring layer- and component-wise quantization effects for weights, vision activations, and text activations, Balanced Fitting combines fine-grained fitting for sensitive components with coarser fitting to exploit potential regularization benefits. Experiments on multiple LVLMs show that our method consistently outperforms prior PTQ approaches under both weight-only and weight-activation quantization, while lower reconstruction loss does not reliably translate into better downstream performance. The source code is publicly available at https://github.com/kmc3661/BFQ

发表机构

• Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
• Hanbat National University（韩巴国立大学）

机构由 AI 辅助整理，请以论文原文为准。