评估后训练量化对多模态大语言模型可靠视觉问答的影响
Evaluating the Impact of Post-Training Quantization on Reliable VQA with Multimodal LLMs
AI总结：
本文研究了后训练量化对多模态大语言模型可靠视觉问答的影响，提出通过选择器置信度估计器提升可靠性，并在不同量化方法中实现了效率与可靠性的最佳平衡。
摘要：
多模态大语言模型（MLLM）正越来越多地应用于对可靠性和效率都至关重要的领域。然而，当前模型仍然过于自信，产生高度确定但错误的答案。同时，它们的大型规模限制了在边缘设备上的部署，需要压缩。我们通过分析后训练量化（PTQ）压缩如何影响视觉问答（VQA）中的准确性和可靠性来研究这两个挑战的交集。我们评估了两个MLLM，Qwen2-VL-7B和Idefics3-8B，在多种位宽下使用无数据（HQQ）和数据感知（MBQ）方法进行量化。为了对抗量化导致的可靠性降低，我们适应了选择器置信度估计器用于量化多模态设置，并测试其在各种量化级别和异常分布（OOD）场景中的鲁棒性。我们发现PTQ会降低准确性和可靠性。数据感知方法减轻了这种影响。选择器显著缓解了可靠性的影响。int4 MBQ与选择器的结合实现了最佳的效率-可靠性权衡，接近未压缩性能，内存需求减少约75%。总体而言，我们首次系统研究了多模态设置中量化与可靠性的关联。
英文摘要：
Multimodal Large Language Models (MLLM) are increasingly deployed in domains where both reliability and efficiency are critical. However, current models remain overconfident, producing highly certain but incorrect answers. At the same time, their large size limits deployment on edge devices, necessitating compression. We study the intersection of these two challenges by analyzing how Post-Training Quantization (PTQ) compression affects both accuracy and reliability in Visual Question Answering (VQA). We evaluate two MLLMs, Qwen2-VL-7B and Idefics3-8B, quantized with data-free (HQQ) and data-aware (MBQ) methods across multiple bit widths. To counteract the reduction in reliability caused by quantization, we adapt the Selector confidence estimator for quantized multimodal settings and test its robustness across various quantization levels and out-of-distribution (OOD) scenarios. We find that PTQ degrades both accuracy and reliability. Data-aware methods soften the effect thereof. The Selector substantially mitigates the reliability impact. The combination of int4 MBQ and the Selector achieves the best efficiency-reliability trade-off, closing in on uncompressed performance at approx. 75% less memory demand. Overall, we present the first systematic study linking quantization and reliability in multimodal settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
