大型语言模型促进图像分类中的视觉反思
Large Language Models Facilitate Vision Reflection in Image Classification
AI总结：
本文发现提示LMM验证专用视觉模型预测可提升ImageNet等基准的识别准确率，揭示视觉反思依赖提炼的文本表示，并提出无需训练的连接器增强细粒度识别性能。
摘要：
本文提出了关于大型多模态模型（LMMs）中视觉反思可解释性的几项新颖发现。首先，我们表明，提示LMM验证专用视觉模型的预测可以提高识别准确率，即使在ImageNet等基准测试上也是如此，尽管先前的证据表明LMM通常表现不如专用视觉编码器。其次，我们分析了视觉反思的内部行为，发现视觉-语言连接器将视觉特征映射为显式的文本概念，使语言模型能够利用常识知识推理预测的合理性。我们进一步观察到，用少量文本标记替换大量视觉标记仍然使LLaVA能够生成相似的答案，这表明LMM可能主要依赖于一组紧凑的提炼文本表示，而不是原始视觉特征。第三，我们表明，无需训练的连接器可以增强LMM在细粒度识别任务中的性能，而无需进行大量的特征对齐训练。总之，这些发现为视觉-语言模型的可解释性提供了新见解，并表明视觉反思是实现鲁棒且可解释的视觉识别的有前景的策略。
英文摘要：
This paper presents several novel findings on the explainability of vision reflection in large multimodal models (LMMs). First, we show that prompting an LMM to verify the prediction of a specialized vision model can improve recognition accuracy, even on benchmarks like ImageNet, despite prior evidence that LMMs typically underperform dedicated vision encoders. Second, we analyze the internal behavior of vision reflection and find that the vision-language connector maps visual features into explicit textual concepts, allowing the language model to reason about prediction plausibility using commonsense knowledge. We further observe that replacing a large number of vision tokens with only a few text tokens still enables LLaVA to generate similar answers, suggesting that LMMs may rely primarily on a compact set of distilled textual representations rather than raw vision features. Third, we show that a training-free connector can enhance LMM performance in fine-grained recognition tasks, without extensive feature-alignment training. Together, these findings offer new insights into the explainability of vision-language models and suggest that vision reflection is a promising strategy for achieving robust and interpretable visual recognition.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
