重新审视MLLMs：图像分类能力的深入分析
Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities
AI总结：
本文深入分析了多模态大语言模型的图像分类能力，发现最新模型在多个数据集上匹敌或超越CLIP风格模型，并指出语言模型进步与训练数据多样性是性能提升的关键原因。
摘要：
随着多模态大语言模型的快速发展，各种基准测试被引入以评估其能力。尽管大多数评估集中在复杂任务上，如科学理解和视觉推理，但很少有人关注评估它们的基本图像分类能力。在本文中，我们通过深入分析图像分类来弥补这一空白。具体而言，基于现有数据集，我们考察了从一般分类任务（如ImageNet、ObjectNet）到更细粒度类别（如鸟类和食物分类）的广泛场景。我们的发现表明，最新的MLLMs在多个数据集上可以匹敌甚至超越CLIP风格的视觉语言模型，这挑战了先前关于MLLMs在图像分类方面表现不佳的假设。为了理解推动这一改进的因素，我们对公开MLLMs的网络架构、数据选择和训练配方进行了深入分析。我们的结果将这一成功归因于语言模型的进步和训练数据源的多样性。基于这些观察，我们进一步分析并将潜在原因分别归因于概念知识迁移和目标概念暴露的增强。我们希望我们的发现能为未来关于MLLMs及其在图像分类任务中的评估研究提供有价值的见解。
英文摘要：
With the rapid advancement of Multimodal Large Language Models (MLLMs), a variety of benchmarks have been introduced to evaluate their capabilities. While most evaluations have focused on complex tasks such as scientific comprehension and visual reasoning, little attention has been given to assessing their fundamental image classification abilities. In this paper, we address this gap by thoroughly revisiting the MLLMs with an in-depth analysis of image classification. Specifically, building on established datasets, we examine a broad spectrum of scenarios, from general classification tasks (e.g., ImageNet, ObjectNet) to more fine-grained categories such as bird and food classification. Our findings reveal that the most recent MLLMs can match or even outperform CLIP-style vision-language models on several datasets, challenging the previous assumption that MLLMs are bad at image classification \cite{VLMClassifier}. To understand the factors driving this improvement, we conduct an in-depth analysis of the network architecture, data selection, and training recipe used in public MLLMs. Our results attribute this success to advancements in language models and the diversity of training data sources. Based on these observations, we further analyze and attribute the potential reasons to conceptual knowledge transfer and enhanced exposure of target concepts, respectively. We hope our findings will offer valuable insights for future research on MLLMs and their evaluation in image classification tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
