arXiv 2405.184152024-11-03cs.CVcs.AIcs.CLcs.LG

为什么视觉接地语言模型在图像分类任务中表现不佳？

Why are Visually-Grounded Language Models Bad at Image Classification?

Stanford University（斯坦福大学）；University of Washington（华盛顿大学）；Tsinghua University（清华大学）

Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, Serena Yeung-Levy

备注：: Published at NeurIPS 2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究探究视觉接地语言模型（VLM）图像分类表现逊于CLIP的原因，发现核心是训练数据不足，类别曝光频率与性能强相关，补充分类数据训练可使其媲美最优分类模型，且性能可迁移至通用能力，在ImageWikiQA上提升11.8%。

摘要：

图像分类是机器视觉智能最基础的能力之一。在本研究中，我们使用视觉接地语言模型（visually-grounded language models, VLMs），例如GPT-4V和LLaVA，重新审视图像分类任务。我们发现，现有的专有和公开VLM尽管通常采用CLIP作为视觉编码器且参数量远多于CLIP，但在ImageNet等标准图像分类基准上的表现显著逊于CLIP。为探究原因，我们围绕VLM的推理算法、训练目标和数据处理提出了若干假设并展开研究。分析表明，核心原因与数据相关：图像分类所需的关键信息已编码在VLM的潜在空间中，但只有具备足够训练数据才能被有效解码。具体而言，VLM训练及指令微调过程中的类别出现频率，与VLM在这些类别上的表现存在强相关性；当使用充足数据训练时，VLM的准确率可媲美当前最优的分类模型。基于这些发现，我们通过将聚焦分类的数据集融入训练来增强一款VLM，并证实该VLM提升后的分类性能可迁移至其通用能力中，在新收集的ImageWikiQA数据集上实现了11.8%的性能提升。

英文摘要：

Image classification is one of the most fundamental capabilities of machine vision intelligence. In this work, we revisit the image classification task using visually-grounded language models (VLMs) such as GPT-4V and LLaVA. We find that existing proprietary and public VLMs, despite often using CLIP as a vision encoder and having many more parameters, significantly underperform CLIP on standard image classification benchmarks like ImageNet. To understand the reason, we explore several hypotheses concerning the inference algorithms, training objectives, and data processing in VLMs. Our analysis reveals that the primary cause is data-related: critical information for image classification is encoded in the VLM's latent space but can only be effectively decoded with enough training data. Specifically, there is a strong correlation between the frequency of class exposure during VLM training and instruction-tuning and the VLM's performance in those classes; when trained with sufficient data, VLMs can match the accuracy of state-of-the-art classification models. Based on these findings, we enhance a VLM by integrating classification-focused datasets into its training, and demonstrate that the enhanced classification performance of the VLM transfers to its general capabilities, resulting in an improvement of 11.8% on the newly collected ImageWikiQA dataset.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。