重新思考用于图像分类的视觉语言模型与大语言模型
Rethinking VLMs and LLMs for Image Classification
AI总结：
本文通过实验发现VLMs与LLMs融合对图像分类的贡献因任务而异，并提出一种基于小型LLM路由器的轻量级方案，在超越或持平GPT-4V等先进方法的同时提升成本效益。
摘要：
视觉语言模型（VLMs）如今正越来越多地与大语言模型（LLMs）融合，以实现新的能力，特别是在提升交互性和开放式响应能力方面。尽管这些能力十分显著，但 LLMs 对增强一个长期存在的关键问题——在一组候选项中对图像进行分类——的贡献仍不明确。通过在七个模型、十个视觉理解数据集以及每个数据集多种提示变体上开展的大量实验，我们发现，对于物体和场景识别，不利用 LLMs 的 VLMs 能够取得比利用 LLMs 的 VLMs 更好的性能。然而与此同时，利用 LLMs 可以提升需要推理和外部知识的任务上的表现。针对这些挑战，我们提出一种务实的解决方案：一种轻量级修复方法，涉及一个相对较小的 LLM，该 LLM 能够高效地将视觉任务路由到最适合该任务的模型。该 LLM 路由器使用由超过 250 万个视觉任务与模型准确率配对示例构成的数据集进行训练。我们的结果表明，这种轻量级修复方法在准确率上超越或持平于最先进的替代方案，包括 GPT-4V 和 HuggingGPT，同时提高了成本效益。
英文摘要：
Visual Language Models (VLMs) are now increasingly being merged with Large Language Models (LLMs) to enable new capabilities, particularly in terms of improved interactivity and open-ended responsiveness. While these are remarkable capabilities, the contribution of LLMs to enhancing the longstanding key problem of classifying an image among a set of choices remains unclear. Through extensive experiments involving seven models, ten visual understanding datasets, and multiple prompt variations per dataset, we find that, for object and scene recognition, VLMs that do not leverage LLMs can achieve better performance than VLMs that do. Yet at the same time, leveraging LLMs can improve performance on tasks requiring reasoning and outside knowledge. In response to these challenges, we propose a pragmatic solution: a lightweight fix involving a relatively small LLM that efficiently routes visual tasks to the most suitable model for the task. The LLM router undergoes training using a dataset constructed from more than 2.5 million examples of pairs of visual task and model accuracy. Our results reveal that this lightweight fix surpasses or matches the accuracy of state-of-the-art alternatives, including GPT-4V and HuggingGPT, while improving cost-effectiveness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
