重新审视多模态大语言模型评估
Revisiting Multi-Modal LLM Evaluation
AI总结：
本文在针对早期数据集缺陷而设计的数据集上系统评估了六种近期多模态大语言模型，揭示了它们此前未被报道的弱点，并将评估代码集成到LAVIS框架中以支持未来模型的快速评测。
摘要：
随着多模态大语言模型（MLLMs）的出现，用于视觉问答（VQA）和指代表达理解的数据集重新受到关注。然而，用于评估MLLMs的最流行数据集大多是早期创建的数据集，它们存在许多已知问题，包括极端偏差、虚假相关性以及无法支持细粒度分析。在本文中，我们率先在专门为解决早期数据集弱点而设计的数据集上评估近期MLLMs（LLaVA 1.5、LLaVA-NeXT、BLIP2、InstructBLIP、GPT-4V和GPT-4o）。我们评估了三个VQA数据集：1) TDIUC，支持对12种问题类型进行细粒度分析；2) TallyQA，包含简单和复杂的计数问题；3) DVQA，需要光学字符识别来进行图表理解。我们还研究了VQDv1，该数据集要求识别满足给定查询的所有图像区域。我们的实验揭示了许多MLLMs此前未被报道的弱点。我们的代码已集成到广泛使用的LAVIS框架中，用于MLLM评估，从而能够快速评估未来的MLLMs。项目网页：https://kevinlujian.github.io/MLLM_Evaluations/
英文摘要：
With the advent of multi-modal large language models (MLLMs), datasets used for visual question answering (VQA) and referring expression comprehension have seen a resurgence. However, the most popular datasets used to evaluate MLLMs are some of the earliest ones created, and they have many known problems, including extreme bias, spurious correlations, and an inability to permit fine-grained analysis. In this paper, we pioneer evaluating recent MLLMs (LLaVA 1.5, LLaVA-NeXT, BLIP2, InstructBLIP, GPT-4V, and GPT-4o) on datasets designed to address weaknesses in earlier ones. We assess three VQA datasets: 1) TDIUC, which permits fine-grained analysis on 12 question types; 2) TallyQA, which has simple and complex counting questions; and 3) DVQA, which requires optical character recognition for chart understanding. We also study VQDv1, a dataset that requires identifying all image regions that satisfy a given query. Our experiments reveal the weaknesses of many MLLMs that have not previously been reported. Our code is integrated into the widely used LAVIS framework for MLLM evaluation, enabling the rapid assessment of future MLLMs. Project webpage: https://kevinlujian.github.io/MLLM_Evaluations/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
