MM-Vet：评估大型多模态模型的综合能力
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
AI总结：
本文提出 MM-Vet 基准，通过定义 6 项核心视觉-语言能力及其 16 种组合，并用基于 LLM 的统一评估器评价开放式输出，系统考查大型多模态模型的综合能力。
摘要：
我们提出 MM-Vet，这是一个在复杂多模态任务上考查大型多模态模型（LMM）的评估基准。近期的 LMM 已展现出多种引人关注的能力，例如解答写在黑板上的数学题、对新闻图像中的事件和名人进行推理，以及解释视觉笑话。模型的快速发展给评估基准构建带来了挑战。问题包括：（1）如何系统地组织并评估复杂的多模态任务；（2）如何设计能够适用于不同问题和答案类型的评估指标；（3）如何提供超越简单性能排名的模型洞见。为此，我们提出 MM-Vet，其设计基于这样一种洞见：解决复杂任务的引人关注能力，通常来自通才模型能够整合不同的核心视觉-语言（VL）能力。MM-Vet 定义了 6 项核心 VL 能力，并考查由这些能力组合得到的 16 种受关注的整合方式。在评估指标方面，我们针对开放式输出提出了一种基于 LLM 的评估器。该评估器能够跨不同问题类型和答案风格进行评估，从而形成统一的评分指标。我们在 MM-Vet 上评估了具有代表性的 LMM，为不同 LMM 系统范式和模型的能力提供了洞见。
英文摘要：
We propose MM-Vet, an evaluation benchmark that examines large multimodal models (LMMs) on complicated multimodal tasks. Recent LMMs have shown various intriguing abilities, such as solving math problems written on the blackboard, reasoning about events and celebrities in news images, and explaining visual jokes. Rapid model advancements pose challenges to evaluation benchmark development. Problems include: (1) How to systematically structure and evaluate the complicated multimodal tasks; (2) How to design evaluation metrics that work well across question and answer types; and (3) How to give model insights beyond a simple performance ranking. To this end, we present MM-Vet, designed based on the insight that the intriguing ability to solve complicated tasks is often achieved by a generalist model being able to integrate different core vision-language (VL) capabilities. MM-Vet defines 6 core VL capabilities and examines the 16 integrations of interest derived from the capability combination. For evaluation metrics, we propose an LLM-based evaluator for open-ended outputs. The evaluator enables the evaluation across different question types and answer styles, resulting in a unified scoring metric. We evaluate representative LMMs on MM-Vet, providing insights into the capabilities of different LMM system paradigms and models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
