arXiv 2404.160062024-04-24cs.CV

MMT-Bench：面向多任务AGI评估大型视觉语言模型的一个全面多模态基准

MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI

Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；The University of Hong Kong（香港大学）；The University of Adelaide（阿德莱德大学）；Zhejiang University（浙江大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）

Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, … 展开作者

Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, Wenqi Shao

备注：: 77 pages, 41 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究提出全面多模态基准MMT-Bench，包含31325个多选视觉问题覆盖32个元任务和162个子任务，评估30个LVLMs揭示了重大挑战，旨在推动实现通用多模态智能。

摘要：

大型视觉语言模型（LVLMs）在视觉对话和具身导航等通用多模态应用中取得了显著进展。然而，现有的多模态评估基准仅覆盖了测试基础能力的有限多模态任务，不足以追踪LVLM的发展。在本研究中，我们提出了MMT-Bench，这是一个旨在跨大量多模态任务评估LVLMs的全面基准，这些任务需要专家知识以及审慎的视觉识别、定位、推理和规划。MMT-Bench包含来自自动驾驶和具身导航等各种多模态场景的$31,325$个精心筛选的多选视觉问题，涵盖了多模态理解中的$32$个核心元任务和$162$个子任务。凭借其广泛的任务覆盖面，MMT-Bench能够使用任务图评估LVLMs，从而促进发现域内和域外任务。涉及$30$个LVLMs（如专有的GPT-4V、GeminiProVision和开源的InternVL-Chat）的评估结果，突显了MMT-Bench带来的重大挑战。我们期望MMT-Bench能激励社区开发旨在实现通用多模态智能的下一代多模态基础模型。

英文摘要：

Large Vision-Language Models (LVLMs) show significant strides in general-purpose multimodal applications such as visual dialogue and embodied navigation. However, existing multimodal evaluation benchmarks cover a limited number of multimodal tasks testing rudimentary capabilities, falling short in tracking LVLM development. In this study, we present MMT-Bench, a comprehensive benchmark designed to assess LVLMs across massive multimodal tasks requiring expert knowledge and deliberate visual recognition, localization, reasoning, and planning. MMT-Bench comprises $31,325$ meticulously curated multi-choice visual questions from various multimodal scenarios such as vehicle driving and embodied navigation, covering $32$ core meta-tasks and $162$ subtasks in multimodal understanding. Due to its extensive task coverage, MMT-Bench enables the evaluation of LVLMs using a task map, facilitating the discovery of in- and out-of-domain tasks. Evaluation results involving $30$ LVLMs such as the proprietary GPT-4V, GeminiProVision, and open-sourced InternVL-Chat, underscore the significant challenges posed by MMT-Bench. We anticipate that MMT-Bench will inspire the community to develop next-generation multimodal foundation models aimed at achieving general-purpose multimodal intelligence.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。