arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.03173cs.CLcs.AIcs.CV
收藏

MULTI:多模态理解排行榜(文本与图像)

MULTI: Multimodal Understanding Leaderboard with Text and Images

X-LANCE Lab, School of Computer Science, Key Laboratory of Artificial Intelligence\ of Education, Shanghai Jiao Tong University, Shanghai 200240 , China;Jiangsu Key Lab of Language Computing, Suzhou 215123 , China;College of Computing;Data Science, Nanyang Technological University, Singapore 639798 , Singapore;Suzhou Laboratory, Suzhou 215123 , China

Zichen Zhu, Yang Xu, Lu Chen, Jingkai Yang, Yichuan Ma, Yiming Sun, Hailin Wen, Jiaqi Liu, Jinyu Cai, Yingzi Ma, Situo Zhang, Zihan Zhao, Liangtai Sun, Kai Yu

AI总结:

本文提出MULTI,一个基于真实考题的中文多模态数据集,含18,000余题及精英子集,评估显示MLLMs与人类专家仍有差距,为专家级AI发展提供平台。

摘要:

多模态大语言模型(MLLMs)的快速发展引发了它们与人类表现相比如何的问题。虽然现有数据集通常包含合成或过于简单的任务,但一些模型已经超越了人类专家基线。在本文中,我们提出了MULTI,一个源自真实考试题目的中文多模态数据集。MULTI包含超过18,000道精心挑选和优化的问题,使用真实考试标准评估模型,涵盖图像-文本理解、复杂推理和知识回忆。此外,我们还引入了MULTI-Elite,一个包含500道精选难题的子集,以及MULTI-Extend,包含超过4,500条外部知识上下文片段,用于测试上下文学习能力。我们的评估显示MLLMs有巨大的进步空间,其中Qwen2-VL-72B在MULTI上达到76.9%的准确率,在MULTI-Elite上达到53.1%,领先于25个被评估的模型,而人类专家基线分别为86.1%和73.1%。MULTI不仅作为一个稳健的评估平台,还为专家级AI的发展铺平了道路。

英文摘要:

The rapid development of multimodal large language models (MLLMs) raises the question of how they compare to human performance. While existing datasets often feature synthetic or overly simplistic tasks, some models have already surpassed human expert baselines. In this paper, we present MULTI, a Chinese multimodal dataset derived from authentic examination questions. Comprising over 18,000 carefully selected and refined questions, MULTI evaluates models using real-world examination standards, encompassing image-text comprehension, complex reasoning, and knowledge recall. Additionally, We also introduce MULTI-Elite, a 500-question selected hard subset, and MULTI-Extend with more than 4,500 external knowledge context pieces for testing in-context learning capabilities. Our evaluation highlights substantial room for MLLM advancement, with Qwen2-VL-72B achieving a 76.9% accuracy on MULTI and 53.1% on MULTI-Elite leading 25 evaluated models, compared to human expert baselines of 86.1% and 73.1%. MULTI serves not only as a robust evaluation platform but also paves the way for the development of expert-level AI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑