arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.05862cs.CLcs.AIcs.CV
收藏

II-Bench:面向多模态大语言模型的图像隐含含义理解基准

II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models

Shenzhen Institutes of Advanced Technology, CAS(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学);M-A-P;IIT Kharagpur(印度理工学院克勒格布尔分校);Huazhong University of Science and Technology(华中科技大学);Yokohama National University(横滨国立大学);Shenzhen University(深圳大学);Zhejiang University(浙江大学);Shenzhen MSU-BIT University(深圳北理莫斯科大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Beihang University(北京航空航天大学);Dartmouth College(达特茅斯学院);University of Waterloo(滑铁卢大学)
Ziqiang Liu, Feiteng Fang, Xi Feng, Xinrun Du, Chenhao Zhang, Zekun Wang, Yuelin Bai, Qixuan Zhao, Liyang Fan, Chengguang Gan, Hongquan Lin, Jiaming Li, Yuanshe… 展开作者

Ziqiang Liu, Feiteng Fang, Xi Feng, Xinrun Du, Chenhao Zhang, Zekun Wang, Yuelin Bai, Qixuan Zhao, Liyang Fan, Chengguang Gan, Hongquan Lin, Jiaming Li, Yuansheng Ni, Haihong Wu, Yaswanth Narsupalli, Zhigang Zheng, Chengming Li, Xiping Hu, Ruifeng Xu, Xiaojun Chen, Min Yang, Jiaheng Liu, Ruibo Liu, Wenhao Huang, Ge Zhang, Shiwen Ni

AI总结:

针对多模态大语言模型高阶感知能力评估缺失的问题,提出图像隐含含义理解基准II-Bench,实验发现模型与人类表现差距显著,情感理解存在缺陷,将推动下一代MLLMs发展。

摘要:

多模态大语言模型(MLLMs)的快速发展不断在各类基准测试上取得新突破。为此,研究者提出了众多具有挑战性的综合基准,以更准确地评估MLLMs的能力。然而,目前针对MLLMs高阶感知能力的探索仍十分匮乏。为填补这一空白,我们提出了图像隐含含义理解基准II-Bench,旨在评估模型对图像的高阶感知能力。\n我们通过在II-Bench上对多个MLLMs开展大量实验,得出了重要发现。首先,MLLMs在II-Bench上的表现与人类存在显著差距:MLLMs的最高准确率为74.8%,而人类平均准确率达90%,最高可达98%。其次,MLLMs在抽象、复杂图像上的表现更差,表明其在理解高层语义和捕捉图像细节方面存在局限。最后,实验发现当在提示词中加入图像情感极性提示时,大多数模型的准确率都会提升,这一结果凸显出它们在图像情感内在理解能力上的明显不足。我们相信II-Bench将激励学界开发下一代MLLMs,推动向专家级通用人工智能(AGI)迈进。II-Bench已在https://huggingface.co/datasets/m-a-p/II-Bench 公开可用。

英文摘要:

The rapid advancements in the development of multimodal large language models (MLLMs) have consistently led to new breakthroughs on various benchmarks. In response, numerous challenging and comprehensive benchmarks have been proposed to more accurately assess the capabilities of MLLMs. However, there is a dearth of exploration of the higher-order perceptual capabilities of MLLMs. To fill this gap, we propose the Image Implication understanding Benchmark, II-Bench, which aims to evaluate the model's higher-order perception of images. Through extensive experiments on II-Bench across multiple MLLMs, we have made significant findings. Initially, a substantial gap is observed between the performance of MLLMs and humans on II-Bench. The pinnacle accuracy of MLLMs attains 74.8%, whereas human accuracy averages 90%, peaking at an impressive 98%. Subsequently, MLLMs perform worse on abstract and complex images, suggesting limitations in their ability to understand high-level semantics and capture image details. Finally, it is observed that most models exhibit enhanced accuracy when image sentiment polarity hints are incorporated into the prompts. This observation underscores a notable deficiency in their inherent understanding of image sentiment. We believe that II-Bench will inspire the community to develop the next generation of MLLMs, advancing the journey towards expert artificial general intelligence (AGI). II-Bench is publicly available at https://huggingface.co/datasets/m-a-p/II-Bench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑