II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models
Shenzhen Institutes of Advanced Technology, CAS(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学);M-A-P;IIT Kharagpur(印度理工学院克勒格布尔分校);Huazhong University of Science and Technology(华中科技大学);Yokohama National University(横滨国立大学);Shenzhen University(深圳大学);Zhejiang University(浙江大学);Shenzhen MSU-BIT University(深圳北理莫斯科大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Beihang University(北京航空航天大学);Dartmouth College(达特茅斯学院);University of Waterloo(滑铁卢大学)
The rapid advancements in the development of multimodal large language models (MLLMs) have consistently led to new breakthroughs on various benchmarks. In response, numerous challenging and comprehensive benchmarks have been proposed to more accurately assess the capabilities of MLLMs. However, there is a dearth of exploration of the higher-order perceptual capabilities of MLLMs. To fill this gap, we propose the Image Implication understanding Benchmark, II-Bench, which aims to evaluate the model's higher-order perception of images. Through extensive experiments on II-Bench across multiple MLLMs, we have made significant findings. Initially, a substantial gap is observed between the performance of MLLMs and humans on II-Bench. The pinnacle accuracy of MLLMs attains 74.8%, whereas human accuracy averages 90%, peaking at an impressive 98%. Subsequently, MLLMs perform worse on abstract and complex images, suggesting limitations in their ability to understand high-level semantics and capture image details. Finally, it is observed that most models exhibit enhanced accuracy when image sentiment polarity hints are incorporated into the prompts. This observation underscores a notable deficiency in their inherent understanding of image sentiment. We believe that II-Bench will inspire the community to develop the next generation of MLLMs, advancing the journey towards expert artificial general intelligence (AGI). II-Bench is publicly available at https://huggingface.co/datasets/m-a-p/II-Bench.
MER-Bench: Benchmarking Multimodal Emotional Reappraisal for Image-Text Memes
School of Artificial Intelligence, Anhui University(安徽大学人工智能学院);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机科学与信息工程学院);College of Information Technology, United Arab Emirates University(阿拉伯联合酋长国大学信息学院);Institute of Advanced Technology, University of Science and Technology of China(中国科学技术大学先进技术研究院)
Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);MIT-IBM Watson AI Lab, IBM Research(MIT-IBM沃森人工智能实验室,IBM研究);Stony Brook University(石溪大学);Brookhaven National Laboratory(布鲁赫斯国家实验室)
MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
Shanghai AI Laboratory(上海人工智能实验室);Shanghai Innovation Institute(上海创新研究院);USTC(中国科学技术大学);RIT(罗切斯特理工学院);HIT(哈尔滨工业大学);WHU(武汉大学);MBZUAI(Mohamed bin Zayed 人工智能大学);NUS(新加坡国立大学)
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
University of Waterloo(滑铁卢大学);Georgia Institute of Technology(佐治亚理工学院);Hong Kong University of Science and Technology(香港科技大学);Google DeepMind(谷歌DeepMind)
UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents
School of Computer Science and Engineering, Northeastern University, Shenyang, China(东北大学计算机科学与工程学院);Department of Computer Science and Technology, Tsinghua University, Beijing, China(清华大学计算机科学与技术系);Alibaba Group, Hangzhou, China(阿里巴巴集团)