Multimodal large language models (MLLMs) have broadened the scope of AI applications. Existing automatic evaluation methodologies for MLLMs are mainly limited in evaluating queries without considering user experiences, inadequately addressing the nuances of creative and associative multimodal tasks. However, the open-ended and subjective nature of such tasks poses a significant challenge to the evaluation methodology, where it is difficult to define the ground-truth answers for them. To this end, in our paper, we propose a new evaluation paradigm for MLLMs, which is evaluating MLLMs with per-sample criteria using potent MLLM as the judge. To validate the feasibility and effectiveness of this paradigm, we design a benchmark, dubbed MLLM-Bench, by curating the evaluation samples across six comprehensive cognitive levels. We benchmark 21 popular MLLMs in a pairwise-comparison fashion, showing diverse performance across models. Moreover, the validity of our benchmark manifests itself in reaching 88.02% agreement with human evaluation. We contend that the proposed paradigm explores the potential of MLLMs as effective evaluation tools with the help of per-sample criteria. See online leaderboard at \url{https://mllm-bench.llmzoo.com}.
EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning
Tencent AI Lab(腾讯 AI 实验室);The University of Hong Kong(香港大学);ARC Lab, Tencent PCG(腾讯平台与内容事业群 ARC 实验室);University of California, Berkeley(加州大学伯克利分校);Peng Cheng Laboratory(鹏城实验室)
II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models
Shenzhen Institutes of Advanced Technology, CAS(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学);M-A-P;IIT Kharagpur(印度理工学院克勒格布尔分校);Huazhong University of Science and Technology(华中科技大学);Yokohama National University(横滨国立大学);Shenzhen University(深圳大学);Zhejiang University(浙江大学);Shenzhen MSU-BIT University(深圳北理莫斯科大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Beihang University(北京航空航天大学);Dartmouth College(达特茅斯学院);University of Waterloo(滑铁卢大学)
11Plus-Bench: Demystifying Multimodal LLM Spatial Reasoning with Cognitive-Inspired Analysis
Microsoft Research(微软研究院);Language Technology Lab, University of Cambridge(语言技术实验室,剑桥大学);Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院);Department of Oncology, University of Cambridge(癌症部门,剑桥大学);Leverhulme Centre for the Future of Intelligence, University of Cambridge(未来智能中心,剑桥大学);VRAIN, Universitat Politècnica de València(VRAIN,巴塞罗那理工大学)
A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Models
The City University of Hong Kong(香港城市大学);The Chinese University of Hong Kong(香港中文大学);Shenzhen University(深圳大学);National Yang Ming Chiao Tung University(国立阳明交通大学);Taipei Veterans General Hospital(台北荣民总医院);Stanford University(斯坦福大学)
MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosis
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Southeast University(东南大学);School of Computing, National University of Singapore(新加坡国立大学计算机学院)
HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models
National Engineering Research Center for Software Engineering, Peking University(北京大学软件工程国家工程研究中心);Institute of Artificial Intelligence, China Telecom (TeleAI)(中国电信人工智能研究院);Tsinghua University(清华大学);Chinese Academy of Sciences(中国科学院);University of British Columbia(不列颠哥伦比亚大学);Renmin University of China(中国人民大学)
Aligning MLLM Benchmark With Human Preferences via Structural Equation Modeling
Interdisciplinary Artificial Intelligence Research Institute, Wuhan College(交叉学科人工智能研究 institute,武汉学院);School of Computer and Artificial Intelligence, Wuhan University of Technology(计算机与人工智能学院,武汉理工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Sanya Science and Education Innovation Park, Wuhan University of Technology(三亚科学教育创新园,武汉理工大学);Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院);School of Mathematics and Statistics, Northwestern Polytechnical University(数学与统计学院,西北工业大学);Institute of Artificial Intelligence (TeleAI) of China Telecom(中国电信人工智能研究所(TeleAI))
CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language
School of Informatics, Xiamen University, China(厦门大学信息学院);Key Lab of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian-Taiwan (XMU), Ministry of Culture and Tourism, China(福建省-台湾非物质文化遗产数字化保护与智能处理重点实验室(XMU),文化和旅游部,中国);National Language Resources Monitoring and Research Center for Education and Teaching Media, Xiamen University, China(教育与教学媒体语言资源监测与研究中心,厦门大学,中国)
Judge Before Answer: Can MLLM Discern the False Premise in Question?
School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学学院);Inner Mongolia Research Institute, Shanghai Jiao Tong University(上海交通大学内蒙古研究院)
ClinHallu: A Benchmark for Diagnosing Stage-Wise Hallucinations in Medical MLLM Reasoning
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));DAMO Academy, Alibaba Group(阿里巴巴达摩院);Hupan Lab(湖畔实验室);Zhejiang University(浙江大学)
Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);MIT-IBM Watson AI Lab, IBM Research(MIT-IBM沃森人工智能实验室,IBM研究);Stony Brook University(石溪大学);Brookhaven National Laboratory(布鲁赫斯国家实验室)
MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI
Nanyang Technological University(南洋理工大学);Tsinghua University(清华大学);Baidu Inc.(百度公司);University of California(加州大学);University of Science and Technology of China(中国科学技术大学)
Can MLLMs Understand the Deep Implication Behind Chinese Images?
Huazhong University of Science and Technology(华中科技大学);Shenzhen Institute of Advanced Technology, CAS(中国科学院深圳先进技术研究院);University of Science and Technology of China(中国科学技术大学);CDUT(成都理工大学);University of Memphis(孟菲斯大学);University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);SICAU(四川农业大学);University of Manchester(曼彻斯特大学);SWU(西南大学);UCAS(中国科学院大学);UMich(密歇根大学)
Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis
University of Virginia(弗吉尼亚大学);Columbia University(哥伦比亚大学);Vanderbilt University(范德比大学);Adobe Research(Adobe研究);Dolby Laboratories(杜比实验室);Cisco Research(思科研究);University of Southern California(南加州大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);University of Oregon(俄勒冈大学);Texas A&M University(德克萨斯大学)
The Instinctive Bias: Spurious Images lead to Illusion in MLLMs
The Hong Kong Polytechnic University(香港理工大学);The Hong Kong University of Science and Technology(香港科技大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);NVIDIA(英伟达)