Multimodal Large Language Models (MLLMs) have gained significant attention recently, showing remarkable potential in artificial general intelligence. However, assessing the utility of MLLMs presents considerable challenges, primarily due to the absence of multimodal benchmarks that align with human preferences. Drawing inspiration from the concept of LLM-as-a-Judge within LLMs, this paper introduces a novel benchmark, termed MLLM-as-a-Judge, to assess the ability of MLLMs in assisting judges across diverse modalities, encompassing three distinct tasks: Scoring Evaluation, Pair Comparison, and Batch Ranking. Our study reveals that, while MLLMs demonstrate remarkable human-like discernment in Pair Comparison, there is a significant divergence from human preferences in Scoring Evaluation and Batch Ranking. Furthermore, a closer examination reveals persistent challenges in the judgment capacities of LLMs, including diverse biases, hallucinatory responses, and inconsistencies in judgment, even in advanced models such as GPT-4V. These findings emphasize the pressing need for enhancements and further research efforts to be undertaken before regarding MLLMs as fully reliable evaluators. In light of this, we advocate for additional efforts dedicated to supporting the continuous development within the domain of MLLM functioning as judges. The code and dataset are publicly available at our project homepage: \url{https://mllm-judge.github.io/}.
Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity
Salesforce AI Research（Salesforce AI 研究院）；University of Cambridge（剑桥大学）；University of Colorado Boulder（科罗拉多大学博尔德分校）；Carnegie Mellon University（卡内基梅隆大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）
Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis
University of Virginia（弗吉尼亚大学）；Columbia University（哥伦比亚大学）；Vanderbilt University（范德比大学）；Adobe Research（Adobe研究）；Dolby Laboratories（杜比实验室）；Cisco Research（思科研究）；University of Southern California（南加州大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of Oregon（俄勒冈大学）；Texas A&M University（德克萨斯大学）
Judge Before Answer: Can MLLM Discern the False Premise in Question?
School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；Inner Mongolia Research Institute, Shanghai Jiao Tong University（上海交通大学内蒙古研究院）
VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI
Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；Tencent Robotics X（腾讯Robotics X实验室）；University of Toronto（多伦多大学）；Zhili College, Tsinghua University（清华大学志道书院）
StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding
Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）
Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Artificial Intelligence, Shenzhen University（深圳大学人工智能学院）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）；The Hong Kong University of Science and Technology（香港科学与技术大学）；Renmin University of China（中国人民大学）；National Yang Ming Chiao Tung University；Taipei Veterans General Hospital（台北荣民总医院）；School of Biomedical Engineering, Shenzhen University（深圳大学生物医学工程学院）
EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Beijing Future Brain Education Technology Co., Ltd.（北京未来脑教育科技有限公司）；The Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）；Guangxi Zhuang Autonomous Region Big Data Research Institute（广西壮族自治区大数据研究院）
Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM
Zhejiang University（浙江大学）；Shanghai AI Laboratory（上海人工智能实验室）；Tongji University（同济大学）；Nanjing University（南京大学）；East China Normal University（华东师范大学）；Shanghai Jiaotong University（上海交通大学）；The Chinese University of Hong Kong（香港中文大学）
Is Cognition Consistent with Perception? Assessing and Mitigating Multimodal Knowledge Conflicts in Document Understanding
Zhejiang Key Laboratory of Accessible Perception and Intelligent Systems, Zhejiang University（浙江可感知智能系统重点实验室，浙江大学）；Alibaba Group（阿里巴巴集团）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and DataSecurity（杭州高新技术区（滨江）区块链与数据安全研究院）
PENDULUM: A Benchmark for Assessing Sycophancy in Multimodal Large Language Models
King Fahd University of Petroleum and Minerals（法赫德国王石油与矿业大学）；University of Western Australia（西澳大学）；Ontario Tech University（安大略理工大学）；SDAIA-KFUPM Joint Research Center for Artificial Intelligence（SDAIA-KFUPM人工智能联合研究中心）
Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math
Lehigh University（里海大学）；Squirrel Ai Learning（松鼠AI学习）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Michigan State University（密歇根州立大学）
