MM-JudgeBias：一个评估MLLM-as-a-Judge中组合偏见的基准
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
AI总结：
本文提出MM-JudgeBias基准，通过引入控制扰动和两个互补指标，评估MLLM-as-a-Judge中的组合偏见，揭示了九种偏见类型，并发现现有模型存在模态忽视和评估倾向性问题。
摘要：
多模态大语言模型（MLLMs）越来越多地被用作自动评估者，即MLLM-as-a-Judge。然而，其可靠性和对偏见的易感性仍缺乏研究。我们发现许多MLLM评估者无法可靠整合关键视觉或文本线索，导致证据缺失或不匹配时评估不可靠，并在语义无关扰动下表现不稳定。为此，我们系统定义了MLLM-as-a-Judge中的组合偏见，并引入MM-JudgeBias基准进行评估。MM-JudgeBias引入了跨查询、图像和响应的受控扰动，并通过两个互补指标：偏见偏差（BD）用于灵敏度，偏见一致性（BC）用于稳定性来评估模型行为。我们的超过1800个经过精心挑选和优化的多模态样本数据集，来自29个源基准，使可以在多样化任务和领域中对九种偏见类型进行细粒度诊断。在26种最先进的MLLMs上进行的实验揭示了系统性的模态忽视和不对称的评估倾向，凸显了对更可靠评估者的需求。
英文摘要：
Multimodal Large Language Models (MLLMs) have been increasingly used as automatic evaluators-a paradigm known as MLLM-as-a-Judge. However, their reliability and vulnerabilities to biases remain underexplored. We find that many MLLM judges fail to reliably integrate key visual or textual cues, yielding unreliable evaluations when evidence is missing or mismatched, and exhibiting instability under semantically irrelevant perturbations. To address this, we systematically define Compositional Bias in MLLM-as-a-Judge systems and introduce MM-JudgeBias, a benchmark for evaluating it. MM-JudgeBias introduces controlled perturbations across Query, Image, and Response, and evaluates model behavior via two complementary metrics: Bias-Deviation (BD) for sensitivity and Bias-Conformity (BC) for stability. Our dataset of over 1,800 curated and refined multimodal samples, drawn from 29 source benchmarks, enables a fine-grained diagnosis of nine bias types across diverse tasks and domains. Experiments on 26 state-of-the-art MLLMs reveal systematic modality neglect and asymmetric evaluation tendencies, underscoring the need for more reliable judges.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
