当视觉-语言模型评判而不看：揭示信息性偏差
When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias
AI总结：
研究揭示视觉-语言模型在评判时存在的信息性偏差问题，提出BIRCH方法通过修正图像与答案的一致性提升评判可靠性，实验显示偏差降低17%，性能提升9.8%。
摘要：
视觉-语言模型作为评判者的可靠性对自动评估至关重要。尽管近期有进展，我们的分析显示，VLM-as-a-Judge在决策时往往忽视图像内容，而倾向于选择更信息丰富的答案，即使这与图像内容冲突。我们称之为信息性偏差，这显著削弱了评判的可靠性。为解决此问题，我们提出了BIRCH（Balanced Informativeness and CoRrectness with a Truthful AnCHor），一种评判范式，首先修正候选答案与图像内容的一致性，然后将答案与修正后的版本进行比较。这将评判焦点从信息性转向图像基础的正确性。在多个模型和基准上的实验表明，BIRCH将信息性偏差降低了最高17%，导致性能提升最高9.8%。我们的工作揭示了当前VLM-as-a-Judge系统中被忽视但根本性的缺陷，并突显了需要更严谨设计的必要性。
英文摘要：
The reliability of VLM-as-a-Judge is critical for the automatic evaluation of vision-language models (VLMs). Despite recent progress, our analysis reveals that VLM-as-a-Judge often pays limited attention to the image when making decisions. Instead, they often blindly favor the more informative answer, even when they can recognize it conflicts with the image content. We call this problem informativeness bias, which significantly undermines judge reliability. To address it, we propose BIRCH (Balanced Informativeness and CoRrectness with a Truthful AnCHor), a judging paradigm that first corrects inconsistencies with the image content in candidate answers, and then compares the answers against this corrected version. This shifts the judge's focus from informativeness to image-grounded correctness. Experiments on multiple models and benchmarks show that BIRCH reduces informativeness bias by up to 17%, resulting in performance gains of up to 9.8%. Our work reveals an overlooked but fundamental flaw in current VLM-as-a-Judge systems and highlights the need for more principled designs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
