原型性偏差揭示多模态评估指标中的盲点
Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics
AI总结：
本文通过构建受控诊断基准PROTOBIAS，发现并验证了多模态评估指标中存在原型性偏差，即倾向于选择视觉或社会原型性高但语义错误的图像，并提出了轻量级对比训练评估器PROTOSCORE作为缓解基线。
摘要：
自动指标广泛用于评估文生图模型，常常在基准测试、模型选择和大规模数据过滤中取代人类判断。然而，它们可能奖励看起来合理或原型性的图像，而非忠实满足提示的图像。我们识别出原型性偏差是多模态评估中的一个系统性盲点：指标可能偏好语义不正确但在视觉或社会层面具有原型性的图像，而非正确但原型性较弱的图像。我们引入PROTOBIAS，一个跨动物、物体和人口统计的受控诊断基准，其中语义正确的图像与包含单个受控语义违反的合理原型性对抗样本进行对比。基于原型理论和社会类别原型性，PROTOBIAS通过多个提示生成器、图像生成器和独立的VLM过滤器构建，并通过提示质量、人工标注和图像质量控制进行验证。使用PROTOBIAS，我们展示了广泛使用的嵌入、奖励、基于VQA和VLM作为评判的指标经常在这些对比中失败，而人类判断仍然更忠实于语义正确性。我们进一步引入PROTOSCORE，一个轻量级对比训练评估器，作为初始缓解基线。PROTOBIAS为测量原型性驱动的指标失败和开发更语义忠实的T2I评估器提供了一个聚焦基准。
英文摘要：
Automatic metrics are widely used to evaluate text-to-image models, often replacing human judgment in benchmarking, model selection, and large-scale data filtering. Yet they may reward images that look plausible or prototypical rather than images that faithfully satisfy the prompt. We identify prototypicality bias as a systematic blindspot in multimodal evaluation: metrics can prefer a semantically incorrect but visually or socially prototypical image over a correct but less prototypical one. We introduce PROTOBIAS, a controlled diagnostic benchmark across Animals, Objects, and Demography, where semantically correct images are contrasted with plausible prototypical adversaries containing a single controlled semantic violation. Grounded in prototype theory and social-category prototypicality, PROTOBIAS is constructed with multiple prompt generators, image generators, and independent VLM filters, and validated through prompt-quality, human-annotation, and image-quality controls. Using PROTOBIAS, we show that widely used embedding, reward, VQA-based, and VLM-as-judge metrics frequently fail these contrasts, while human judgments remain more faithful to semantic correctness. We further introduce PROTOSCORE, a lightweight contrastively trained evaluator, as an initial mitigation baseline. PROTOBIAS provides a focused benchmark for measuring prototypicality-driven metric failures and developing more semantically faithful T2I evaluators.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
