什么是好的指标？评估文本到图像一致性的自动指标
What makes a good metric? Evaluating automatic metrics for text-to-image consistency
AI总结：
本研究分析了CLIPScore、TIFA、VPEval和DSG四种文本到图像一致性自动指标的构建效度，发现它们均无法满足所有期望属性，对语言和视觉属性敏感度不足，且基于VQA的指标可能依赖文本捷径，其定量评估能力存疑。
摘要：
语言模型正越来越多地作为组件被纳入更大的AI系统中，用于从提示优化到自动评估的各种目的。在本研究中，我们分析了四种近期常用的测量文本到图像一致性方法的构建效度——CLIPScore、TIFA、VPEval和DSG——这些方法依赖语言模型和/或VQA模型作为组件。我们将文本-图像一致性指标的构建效度定义为该类指标应具备的一组期望属性，并发现没有测试的指标能全部满足。我们发现这些指标对语言和视觉属性缺乏足够的敏感性。其次，我们发现TIFA、VPEval和DSG提供了超越CLIPScore的新颖信息，但它们彼此之间也高度相关。我们还消融了文本-图像一致性指标的不同方面，发现并非所有模型组件都是严格必要的，这也是对视觉信息敏感度不足的另一个症状。最后，我们表明所有三个基于VQA的指标可能都依赖于熟悉的文本捷径（如QA中的yes-bias），这使它们作为模型性能定量评估的适用性受到质疑。
英文摘要：
Language models are increasingly being incorporated as components in larger AI systems for various purposes, from prompt optimization to automatic evaluation. In this work, we analyze the construct validity of four recent, commonly used methods for measuring text-to-image consistency - CLIPScore, TIFA, VPEval, and DSG - which rely on language models and/or VQA models as components. We define construct validity for text-image consistency metrics as a set of desiderata that text-image consistency metrics should have, and find that no tested metric satisfies all of them. We find that metrics lack sufficient sensitivity to language and visual properties. Next, we find that TIFA, VPEval and DSG contribute novel information above and beyond CLIPScore, but also that they correlate highly with each other. We also ablate different aspects of the text-image consistency metrics and find that not all model components are strictly necessary, also a symptom of insufficient sensitivity to visual information. Finally, we show that all three VQA-based metrics likely rely on familiar text shortcuts (such as yes-bias in QA) that call their aptitude as quantitative evaluations of model performance into question.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
