评估评估者：面向组合式文本到图像生成的评价指标
Evaluating the Evaluators: Metrics for Compositional Text-to-Image Generation
AI总结：
本文系统研究了组合式文本到图像生成中的自动化评估指标，发现没有单一指标能在所有组合任务上一致表现，VQA类指标并非普遍最优，而某些嵌入类指标在特定场景下更强，强调了谨慎选择指标对可信评估和奖励模型的重要性。
摘要：
文本到图像生成技术发展迅速，但评估其输出是否真正捕捉到提示中描述的对象、属性和关系，仍是一项核心挑战。该领域的评估高度依赖自动化指标，但这些指标往往因惯例或流行度而被采用，而非经过与人类判断的验证。由于该领域的评估和所报告的进展直接取决于这些指标，理解它们在多大程度上反映人类偏好至关重要。为此，我们对广泛使用的组合式文本到图像评估指标进行了全面研究。我们的分析超越了简单的相关性，考察了这些指标在多种组合挑战下的行为，并比较了不同指标家族与人类判断的一致性。结果表明，没有任何单一指标能在所有任务上表现一致：性能随组合问题的类型而变化。值得注意的是，基于VQA的指标虽然流行，但并非始终优于其他指标，而某些基于嵌入的指标在特定场景下表现更强。仅基于图像的指标如预期那样对组合式评估贡献甚微，因为它们是为感知质量而非语义对齐而设计的。这些发现强调了谨慎且透明地选择指标的重要性，无论对于可信的评估，还是对于将其用作生成中的奖励模型。项目页面可在 https://amirkasaei.com/eval-the-evals/ 获取。
英文摘要：
Text-image generation has advanced rapidly, but assessing whether outputs truly capture the objects, attributes, and relations described in prompts remains a central challenge. Evaluation in this space relies heavily on automated metrics, yet these are often adopted by convention or popularity rather than validated against human judgment. Because evaluation and reported progress in the field depend directly on these metrics, it is critical to understand how well they reflect human preferences. To address this, we present a broad study of widely used metrics for compositional text-image evaluation. Our analysis goes beyond simple correlation, examining their behavior across diverse compositional challenges and comparing how different metric families align with human judgments. The results show that no single metric performs consistently across tasks: performance varies with the type of compositional problem. Notably, VQA-based metrics, though popular, are not uniformly superior, while certain embedding-based metrics prove stronger in specific cases. Image-only metrics, as expected, contribute little to compositional evaluation, as they are designed for perceptual quality rather than alignment. These findings underscore the importance of careful and transparent metric selection, both for trustworthy evaluation and for their use as reward models in generation. Project page is available at https://amirkasaei.com/eval-the-evals/ .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
