arXiv 2505.007592026-09-03cs.CVcs.AIcs.CL

多模态语言模型作为文本到图像模型的评估器

Multimodal Language Models as Text-to-Image Model Evaluators

FAIR at Meta - Montreal, New York（Meta 蒙特利尔 FAIR）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Mila；McGill University（麦吉尔大学）；Canada CIFAR AI chair（加拿大 CIFAR 人工智能主席）

Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat, Koustuv Sinha, Melissa Hall, Amy Zhang, Michal Drozdzal, Adriana Romero-Soriano

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究提出MT2IE框架，以MLLM为评估智能体，生成少量提示即可高效准确评估T2I模型，其排名比现有指标更忠实一致，还可定制基准，助力动态评估框架发展。

摘要：

文本到图像（T2I）生成模型的持续改进导致依赖静态数据集的自动评估基准逐渐被淘汰，促使研究人员寻找评估T2I进展的替代方法。我们提出了多模态文本到图像评估框架（MT2IE），该框架中单个多模态大语言模型（MLLM）充当评估智能体，迭代生成评估提示并对生成的图像进行评分。我们表明，MT2IE的图像-文本一致性得分与人类判断的相关性高于文献中先前提出的指标。MT2IE生成的提示在探测T2I模型性能方面效率很高：仅用20个生成的评估提示就能准确恢复三个结构不同基准的官方T2I模型排名，比基准自身的提示集少28至105倍。与CLIPScore、VIEScore和VQAScore等现有评估指标相比，在使用相同数量提示时，MT2IE的T2I模型排名更忠实，且在多个评估种子间的一致性高得多。MT2IE还能根据被测模型调整评估：基于模型自身测得的性能重写每个提示，生成定制的每模型基准，该基准仍能恢复官方排名，并将被测模型保持在信息丰富的评分范围内。我们希望这些结果能鼓励动态和交互式评估框架的开发，并缓解自动评估基准被淘汰的情况。

英文摘要：

The steady improvements of text-to-image (T2I) generative models lead to slow deprecation of automatic evaluation benchmarks that rely on static datasets, motivating researchers to seek alternative ways to evaluate T2I progress. We present Multimodal Text-to-Image Eval (MT2IE), an evaluation framework in which a single multimodal large language model (MLLM) acts as an evaluator agent, iteratively generating the evaluation prompts and scoring the resulting images. We show that MT2IE's image-text consistency scores have higher correlation with human judgment than metrics previously introduced in the literature. MT2IE generates prompts that are efficient at probing T2I model performance: closely recovering the official T2I model rankings of three structurally distinct benchmarks from just 20 generated evaluation prompts, 28-105x fewer than the benchmarks' own prompt sets. When compared to existing evaluation metrics such as CLIPScore, VIEScore, and VQAScore, MT2IE's T2I model rankings are more faithful and far more consistent across multiple evaluation seeds when using the same number of prompts. MT2IE can also adapt evaluation to the model being tested: rewriting each prompt based on the model's own measured performance to produce a bespoke per-model benchmark that still recovers the official rankings and keeps the evaluated model in an informative scoring range. We hope that these results will encourage the development of dynamic and interactive evaluation frameworks, and mitigate the deprecation of automatic evaluation benchmarks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。