FineGRAIN：通过视觉语言模型法官评估文本到图像模型的故障模式
FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges
AI总结：
FineGRAIN通过视觉语言模型评估文本到图像模型的故障模式，揭示属性保真度和物体表示的系统性错误，强调了针对性基准测试对生成模型可靠性的重要性。
摘要：
文本到图像（T2I）模型能够生成视觉上令人印象深刻的图像，但它们常常无法准确捕捉用户提示中的特定属性，例如指定颜色的正确物体数量。这种错误的多样性凸显了需要一个分层评估框架，该框架能够比较不同图像生成模型的提示遵循能力。同时，视觉语言模型（VLMs）的基准测试并未跟上VLMs所用于注释的场景复杂性的步伐。在本工作中，我们提出了一种联合评估T2I模型和VLMs的结构化方法，通过测试VLMs是否能够识别由T2I模型在具有挑战性提示下生成的图像中的27种特定故障模式。我们的第二个贡献是包含由5个T2I模型（Flux、SD3-Medium、SD3-Large、SD3.5-Medium、SD3.5-Large）生成的提示和图像以及相应由VLMs（Molmo、InternVL3、Pixtral）注释的图像，由LLM（Llama3）注释的测试集，以测试VLMs是否能够正确识别生成图像中的故障模式。通过在精心整理的提示集上分析故障模式，我们揭示了属性保真度和物体表示中的系统性错误。我们的发现表明，当前的度量标准不足以捕捉这些细微的错误，突显了针对生成模型可靠性与可解释性的针对性基准测试的重要性。
英文摘要：
Text-to-image (T2I) models are capable of generating visually impressive images, yet they often fail to accurately capture specific attributes in user prompts, such as the correct number of objects with the specified colors. The diversity of such errors underscores the need for a hierarchical evaluation framework that can compare prompt adherence abilities of different image generation models. Simultaneously, benchmarks of vision language models (VLMs) have not kept pace with the complexity of scenes that VLMs are used to annotate. In this work, we propose a structured methodology for jointly evaluating T2I models and VLMs by testing whether VLMs can identify 27 specific failure modes in the images generated by T2I models conditioned on challenging prompts. Our second contribution is a dataset of prompts and images generated by 5 T2I models (Flux, SD3-Medium, SD3-Large, SD3.5-Medium, SD3.5-Large) and the corresponding annotations from VLMs (Molmo, InternVL3, Pixtral) annotated by an LLM (Llama3) to test whether VLMs correctly identify the failure mode in a generated image. By analyzing failure modes on a curated set of prompts, we reveal systematic errors in attribute fidelity and object representation. Our findings suggest that current metrics are insufficient to capture these nuanced errors, highlighting the importance of targeted benchmarks for advancing generative model reliability and interpretability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
