arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.11897cs.CV
收藏

TIFA:利用问答进行准确且可解释的文本到图像忠实度评估

TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering

University of Washington(华盛顿大学);Allen Institute for AI(艾伦人工智能研究所)

Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, Noah A Smith

AI总结:

本文提出基于 VQA 的无参考自动指标 TIFA,并构建含 4K 文本、25K 问题的 TIFA v1.0 基准,以更贴合人类判断地评估并揭示文本到图像模型的组合等短板。

摘要:

尽管成千上万的研究人员、工程师和艺术家正在积极致力于改进文本到图像生成模型,但这些系统往往无法生成与文本输入准确对齐的图像。我们提出 TIFA(Text-to-Image Faithfulness evaluation with question Answering,基于问答的文本到图像忠实度评估),这是一种通过视觉问答(VQA)来衡量生成图像对其文本输入忠实度的自动评估指标。具体而言,给定一个文本输入,我们使用语言模型自动生成若干问答对。我们通过检查现有 VQA 模型能否利用生成图像回答这些问题来计算图像忠实度。TIFA 是一种无参考指标,支持对生成图像进行细粒度且可解释的评估。与现有指标相比,TIFA 与人类判断具有更好的相关性。基于这一方法,我们推出 TIFA v1.0,该基准包含 4000 条多样化文本输入以及覆盖 12 个类别(物体、计数等)的 25000 个问题。我们使用 TIFA v1.0 对现有文本到图像模型进行了全面评估,并强调当前模型的局限与挑战。例如,我们发现当前文本到图像模型尽管在颜色和材质方面表现良好,但在计数、空间关系以及组合多个物体方面仍然困难。我们希望该基准能够帮助审慎衡量文本到图像合成的研究进展,并为后续研究提供有价值的洞见。

英文摘要:

Despite thousands of researchers, engineers, and artists actively working on improving text-to-image generation models, systems often fail to produce images that accurately align with the text inputs. We introduce TIFA (Text-to-Image Faithfulness evaluation with question Answering), an automatic evaluation metric that measures the faithfulness of a generated image to its text input via visual question answering (VQA). Specifically, given a text input, we automatically generate several question-answer pairs using a language model. We calculate image faithfulness by checking whether existing VQA models can answer these questions using the generated image. TIFA is a reference-free metric that allows for fine-grained and interpretable evaluations of generated images. TIFA also has better correlations with human judgments than existing metrics. Based on this approach, we introduce TIFA v1.0, a benchmark consisting of 4K diverse text inputs and 25K questions across 12 categories (object, counting, etc.). We present a comprehensive evaluation of existing text-to-image models using TIFA v1.0 and highlight the limitations and challenges of current models. For instance, we find that current text-to-image models, despite doing well on color and material, still struggle in counting, spatial relations, and composing multiple objects. We hope our benchmark will help carefully measure the research progress in text-to-image synthesis and provide valuable insights for further research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑