利用图像到文本生成评估文本到视觉生成
Evaluating Text-to-Visual Generation with Image-to-Text Generation
AI总结：
本文提出VQAScore，借助VQA模型评估生成视觉内容与复杂文本提示的对齐，并构建含1600条组合提示和15000余条人工评分的GenAI-Bench。
摘要：
尽管生成式AI取得了重大进展，但由于缺乏有效的指标和标准化基准，全面评估仍然具有挑战性。例如，广泛使用的CLIPScore衡量（生成的）图像与文本提示之间的对齐程度，但对于涉及物体、属性和关系组合的复杂提示，它无法给出可靠的分数。原因之一是CLIP的文本编码器会出了名地表现得像一个“词袋”，将“马正在吃草”与“草正在吃马”这样的提示混为一谈。为解决这一问题，我们提出VQAScore，它使用视觉问答（VQA）模型，通过计算对简单问题“这幅图是否展示了‘{text}’？”回答“Yes”的概率来生成对齐分数。尽管比现有方法更简单，但使用现成模型计算的VQAScore在多达8个图文对齐基准上取得了最先进结果。我们还使用一个遵循文献中最佳实践的内部模型来计算VQAScore。例如，我们采用双向图像-问题编码器，使图像嵌入依赖于所问的问题（反之亦然）。我们的内部模型CLIP-FlanT5甚至优于利用专有GPT-4V的最强基线。有趣的是，尽管我们仅使用图像进行训练，VQAScore也能将文本与视频和3D模型对齐。VQAScore使研究人员能够使用捕捉真实提示组合结构的复杂文本来基准测试文本到视觉生成。我们推出GenAI-Bench，这是一个更具挑战性的基准，包含1600条组合性文本提示，需要解析场景、物体、属性、关系以及比较和逻辑等高阶推理。GenAI-Bench还为Stable Diffusion、DALL-E 3和Gen2等领先图像和视频生成模型提供了超过15000条人工评分。
英文摘要：
Despite significant progress in generative AI, comprehensive evaluation remains challenging because of the lack of effective metrics and standardized benchmarks. For instance, the widely-used CLIPScore measures the alignment between a (generated) image and text prompt, but it fails to produce reliable scores for complex prompts involving compositions of objects, attributes, and relations. One reason is that text encoders of CLIP can notoriously act as a "bag of words", conflating prompts such as "the horse is eating the grass" with "the grass is eating the horse". To address this, we introduce the VQAScore, which uses a visual-question-answering (VQA) model to produce an alignment score by computing the probability of a "Yes" answer to a simple "Does this figure show '{text}'?" question. Though simpler than prior art, VQAScore computed with off-the-shelf models produces state-of-the-art results across many (8) image-text alignment benchmarks. We also compute VQAScore with an in-house model that follows best practices in the literature. For example, we use a bidirectional image-question encoder that allows image embeddings to depend on the question being asked (and vice versa). Our in-house model, CLIP-FlanT5, outperforms even the strongest baselines that make use of the proprietary GPT-4V. Interestingly, although we train with only images, VQAScore can also align text with video and 3D models. VQAScore allows researchers to benchmark text-to-visual generation using complex texts that capture the compositional structure of real-world prompts. We introduce GenAI-Bench, a more challenging benchmark with 1,600 compositional text prompts that require parsing scenes, objects, attributes, relationships, and high-order reasoning like comparison and logic. GenAI-Bench also offers over 15,000 human ratings for leading image and video generation models such as Stable Diffusion, DALL-E 3, and Gen2.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
