Commonsense-T2I挑战：文本到图像生成模型能理解常识吗？
Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?
AI总结：
该研究提出Commonsense-T2I任务与基准，通过成对对抗提示评估T2I模型的视觉常识推理能力，测试发现现有SOTA模型准确率较低，GPT丰富提示也无法解决该问题，旨在推动相关领域发展。
摘要：
我们提出了一项用于评估文本到图像（T2I）生成模型生成符合现实生活常识的图像能力的新任务与基准，将其命名为Commonsense-T2I。给定两个包含相同动作词集合、仅存在细微差异的对抗性文本提示，例如“没有通电的灯泡”与“通了电的灯泡”，我们评估T2I模型是否能够进行视觉常识推理，例如对应生成符合“灯泡未点亮”与“灯泡点亮”的图像。Commonsense-T2I构成一项对抗性挑战，提供成对的文本提示以及对应的预期输出。该数据集由专家精心手工整理，并标注了细粒度标签，例如常识类型、预期输出的可能性，以辅助分析模型行为。我们对多种最先进（SOTA）的T2I模型进行了基准测试，结果令人意外：图像合成效果与现实照片之间仍存在巨大差距——即使是DALL-E 3模型在Commonsense-T2I上的准确率仅为48.92%，而Stable Diffusion XL模型的准确率仅为24.92%。我们的实验表明，经GPT丰富的提示无法解决这一挑战，我们还针对该缺陷的可能原因进行了详细分析。我们期望Commonsense-T2I能够作为T2I常识检验的高质量评估基准，推动现实图像生成领域的发展。
英文摘要：
We present a novel task and benchmark for evaluating the ability of text-to-image(T2I) generation models to produce images that align with commonsense in real life, which we call Commonsense-T2I. Given two adversarial text prompts containing an identical set of action words with minor differences, such as "a lightbulb without electricity" v.s. "a lightbulb with electricity", we evaluate whether T2I models can conduct visual-commonsense reasoning, e.g. produce images that fit "the lightbulb is unlit" vs. "the lightbulb is lit" correspondingly. Commonsense-T2I presents an adversarial challenge, providing pairwise text prompts along with expected outputs. The dataset is carefully hand-curated by experts and annotated with fine-grained labels, such as commonsense type and likelihood of the expected outputs, to assist analyzing model behavior. We benchmark a variety of state-of-the-art (sota) T2I models and surprisingly find that, there is still a large gap between image synthesis and real life photos--even the DALL-E 3 model could only achieve 48.92% on Commonsense-T2I, and the stable diffusion XL model only achieves 24.92% accuracy. Our experiments show that GPT-enriched prompts cannot solve this challenge, and we include a detailed analysis about possible reasons for such deficiency. We aim for Commonsense-T2I to serve as a high-quality evaluation benchmark for T2I commonsense checking, fostering advancements in real life image generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
