打破常识:WHOOPS!——一个合成与组合图像的视觉-语言基准
Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images
AI总结:
本文提出WHOOPS!视觉常识基准数据集,包含故意违背常识的合成图像,并通过解释生成等任务证明GPT3、BLIP2等先进模型仍落后于人类,旨在推动AI视觉常识推理能力的发展。
摘要:
怪异、不寻常和离奇的图像会激发观察者的好奇心,因为它们挑战了常识。例如,2022年世界杯期间发布的一张图片描绘了著名足球明星莱昂内尔·梅西和克里斯蒂亚诺·罗纳尔多下国际象棋,这俏皮地违背了我们的预期,即他们的竞争应该发生在足球场上。人类可以轻松识别和解释这些非常规图像,但AI模型能做到同样的事情吗?我们引入了WHOOPS!,一个用于视觉常识的新数据集和基准。该数据集由设计师使用公开可用的图像生成工具(如Midjourney)创建的、故意违背常识的图像组成。我们考虑了在该数据集上提出的若干任务。除了图像描述、跨模态匹配和视觉问答之外,我们还引入了一个困难的解释生成任务,在该任务中,模型必须识别并解释给定图像为何不寻常。我们的结果表明,最先进的模型(如GPT3和BLIP2)在WHOOPS!上的表现仍然落后于人类。我们希望我们的数据集能够激励开发具有更强视觉常识推理能力的AI模型。数据、模型和代码可在项目网站上获取:whoops-benchmark.github.io
英文摘要:
Weird, unusual, and uncanny images pique the curiosity of observers because they challenge commonsense. For example, an image released during the 2022 world cup depicts the famous soccer stars Lionel Messi and Cristiano Ronaldo playing chess, which playfully violates our expectation that their competition should occur on the football field. Humans can easily recognize and interpret these unconventional images, but can AI models do the same? We introduce WHOOPS!, a new dataset and benchmark for visual commonsense. The dataset is comprised of purposefully commonsense-defying images created by designers using publicly-available image generation tools like Midjourney. We consider several tasks posed over the dataset. In addition to image captioning, cross-modal matching, and visual question answering, we introduce a difficult explanation generation task, where models must identify and explain why a given image is unusual. Our results show that state-of-the-art models such as GPT3 and BLIP2 still lag behind human performance on WHOOPS!. We hope our dataset will inspire the development of AI models with stronger visual commonsense reasoning abilities. Data, models and code are available at the project website: whoops-benchmark.github.io
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。