不要买它！重新评估对比多模态模型的广告理解能力
Don't Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models
AI总结：
研究发现对比VLMs在广告理解任务中依赖启发式方法而非多模态推理，作者引入对抗性测试集TRADE揭示该缺陷，呼吁改进自动广告理解的操作化定义。
摘要：
图像广告是复杂的多模态刺激，通常包含不寻常的视觉元素和比喻性语言。此前关于自动广告理解的研究报告称，对比视觉与语言模型（VLMs）在广告解释检索任务上取得了令人印象深刻的零样本准确率。在此，我们检查了原始任务设置，并表明对比VLMs可以通过利用基础启发式方法来解决该任务。为了控制这一混淆因素，我们引入了TRADE，这是一个带有对抗性基础解释的新评估测试集。虽然这些解释在人类看来难以置信，但我们表明它们“愚弄”了四种不同的对比VLMs。我们的发现强调，需要改进自动广告理解的操作化定义，以真正评估VLMs的多模态推理能力。我们在 https://github.com/dmg-illc/trade 上公开了我们的代码和TRADE。
英文摘要：
Image-based advertisements are complex multimodal stimuli that often contain unusual visual elements and figurative language. Previous research on automatic ad understanding has reported impressive zero-shot accuracy of contrastive vision-and-language models (VLMs) on an ad-explanation retrieval task. Here, we examine the original task setup and show that contrastive VLMs can solve it by exploiting grounding heuristics. To control for this confound, we introduce TRADE, a new evaluation test set with adversarial grounded explanations. While these explanations look implausible to humans, we show that they "fool" four different contrastive VLMs. Our findings highlight the need for an improved operationalisation of automatic ad understanding that truly evaluates VLMs' multimodal reasoning abilities. We make our code and TRADE available at https://github.com/dmg-illc/trade .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
