COCO-Counterfactuals：为图像-文本对自动构建的反事实示例
COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs
AI总结：
本文提出利用文本到图像扩散模型自动生成反事实示例的框架，并构建基于MS-COCO的多模态反事实数据集COCO-Counterfactuals，用于评估模型鲁棒性并提升视觉-语言模型的域外泛化能力。
摘要：
反事实示例在自然语言处理（NLP）领域已被证明对评估和提升语言模型对数据集中虚假相关性的鲁棒性具有重要价值。尽管反事实示例在NLP中展现了实用性，多模态反事实示例由于难以创建仅发生最小反事实变化的成对图像-文本数据而相对未被探索。为应对这一挑战，我们引入了一个可扩展的框架，利用文本到图像扩散模型自动生成反事实示例。我们使用该框架创建了COCO-Counterfactuals，这是一个基于MS-COCO数据集的成对图像和文本描述的多模态反事实数据集。我们通过人工评估验证了COCO-Counterfactuals的质量，并表明现有的多模态模型受到我们的反事实图像-文本对的挑战。此外，我们证明了COCO-Counterfactuals在通过训练数据增强改善多模态视觉-语言模型的域外泛化方面的实用性。
英文摘要：
Counterfactual examples have proven to be valuable in the field of natural language processing (NLP) for both evaluating and improving the robustness of language models to spurious correlations in datasets. Despite their demonstrated utility for NLP, multimodal counterfactual examples have been relatively unexplored due to the difficulty of creating paired image-text data with minimal counterfactual changes. To address this challenge, we introduce a scalable framework for automatic generation of counterfactual examples using text-to-image diffusion models. We use our framework to create COCO-Counterfactuals, a multimodal counterfactual dataset of paired image and text captions based on the MS-COCO dataset. We validate the quality of COCO-Counterfactuals through human evaluations and show that existing multimodal models are challenged by our counterfactual image-text pairs. Additionally, we demonstrate the usefulness of COCO-Counterfactuals for improving out-of-domain generalization of multimodal vision-language models via training data augmentation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
