arXiv 2306.055232023-10-31cs.CLcs.AIcs.CVcs.MM

FACTIFY3M：一个通过5W问答实现可解释性的多模态事实核查基准

FACTIFY3M: A Benchmark for Multimodal Fact Verification with Explainability through 5W Question-Answering

University of South Carolina（南卡罗来纳大学）；UCLA（加州大学洛杉矶分校）；Amazon AI（亚马逊人工智能）；Stanford University（斯坦福大学）

Megha Chakraborty, Khushbu Pahwa, Anku Rani, Shreyas Chatterjee, Dwip Dalal, Harshit Dave, Ritvik G, Preethi Gurumurthy, Adarsh Mahor, Samahriti Mukherjee, Adit… 展开作者

Megha Chakraborty, Khushbu Pahwa, Anku Rani, Shreyas Chatterjee, Dwip Dalal, Harshit Dave, Ritvik G, Preethi Gurumurthy, Adarsh Mahor, Samahriti Mukherjee, Aditya Pakala, Ishan Paul, Janvita Reddy, Arghya Sarkar, Kinjal Sensharma, Aman Chadha, Amit P. Sheth, Amitava Das

备注：: arXiv admin note: text overlap with arXiv:2305.04329

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出包含300万样本的多模态事实核查基准FACTIFY3M，融合文本声明、ChatGPT改写、关联图像、稳定扩散生成图像、像素级热力图、5W问答对及对抗性假新闻，以提升可解释性和检测能力。

摘要：

打击虚假信息是当前紧迫的社会危机之一——约67%的美国民众认为虚假信息带来了大量不确定性，其中10%的人明知故犯地传播虚假信息。有证据表明，虚假信息能够操纵民主进程和公众舆论，引发股市动荡、社会恐慌与焦虑，甚至在危机期间导致死亡。因此，虚假信息应当被及时识别，并在可能的情况下加以缓解。社交媒体平台上每天约有32亿张图片和72万小时的视频被分享，对多模态虚假信息进行可扩展检测需要高效的事实核查。尽管基于文本的自动事实核查（如FEVER、LIAR）已取得进展，但研究界在多模态事实核查方面仍缺乏实质性投入。为填补这一空白，我们提出了FACTIFY 3M，这是一个包含300万个样本的数据集，通过多模态假新闻数据集拓展了事实核查领域的边界，同时借助5W问答概念提供可解释性。该数据集的显著特征包括：（i）文本声明，（ii）ChatGPT生成的改写声明，（iii）关联图像，（iv）稳定扩散生成的附加图像（即视觉改写），（v）像素级图像热力图以促进声明的图文可解释性，（vi）5W问答对，以及（vii）对抗性假新闻故事。

英文摘要：

Combating disinformation is one of the burning societal crises -- about 67% of the American population believes that disinformation produces a lot of uncertainty, and 10% of them knowingly propagate disinformation. Evidence shows that disinformation can manipulate democratic processes and public opinion, causing disruption in the share market, panic and anxiety in society, and even death during crises. Therefore, disinformation should be identified promptly and, if possible, mitigated. With approximately 3.2 billion images and 720,000 hours of video shared online daily on social media platforms, scalable detection of multimodal disinformation requires efficient fact verification. Despite progress in automatic text-based fact verification (e.g., FEVER, LIAR), the research community lacks substantial effort in multimodal fact verification. To address this gap, we introduce FACTIFY 3M, a dataset of 3 million samples that pushes the boundaries of the domain of fact verification via a multimodal fake news dataset, in addition to offering explainability through the concept of 5W question-answering. Salient features of the dataset include: (i) textual claims, (ii) ChatGPT-generated paraphrased claims, (iii) associated images, (iv) stable diffusion-generated additional images (i.e., visual paraphrases), (v) pixel-level image heatmap to foster image-text explainability of the claim, (vi) 5W QA pairs, and (vii) adversarial fake news stories.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。