Factify 2022:多模态事实验证
Logically at Factify 2022: Multimodal Fact Verification
摘要:
本文描述了我们在AAAI 2022上的多模态事实验证(Factify)挑战的参与者系统。尽管最近在基于文本的验证技术和跨视觉和语言的大型预训练多模态模型方面取得了进展,但在应用多模态技术自动化事实检查过程方面所做的工作非常有限,特别是考虑到社交媒体上关于图像和视频的索赔和假新闻越来越普遍。在我们的工作中,挑战被视为多模态蕴涵任务和框架的多类分类。提出并探索了两种基线方法,包括集成模型(结合两个单模态模型)和多模态注意力网络(模拟索赔和证据文件中图像和文本对之间的相互作用)。在这项工作中,我们进行了几个实验,调查和基准测试不同的SoTA预训练的Transformers和视觉模型。我们最好的模型在排行榜中排名第一,在验证集和测试集上获得了0.77的加权平均F-测量值。还对Factify数据集进行了数据集的探索性分析,并发现了突出的模式和问题(例如,词重叠,视觉蕴涵相关性,源偏差),激发我们的假设。最后,我们强调了任务和多模态数据集的挑战,为未来的研究。
英文摘要:
This paper describes our participant system for the multi-modal fact verification (Factify) challenge at AAAI 2022. Despite the recent advance in text based verification techniques and large pre-trained multimodal models cross vision and language, very limited work has been done in applying multimodal techniques to automate fact checking process, particularly considering the increasing prevalence of claims and fake news about images and videos on social media. In our work, the challenge is treated as multimodal entailment task and framed as multi-class classification. Two baseline approaches are proposed and explored including an ensemble model (combining two uni-modal models) and a multi-modal attention network (modeling the interaction between image and text pair from claim and evidence document). We conduct several experiments investigating and benchmarking different SoTA pre-trained transformers and vision models in this work. Our best model is ranked first in leaderboard which obtains a weighted average F-measure of 0.77 on both validation and test set. Exploratory analysis of dataset is also carried out on the Factify data set and uncovers salient patterns and issues (e.g., word overlapping, visual entailment correlation, source bias) that motivates our hypothesis. Finally, we highlight challenges of the task and multimodal dataset for future research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。