相似性优于事实性：我们在多模态脱离语境错误信息检测方面是否取得了进展？
Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?
AI总结：
本研究提出基于图像-文本与外部证据相似性的简单基线MUSE，在OOC检测中媲美甚至超越复杂方法，并揭示依赖表面捷径的进展评估问题。
摘要：
脱离语境（OOC）错误信息在多模态事实核查中构成了重大挑战，其中图像与歪曲其原始语境以支持虚假叙事的文本配对。近期基于证据的OOC检测研究呈现出日益复杂架构的趋势，融合了Transformer、基础模型和大语言模型。在本研究中，我们引入了一个简单而稳健的基线，即评估多模态相似性（MUSE），具体评估图像-文本对与外部图像和文本证据之间的相似性。我们的结果表明，当MUSE与决策树、随机森林和多层感知机等传统分类器结合使用时，能够在NewsCLIPpings和VERITE数据集上与最先进方法竞争，甚至超越它们。此外，将MUSE整合到我们提出的“注意力中间Transformer表示”（AITR）中，显著提升了性能，在NewsCLIPpings和VERITE上分别提高了3.3%和7.5%。然而，MUSE的成功依赖于表面模式和捷径，而未检查事实性和逻辑不一致性，这引发了关于我们如何定义任务、构建数据集、收集外部证据以及总体上如何评估该领域进展的关键问题。我们在https://github.com/stevejpapad/outcontext-misinfo-progress发布了代码。
英文摘要：
Out-of-context (OOC) misinformation poses a significant challenge in multimodal fact-checking, where images are paired with texts that misrepresent their original context to support false narratives. Recent research in evidence-based OOC detection has seen a trend towards increasingly complex architectures, incorporating Transformers, foundation models, and large language models. In this study, we introduce a simple yet robust baseline, which assesses MUltimodal SimilaritiEs (MUSE), specifically the similarity between image-text pairs and external image and text evidence. Our results demonstrate that MUSE, when used with conventional classifiers like Decision Tree, Random Forest, and Multilayer Perceptron, can compete with and even surpass the state-of-the-art on the NewsCLIPpings and VERITE datasets. Furthermore, integrating MUSE in our proposed "Attentive Intermediate Transformer Representations" (AITR) significantly improved performance, by 3.3% and 7.5% on NewsCLIPpings and VERITE, respectively. Nevertheless, the success of MUSE, relying on surface-level patterns and shortcuts, without examining factuality and logical inconsistencies, raises critical questions about how we define the task, construct datasets, collect external evidence and overall, how we assess progress in the field. We release our code at: https://github.com/stevejpapad/outcontext-misinfo-progress
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
