检测并定位多模态媒体篡改及其他
Detecting and Grounding Multi-Modal Media Manipulation and Beyond
AI总结：
本文提出多模态媒体篡改检测与定位问题DGM^4，构建首个图文篡改数据集，并通过层级多模态推理模型HAMMER及HAMMER++实现检测与篡改定位。
摘要：
虚假信息已成为一个紧迫问题。视觉和文本形式的虚假媒体在网络上广泛传播。尽管研究者已经提出了各种深度伪造检测和文本假新闻检测方法，但它们仅针对基于二分类的单模态伪造而设计，更不用说分析和推理不同模态之间细微的伪造痕迹。本文中，我们强调一个面向多模态虚假媒体的新研究问题，即检测并定位多模态媒体篡改（Detecting and Grounding Multi-Modal Media Manipulation，DGM^4）。DGM^4不仅旨在检测多模态媒体的真实性，还旨在定位被篡改的内容，这需要对多模态媒体篡改进行更深入的推理。为支持大规模研究，我们构建了首个DGM^4数据集，其中图文对通过多种方法被篡改，并带有针对多样化篡改的丰富标注。此外，我们提出了一种新颖的层级多模态篡改推理Transformer（HierArchical Multi-modal Manipulation rEasoning tRansformer，HAMMER），以充分捕捉不同模态之间的细粒度交互。HAMMER执行：1）在两个单模态编码器之间进行篡改感知对比学习，作为浅层篡改推理；2）通过多模态聚合器进行模态感知交叉注意力，作为深层篡改推理。基于交互后的多模态信息，从浅层到深层集成了专门的篡改检测和定位头。为利用更细粒度的对比学习实现跨模态语义对齐，我们进一步集成了带局部视图的篡改感知对比损失，并构建了更先进的模型HAMMER++。最后，我们为这一新研究问题建立了广泛的基准并设置了严格的评估指标。综合实验证明了HAMMER和HAMMER++的优越性。
英文摘要：
Misinformation has become a pressing issue. Fake media, in both visual and textual forms, is widespread on the web. While various deepfake detection and text fake news detection methods have been proposed, they are only designed for single-modality forgery based on binary classification, let alone analyzing and reasoning subtle forgery traces across different modalities. In this paper, we highlight a new research problem for multi-modal fake media, namely Detecting and Grounding Multi-Modal Media Manipulation (DGM^4). DGM^4 aims to not only detect the authenticity of multi-modal media, but also ground the manipulated content, which requires deeper reasoning of multi-modal media manipulation. To support a large-scale investigation, we construct the first DGM^4 dataset, where image-text pairs are manipulated by various approaches, with rich annotation of diverse manipulations. Moreover, we propose a novel HierArchical Multi-modal Manipulation rEasoning tRansformer (HAMMER) to fully capture the fine-grained interaction between different modalities. HAMMER performs 1) manipulation-aware contrastive learning between two uni-modal encoders as shallow manipulation reasoning, and 2) modality-aware cross-attention by multi-modal aggregator as deep manipulation reasoning. Dedicated manipulation detection and grounding heads are integrated from shallow to deep levels based on the interacted multi-modal information. To exploit more fine-grained contrastive learning for cross-modal semantic alignment, we further integrate Manipulation-Aware Contrastive Loss with Local View and construct a more advanced model HAMMER++. Finally, we build an extensive benchmark and set up rigorous evaluation metrics for this new research problem. Comprehensive experiments demonstrate the superiority of HAMMER and HAMMER++.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
