检测与定位多模态媒体篡改
Detecting and Grounding Multi-Modal Media Manipulation
AI总结：
本文针对多模态虚假媒体篡改提出DGM^4问题，构建首个相关数据集，并设计分层多模态篡改推理Transformer（HAMMER）模型，通过浅层对比学习与深层交叉注意力实现篡改检测与内容定位，在建立的基准上展现出优越性能。
摘要：
虚假信息已成为一个紧迫问题。视觉和文本形式的虚假媒体在网络上广泛传播。尽管已有各种深度伪造检测和文本假新闻检测方法被提出，但它们仅针对基于二分类的单模态伪造进行设计，更不用说分析和推理跨不同模态的细微伪造痕迹了。在本文中，我们针对多模态虚假媒体提出了一个新的研究问题，即检测与定位多模态媒体篡改（DGM^4）。DGM^4旨在不仅检测多模态媒体的真实性，还要定位被篡改的内容（即图像边界框和文本token），这需要对多模态媒体篡改进行更深入的推理。为了支持大规模研究，我们构建了首个DGM^4数据集，其中的图文对通过各种方法进行篡改，并提供了丰富的多样化篡改标注。此外，我们提出了一种新颖的分层多模态篡改推理Transformer（HAMMER），以充分捕获不同模态间的细粒度交互。HAMMER执行1)在两个单模态编码器之间进行篡改感知对比学习，作为浅层篡改推理；2)由多模态聚合器进行模态感知交叉注意力，作为深层篡改推理。基于交互的多模态信息，从浅层到深层集成了专门的篡改检测和定位头。最后，我们为这一新研究问题构建了广泛的基准并设立了严格的评估指标。综合实验证明了我们模型的优越性；同时还揭示了一些有价值的观察结果，以促进未来在多模态媒体篡改方面的研究。
英文摘要：
Misinformation has become a pressing issue. Fake media, in both visual and textual forms, is widespread on the web. While various deepfake detection and text fake news detection methods have been proposed, they are only designed for single-modality forgery based on binary classification, let alone analyzing and reasoning subtle forgery traces across different modalities. In this paper, we highlight a new research problem for multi-modal fake media, namely Detecting and Grounding Multi-Modal Media Manipulation (DGM^4). DGM^4 aims to not only detect the authenticity of multi-modal media, but also ground the manipulated content (i.e., image bounding boxes and text tokens), which requires deeper reasoning of multi-modal media manipulation. To support a large-scale investigation, we construct the first DGM^4 dataset, where image-text pairs are manipulated by various approaches, with rich annotation of diverse manipulations. Moreover, we propose a novel HierArchical Multi-modal Manipulation rEasoning tRansformer (HAMMER) to fully capture the fine-grained interaction between different modalities. HAMMER performs 1) manipulation-aware contrastive learning between two uni-modal encoders as shallow manipulation reasoning, and 2) modality-aware cross-attention by multi-modal aggregator as deep manipulation reasoning. Dedicated manipulation detection and grounding heads are integrated from shallow to deep levels based on the interacted multi-modal information. Finally, we build an extensive benchmark and set up rigorous evaluation metrics for this new research problem. Comprehensive experiments demonstrate the superiority of our model; several valuable observations are also revealed to facilitate future research in multi-modal media manipulation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
