arXiv 2412.127182025-10-22cs.CVcs.MM

ASAP：推进语义对齐促进多模态篡改检测与定位

ASAP: Advancing Semantic Alignment Promotes Multi-Modal Manipulation Detecting and Grounding

School of Computer Science and Information Engineering, Hefei University of Technology, China（计算机科学与信息工程学院，合肥工业大学，中国）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China（人工智能研究所，合肥综合性国家科学中心，中国）

Zhenxing Zhang, Yaxiong Wang, Lechao Cheng, Zhun Zhong, Dan Guo, Meng Wang

备注：: 12 pages, 6 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有多模态媒体篡改检测与定位方法忽视跨模态语义对齐的问题，本研究提出ASAP框架，结合MLLMs/LLMs构建图文对增强对齐，并设计MGCA模块强化篡改感知，在DGM4数据集上性能显著优于对比方法。

摘要：

我们提出了ASAP，一种用于多模态媒体篡改检测与定位（DGM4）的新型框架。经深入研究我们发现，图像与文本之间精准的细粒度跨模态语义对齐，对准确实现篡改检测与定位至关重要。而现有DGM4方法极少关注跨模态对齐问题，这制约了篡改检测精度的进一步提升。为解决该问题，本研究旨在推进语义对齐学习以助力该任务。具体而言，我们利用现成的多模态大语言模型（Multimodal Large-Language Models, MLLMs）和大语言模型（Large Language Models, LLMs）构建成对的图文对，尤其针对篡改样本。随后开展跨模态对齐学习以增强语义对齐效果。除了显式辅助线索，我们还设计了篡改引导交叉注意力（Manipulation-Guided Cross Attention, MGCA），为增强篡改感知提供隐式引导。借助训练时可用的定位真值，MGCA促使模型更关注被篡改的组件同时弱化正常组件，提升模型捕捉篡改的能力。我们在DGM4数据集上开展了大量实验，结果表明我们的模型能以明显优势超越对比方法。

英文摘要：

We present ASAP, a new framework for detecting and grounding multi-modal media manipulation (DGM4).Upon thorough examination, we observe that accurate fine-grained cross-modal semantic alignment between the image and text is vital for accurately manipulation detection and grounding. While existing DGM4 methods pay rare attention to the cross-modal alignment, hampering the accuracy of manipulation detecting to step further. To remedy this issue, this work targets to advance the semantic alignment learning to promote this task. Particularly, we utilize the off-the-shelf Multimodal Large-Language Models (MLLMs) and Large Language Models (LLMs) to construct paired image-text pairs, especially for the manipulated instances. Subsequently, a cross-modal alignment learning is performed to enhance the semantic alignment. Besides the explicit auxiliary clues, we further design a Manipulation-Guided Cross Attention (MGCA) to provide implicit guidance for augmenting the manipulation perceiving. With the grounding truth available during training, MGCA encourages the model to concentrate more on manipulated components while downplaying normal ones, enhancing the model's ability to capture manipulations. Extensive experiments are conducted on the DGM4 dataset, the results demonstrate that our model can surpass the comparison method with a clear margin.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。