CIEC: 通过隐式和显式线索耦合实现多模态弱监督操作定位
CIEC: Coupling Implicit and Explicit Cues for Multimodal Weakly Supervised Manipulation Localization
AI总结：
CIEC通过耦合隐式和显式线索，实现多模态弱监督操作定位，利用粗粒度标注提升图像-文本对的定位效果。
摘要：
为缓解虚假信息带来的威胁，多模态操作定位受到了越来越多的关注。考虑到当前方法依赖于成本高且耗时的细粒度标注，如补丁/词级标注。本文提出了一种名为耦合隐式和显式线索（CIEC）的新框架，旨在利用仅有的粗粒度图像/句子级标注，实现图像-文本对的多模态弱监督操作定位。它包含两个分支，即图像基于和文本基于的弱监督定位。对于前者，我们设计了文本引导的补丁选择（TRPS）模块。它整合了来自视觉和文本视角的伪造线索，借助空间先验锁定可疑区域。随后通过背景沉默和空间对比约束来抑制无关区域的干扰。对于后者，我们设计了视觉偏差校准的词格模块（VCTG）。它专注于有意义的内容词，并利用相对视觉偏差来辅助词定位。随后通过不对称稀疏和语义一致性约束来缓解标签噪声并确保可靠性。广泛的实验表明了我们CIEC的有效性，在多个评估指标上取得了与全监督方法相当的结果。
英文摘要：
To mitigate the threat of misinformation, multimodal manipulation localization has garnered growing attention. Consider that current methods rely on costly and time-consuming fine-grained annotations, such as patch/token-level annotations. This paper proposes a novel framework named Coupling Implicit and Explicit Cues (CIEC), which aims to achieve multimodal weakly-supervised manipulation localization for image-text pairs utilizing only coarse-grained image/sentence-level annotations. It comprises two branches, image-based and text-based weakly-supervised localization. For the former, we devise the Textual-guidance Refine Patch Selection (TRPS) module. It integrates forgery cues from both visual and textual perspectives to lock onto suspicious regions aided by spatial priors. Followed by the background silencing and spatial contrast constraints to suppress interference from irrelevant areas. For the latter, we devise the Visual-deviation Calibrated Token Grounding (VCTG) module. It focuses on meaningful content words and leverages relative visual bias to assist token localization. Followed by the asymmetric sparse and semantic consistency constraints to mitigate label noise and ensure reliability. Extensive experiments demonstrate the effectiveness of our CIEC, yielding results comparable to fully supervised methods on several evaluation metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
