重新思考视觉语言模型用于图像伪造检测与定位
Rethinking VLMs for Image Forgery Detection and Localization
AI总结：
本文研究如何利用视觉语言模型提升图像伪造检测与定位性能，发现传统先验知识对检测效果有负面影响，提出IFDL-VLM新方法，通过位置掩码增强模型可解释性，并在多个基准上取得新最优结果。
摘要：
随着人工智能生成内容（AIGC）的迅速崛起，图像篡改已成为越来越容易被实施的技术，这对图像伪造检测与定位（IFDL）提出了重大挑战。本文研究如何充分利用视觉语言模型（VLMs）来辅助IFDL任务。特别地，我们发现VLMs的先验知识对检测和定位性能几乎没有帮助，甚至由于其内在对语义合理性而非真实性的偏见而产生负面影响。此外，位置掩码明确编码了伪造概念，可以作为额外的先验知识为VLMs提供，从而减轻其训练优化，进而提升检测和定位结果的可解释性。基于这些发现，我们提出了一种新的IFDL流程，称为IFDL-VLM。为了展示我们方法的有效性，我们在9个流行的基准上进行了实验，并在域内和跨数据集泛化设置下评估模型性能。实验结果表明，我们一致在检测、定位和可解释性方面取得了新的最佳性能。代码可在：https://github.com/sha0fengGuo/IFDL-VLM 上获得。
英文摘要：
With the rapid rise of Artificial Intelligence Generated Content (AIGC), image manipulation has become increasingly accessible, posing significant challenges for image forgery detection and localization (IFDL). In this paper, we study how to fully leverage vision-language models (VLMs) to assist the IFDL task. In particular, we observe that priors from VLMs hardly benefit the detection and localization performance and even have negative effects due to their inherent biases toward semantic plausibility rather than authenticity. Additionally, the location masks explicitly encode the forgery concepts, which can serve as extra priors for VLMs to ease their training optimization, thus enhancing the interpretability of detection and localization results. Building on these findings, we propose a new IFDL pipeline named IFDL-VLM. To demonstrate the effectiveness of our method, we conduct experiments on 9 popular benchmarks and assess the model performance under both in-domain and cross-dataset generalization settings. The experimental results show that we consistently achieve new state-of-the-art performance in detection, localization, and interpretability.Code is available at: https://github.com/sha0fengGuo/IFDL-VLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
