CLIPping the Deception：适配视觉-语言模型以实现通用深度伪造检测
CLIPping the Deception: Adapting Vision-Language Models for Universal Deepfake Detection
AI总结：
本文提出利用保留文本组件的CLIP并采用轻量级提示微调策略进行通用深度伪造检测，仅用ProGAN数据集和不到三分之一训练数据即超越先前SOTA方法，并在21个数据集上验证了其泛化能力。
摘要：
生成对抗网络（GANs）的近期进展以及扩散模型（Diffusion models）的出现，显著简化了高度逼真且易于获取的合成内容的生成过程。因此，迫切需要有效的通用检测机制来降低深度伪造（deepfakes）可能带来的风险。本文中，我们探索了预训练视觉-语言模型（VLMs）与近期适配方法相结合在通用深度伪造检测中的有效性。遵循该领域先前的研究，我们仅使用单一数据集（ProGAN）来适配CLIP以进行深度伪造检测。然而，与先前仅依赖CLIP视觉部分而忽略其文本组件的研究不同，我们的分析表明保留文本部分至关重要。因此，我们所采用的简单且轻量级的基于提示微调（Prompt Tuning）的适配策略，在仅使用不到三分之一训练数据（20万张图像，对比72万张）的情况下，比先前的SOTA方法在mAP上高出5.01%，在准确率上高出6.61%。为评估所提模型在实际应用中的可行性，我们在多种场景下进行了全面评估，包括对来自21个不同数据集的图像进行严格测试，这些数据集涵盖基于GAN、基于扩散模型以及商业工具生成的图像。
英文摘要：
The recent advancements in Generative Adversarial Networks (GANs) and the emergence of Diffusion models have significantly streamlined the production of highly realistic and widely accessible synthetic content. As a result, there is a pressing need for effective general purpose detection mechanisms to mitigate the potential risks posed by deepfakes. In this paper, we explore the effectiveness of pre-trained vision-language models (VLMs) when paired with recent adaptation methods for universal deepfake detection. Following previous studies in this domain, we employ only a single dataset (ProGAN) in order to adapt CLIP for deepfake detection. However, in contrast to prior research, which rely solely on the visual part of CLIP while ignoring its textual component, our analysis reveals that retaining the text part is crucial. Consequently, the simple and lightweight Prompt Tuning based adaptation strategy that we employ outperforms the previous SOTA approach by 5.01% mAP and 6.61% accuracy while utilizing less than one third of the training data (200k images as compared to 720k). To assess the real-world applicability of our proposed models, we conduct a comprehensive evaluation across various scenarios. This involves rigorous testing on images sourced from 21 distinct datasets, including those generated by GANs-based, Diffusion-based and Commercial tools.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
