利用模态内与跨模态不一致性的无监督多模态Deepfake检测
Unsupervised Multimodal Deepfake Detection Using Intra- and Cross-Modal Inconsistencies
AI总结：
针对现有有监督Deepfake检测依赖标注数据、难泛化到新生成方法的问题，提出基于模态内与跨模态不一致性的无监督检测方法，在FakeAVCeleb数据集上性能超越此前SOTA，兼具可扩展性、泛化性、可靠性与可解释性。
摘要：
Deepfake视频对社会构成的威胁日益加剧，可能对刑事司法、民主以及人身安全和隐私造成负面影响。与此同时，大规模检测Deepfake仍是一项极具挑战性的任务，通常需要来自现有Deepfake生成方法的标注训练数据。此外，即便准确率最高的有监督Deepfake检测方法，也无法泛化到采用新生成方法制作的Deepfake上。本文提出了一种新颖的无监督Deepfake视频检测方法，该方法通过直接识别视频片段之间的模态内与跨模态不一致性来实现检测。该检测方法背后的核心假设是，运动或身份不一致性在Deepfake视频中是不可避免的。我们将从数学层面和实证层面为这一假设提供支撑，随后基于理论分析构建我们的方法。在极具挑战性的FakeAVCeleb数据集上，我们提出的方法性能优于此前最先进的无监督Deepfake检测方法，同时还具备多项额外优势：它具备可扩展性，因为推理阶段无需为每个身份提供原始（真实）样本，因此可应用于任意数量的身份；它具备泛化性，因为仅在真实视频上训练，因此不依赖于特定的Deepfake生成方法；它具备可靠性，因为不依赖任何高维空间中的似然估计；它具备可解释性，因为能够精准定位模态不一致性的确切位置，随后可供人类专家验证。
英文摘要：
Deepfake videos present an increasing threat to society with potentially negative impact on criminal justice, democracy, and personal safety and privacy. Meanwhile, detecting deepfakes, at scale, remains a very challenging task that often requires labeled training data from existing deepfake generation methods. Further, even the most accurate supervised deepfake detection methods do not generalize to deepfakes generated using new generation methods. In this paper, we propose a novel unsupervised method for detecting deepfake videos by directly identifying intra-modal and cross-modal inconsistency between video segments. The fundamental hypothesis behind the proposed detection method is that motion or identity inconsistencies are inevitable in deepfake videos. We will mathematically and empirically support this hypothesis, and then proceed to constructing our method grounded in our theoretical analysis. Our proposed method outperforms prior state-of-the-art unsupervised deepfake detection methods on the challenging FakeAVCeleb dataset, and also has several additional advantages: it is scalable because it does not require pristine (real) samples for each identity during inference and therefore can apply to arbitrarily many identities, generalizable because it is trained only on real videos and therefore does not rely on a particular deepfake method, reliable because it does not rely on any likelihood estimation in high dimensions, and explainable because it can pinpoint the exact location of modality inconsistencies which are then verifiable by a human expert.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
