检测对口型深度伪造：用于分析嘴部不一致性的视觉时间Transformer
Detecting Lip-Syncing Deepfakes: Vision Temporal Transformer for Analyzing Mouth Inconsistencies
AI总结：
本文提出LIPINC-V2检测框架，结合视觉时间Transformer与多头交叉注意力，通过识别嘴部区域时空不一致性来检测对口型深度伪造，并构建新数据集LipSyncTIMIT验证其达到SOTA性能。
摘要：
深度伪造是AI生成的媒体，其中原始内容被数字化篡改以创建令人信服但被操纵的图像、视频或音频。在各种类型的深度伪造中，对口型深度伪造是最具挑战性的检测对象之一。在这些视频中，使用AI模型合成人的唇部动作以匹配被篡改或全新的音频。因此，与其他类型的深度伪造不同，对口型深度伪造中的伪影仅限于嘴部区域，使其更加细微，因而更难辨别。在本文中，我们提出了LIPINC-V2，这是一种新颖的检测框架，它利用视觉时间Transformer与多头交叉注意力的组合，通过识别嘴部区域的时空不一致性来检测对口型深度伪造。这些不一致性出现在相邻帧之间，并持续贯穿整个视频。我们的模型能够成功捕获嘴部运动的短期和长期变化，从而增强了其检测这些不一致性的能力。此外，我们创建了一个新的对口型深度伪造数据集LipSyncTIMIT，该数据集是使用五个最先进的对口型模型生成的，以模拟真实世界场景。在我们提出的LipSyncTIMIT数据集和另外两个基准深度伪造数据集上进行的大量实验表明，我们的模型实现了最先进的性能。代码和数据集可在 https://github.com/skrantidatta/LIPINC-V2 获取。
英文摘要：
Deepfakes are AI-generated media in which the original content is digitally altered to create convincing but manipulated images, videos, or audio. Among the various types of deepfakes, lip-syncing deepfakes are one of the most challenging deepfakes to detect. In these videos, a person's lip movements are synthesized to match altered or entirely new audio using AI models. Therefore, unlike other types of deepfakes, the artifacts in lip-syncing deepfakes are confined to the mouth region, making them more subtle and, thus harder to discern. In this paper, we propose LIPINC-V2, a novel detection framework that leverages a combination of vision temporal transformer with multihead cross-attention to detect lip-syncing deepfakes by identifying spatiotemporal inconsistencies in the mouth region. These inconsistencies appear across adjacent frames and persist throughout the video. Our model can successfully capture both short-term and long-term variations in mouth movement, enhancing its ability to detect these inconsistencies. Additionally, we created a new lip-syncing deepfake dataset, LipSyncTIMIT, which was generated using five state-of-the-art lip-syncing models to simulate real-world scenarios. Extensive experiments on our proposed LipSyncTIMIT dataset and two other benchmark deepfake datasets demonstrate that our model achieves state-of-the-art performance. The code and the dataset are available at https://github.com/skrantidatta/LIPINC-V2 .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
