Better generative models and larger datasets have led to more realistic fake videos that can fool the human eye but produce temporal and spatial artifacts that deep learning approaches can detect. Most current Deepfake detection methods only use individual video frames and therefore fail to learn from temporal information. We created a benchmark of the performance of spatiotemporal convolutional methods using the Celeb-DF dataset. Our methods outperformed state-of-the-art frame-based detection methods. Code for our paper is publicly available at https://github.com/oidelima/Deepfake-Detection.
Recent advances in deepfake generating algorithms that produce manipulated media have had dangerous implications in privacy, security and mass communication. Efforts to combat this issue have risen in the form of competitions and funding for research to detect deepfakes. This paper presents three techniques and algorithms: convolutional LSTM, eye blink detection and grayscale histograms-pursued while participating in the Deepfake Detection Challenge. We assessed the current knowledge about deepfake videos, a more severe version of manipulated media, and previous methods used, and found relevan…
Learning Spatiotemporal Inconsistency via Thumbnail Layout for Face Deepfake Detection
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of Science and Technology of China（中国科学技术大学）
CYENS Centre of Excellence（CYENS卓越中心）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
Deepfakes are synthetically generated images, videos or audios, which fraudsters use to manipulate legitimate information. Current deepfake detection systems struggle against unseen data. To address this, we employ three different deep Convolutional Neural Network (CNN) models, (1) VGG16, (2) InceptionV3, and (3) XceptionNet to classify fake and real images extracted from videos. We also constructed a fusion of the deep CNN models to improve the robustness and generalisation capability. The proposed technique outperforms state-of-the-art models with 96.5% accuracy, when tested on publicly avai…
In recent years, deep learning-based video manipulation methods have become widely accessible to masses. With little to no effort, people can easily learn how to generate deepfake videos with only a few victims or target images. This creates a significant social problem for everyone whose photos are publicly available on the Internet, especially on social media websites. Several deep learning-based detection methods have been developed to identify these deepfakes. However, these methods lack generalizability, because they perform well only for a specific type of deepfake method. Therefore, tho…
The University of Hong Kong（香港大学）；Shandong University（山东大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Recently, Deepfake has drawn considerable public attention due to security and privacy concerns in social media digital forensics. As the wildly spreading Deepfake videos on the Internet become more realistic, traditional detection techniques have failed in distinguishing between real and fake. Most existing deep learning methods mainly focus on local features and relations within the face image using convolutional neural networks as a backbone. However, local features and relations are insufficient for model training to learn enough general information for Deepfake detection. Therefore, the e…
The rapid advancement of deep learning models that can generate and synthesis hyper-realistic videos known as Deepfakes and their ease of access to the general public have raised concern from all concerned bodies to their possible malicious intent use. Deep learning techniques can now generate faces, swap faces between two subjects in a video, alter facial expressions, change gender, and alter facial features, to list a few. These powerful video manipulation methods have potential use in many fields. However, they also pose a looming threat to everyone if used for harmful purposes such as iden…
The Deepfake phenomenon has become very popular nowadays thanks to the possibility to create incredibly realistic images using deep learning tools, based mainly on ad-hoc Generative Adversarial Networks (GAN). In this work we focus on the analysis of Deepfakes of human faces with the objective of creating a new detection method able to detect a forensics trace hidden in images: a sort of fingerprint left in the image generation process. The proposed technique, by means of an Expectation Maximization (EM) algorithm, extracts a set of local features specifically addressed to model the underlying…
Multimedia Signal Processing Group (MMSPG)（多媒体信号处理组）
Deep convolutional neural networks have shown remarkable results on multiple detection tasks. Despite the significant progress, the performance of such detectors are often assessed in public benchmarks under non-realistic conditions. Specifically, impact of conventional distortions and processing operations such as compression, noise, and enhancement are not sufficiently studied. This paper proposes a rigorous framework to assess performance of learning-based detectors in more realistic situations. An illustrative example is shown under deepfake detection context. Inspired by the assessment re…
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
Deep convolutional neural networks have achieved exceptional results on multiple detection and recognition tasks. However, the performance of such detectors are often evaluated in public benchmarks under constrained and non-realistic situations. The impact of conventional distortions and processing operations found in imaging workflows such as compression, noise, and enhancement are not sufficiently studied. Currently, only a few researches have been done to improve the detector robustness to unseen perturbations. This paper proposes a more effective data augmentation scheme based on real-worl…
This work uses adversarial perturbations to enhance deepfake images and fool common deepfake detectors. We created adversarial perturbations using the Fast Gradient Sign Method and the Carlini and Wagner L2 norm attack in both blackbox and whitebox settings. Detectors achieved over 95% accuracy on unperturbed deepfakes, but less than 27% accuracy on perturbed deepfakes. We also explore two improvements to deepfake detectors: (i) Lipschitz regularization, and (ii) Deep Image Prior (DIP). Lipschitz regularization constrains the gradient of the detector with respect to the input in order to incre…
Deepfakes are the result of digital manipulation to forge realistic yet fake imagery. With the astonishing advances in deep generative models, fake images or videos are nowadays obtained using variational autoencoders (VAEs) or Generative Adversarial Networks (GANs). These technologies are becoming more accessible and accurate, resulting in fake videos that are very difficult to be detected. Traditionally, Convolutional Neural Networks (CNNs) have been used to perform video deepfake detection, with the best results obtained using methods based on EfficientNet B7. In this study, we focus on vid…
This work introduces a novel DeepFake detection framework based on physiological measurement. In particular, we consider information related to the heart rate using remote photoplethysmography (rPPG). rPPG methods analyze video sequences looking for subtle color changes in the human skin, revealing the presence of human blood under the tissues. In this work we investigate to what extent rPPG is useful for the detection of DeepFake videos. The proposed fake detector named DeepFakesON-Phys uses a Convolutional Attention Network (CAN), which extracts spatial and temporal information from video fr…
Indian Institute of Technology Ropar（印度理工学院罗巴尔分校）；Monash University（莫纳什大学）
This paper proposes a new DeepFake detector FakeBuster for detecting impostors during video conferencing and manipulated faces on social media. FakeBuster is a standalone deep learning based solution, which enables a user to detect if another person's video is manipulated or spoofed during a video conferencing based meeting. This tool is independent of video conferencing solutions and has been tested with Zoom and Skype applications. It uses a 3D convolutional neural network for predicting video segment-wise fakeness scores. The network is trained on a combination of datasets such as Deeperfor…
Video and Image Processing Laboratory (VIPER)（视频与图像处理实验室（VIPER））；School of Electrical Engineering（电气工程学院）；Purdue University（普渡大学）
Altered and manipulated multimedia is increasingly present and widely distributed via social media platforms. Advanced video manipulation tools enable the generation of highly realistic-looking altered multimedia. While many methods have been presented to detect manipulations, most of them fail when evaluated with data outside of the datasets used in research environments. In order to address this problem, the Deepfake Detection Challenge (DFDC) provides a large dataset of videos containing realistic manipulations and an evaluation system that ensures that methods work quickly and accurately, …
Exposing and Mitigating Temporal Attack in Deepfake Video Detection
Peking University（北京大学）；New York University（纽约大学）；Huzhou University（湖州大学）；Institute of Science Tokyo（东京科学研究院）；Guangdong Institute of Intelligence Science and Technology（广东省智能科学与技术研究院）
Deep learning has enabled realistic face manipulation (i.e., deepfake), which poses significant concerns over the integrity of the media in circulation. Most existing deep learning techniques for deepfake detection can achieve promising performance in the intra-dataset evaluation setting (i.e., training and testing on the same dataset), but are unable to perform satisfactorily in the inter-dataset evaluation setting (i.e., training on one dataset and testing on another). Most of the previous methods use the backbone network to extract global features for making predictions and only employ bina…
DeepShield: Fortifying Deepfake Video Detection with Local and Global Forgery Analysis
Sun Yat-sen University（中山大学）；Pengcheng Laboratory（鹏城实验室）；Guilin University of Electronic Technology（桂林电子科技大学）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东大数据分析与处理重点实验室）
Deepfake is the manipulated video made with a generative deep learning technique such as Generative Adversarial Networks (GANs) or Auto Encoder that anyone can utilize. Recently, with the increase of Deepfake videos, some classifiers consisting of the convolutional neural network that can distinguish fake videos as well as deepfake datasets have been actively created. However, the previous studies based on the CNN structure have the problem of not only overfitting, but also considerable misjudging fake video as real ones. In this paper, we propose a Vision Transformer model with distillation m…
Institute of Information Engineering, Chinese Academy of Science（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；School of Communication and Information Engineering, Shanghai University（上海大学通信与信息工程学院）
While the abuse of deepfake technology has caused serious concerns recently, how to detect deepfake videos is still a challenge due to the high photo-realistic synthesis of each frame. Existing image-level approaches often focus on single frame and ignore the spatiotemporal cues hidden in deepfake videos, resulting in poor generalization and robustness. The key of a video-level detector is to fully exploit the spatiotemporal inconsistency distributed in local facial regions across different frames in deepfake videos. Inspired by that, this paper proposes a simple yet effective patch-level appr…
School of Computing（计算机学院）；Wichita State University（威奇托州立大学）
Facial manipulation by deep fake has caused major security risks and raised severe societal concerns. As a countermeasure, a number of deep fake detection methods have been proposed recently. Most of them model deep fake detection as a binary classification problem using a backbone convolutional neural network (CNN) architecture pretrained for the task. These CNN-based methods have demonstrated very high efficacy in deep fake detection with the Area under the Curve (AUC) as high as 0.99. However, the performance of these methods degrades significantly when evaluated across datasets. In this pa…
Asia University（亚洲大学）；China Medical University Hospital, China Medical University（中国医药大学附设医院，中国医药大学）；Universitas Muhammadiyah Yogyakarta（穆罕默迪亚日惹大学）；Western Norway University of Applied Sciences（挪威西部应用科学大学）
The spread of PM2.5 pollutants that endanger health is difficult to predict because it involves many atmospheric variables. These micron particles can spread rapidly from their source to residential areas, increasing the risk of respiratory disease if exposed for long periods. The prediction system of PM2.5 propagation provides more detailed and accurate information as an early warning system to reduce health impacts on the community. According to the idea of transformative computing, the approach we propose in this paper allows computation on the dataset obtained from massive-scale PM2.5 sens…
Frequency-Aware Deepfake Detection: Improving Generalizability through Frequency Space Learning
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；School of Information Science and Engineering, Yanshan University（燕山大学信息科学与工程学院）；Center for Frontier AI Research, IHPC, A*STAR, Singapore（新加坡科技研究局前沿人工智能研究中心）
Infrared (IR) imaging has the potential to enable more robust action recognition systems compared to visible spectrum cameras due to lower sensitivity to lighting conditions and appearance variability. While the action recognition task on videos collected from visible spectrum imaging has received much attention, action recognition in IR videos is significantly less explored. Our objective is to exploit imaging data in this modality for the action recognition task. In this work, we propose a novel two-stream 3D convolutional neural network (CNN) architecture by introducing the discriminative c…
GIFGuard: Proactive Forensics against Deepfakes in Facial GIFs via Spatiotemporal Watermarking
School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）；College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机科学与电子工程学院）
In this paper, we introduce MINTIME, a video deepfake detection approach that captures spatial and temporal anomalies and handles instances of multiple people in the same video and variations in face sizes. Previous approaches disregard such information either by using simple a-posteriori aggregation schemes, i.e., average or max operation, or using only one identity for the inference, i.e., the largest one. On the contrary, the proposed approach builds on a Spatio-Temporal TimeSformer combined with a Convolutional Neural Network backbone to capture spatio-temporal anomalies from the face sequ…
Deepfakes are the synthesized digital media in order to create ultra-realistic fake videos to trick the spectator. Deep generative algorithms, such as, Generative Adversarial Networks(GAN) are widely used to accomplish such tasks. This approach synthesizes pseudo-realistic contents that are very difficult to distinguish by traditional detection methods. In most cases, Convolutional Neural Network(CNN) based discriminators are being used for detecting such synthesized media. However, it emphasise primarily on the spatial attributes of individual video frames, thereby fail to learn the temporal …
ARMOR++: Agentic Orchestration of a Multi-Domain Primitive Set for Transferable Attacks on Deepfake Detectors
Department of Informatics, Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学信息学系）；Infocomm Technology Cluster, Singapore Institute of Technology（新加坡科技学院信息通信技术集群）；Department of Applied Physics and Applied Mathematics, Columbia University（哥伦比亚大学应用物理与应用数学系）；Division of Natural and Applied Sciences, Duke Kunshan University（昆山杜克大学自然科学与应用科学部）；Department of Electrical and Computer Engineering, University of Toronto（多伦多大学电气与计算机工程系）
AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Deepfake Detection of Frontal Face Videos
Social Networks and Human-Centered Computing Program, Taiwan International Graduate Program, Academia Sinica（学术.sinica.edu.tw 社会网络与人本计算计划，台湾国际研究生计划，台湾.sinica.edu.tw）；Department of Computer Science, National Chengchi University（计算机科学系，国立政治大学）；Institute of Information Systems and Applications, National Tsing Hua University（信息系统与应用研究所，国立清华大学）；Research Center for Information Technology Innovation, Academia Sinica（资讯技术创新研究中心，学术.sinica.edu.tw）；Institute of Information Science, Academia Sinica（资讯研究所，学术.sinica.edu.tw）
