The fast and continuous growth in number and quality of deepfake videos calls for the development of reliable detection systems capable of automatically warning users on social media and on the Internet about the potential untruthfulness of such contents. While algorithms, software, and smartphone apps are getting better every day in generating manipulated videos and swapping faces, the accuracy of automated systems for face forgery detection in videos is still quite limited and generally biased toward the dataset used to design and train a specific detection system. In this paper we analyze how different training strategies and data augmentation techniques affect CNN-based deepfake detectors when training and testing on the same dataset or across different datasets.
Shahjalal University of Science and Technology（沙贾拉尔科技大学）；Mapbox；University of Alberta（阿尔伯塔大学）；Fordham University（福特汉姆大学）
The creation of altered and manipulated faces has become more common due to the improvement of DeepFake generation methods. Simultaneously, we have seen detection models' development for differentiating between a manipulated and original face from image or video content. In this paper, we focus on identifying the limitations and shortcomings of existing deepfake detection frameworks. We identified some key problems surrounding deepfake detection through quantitative and qualitative analysis of existing methods and datasets. We found that deepfake datasets are highly oversampled, causing models…
University of Wisconsin-Eau Claire（威斯康星大学欧克莱尔分校）；Minnesota State University, Mankato（明尼苏达州立大学曼卡托分校）
Social media is currently being used by many individuals online as a major source of information. However, not all information shared online is true, even photos and videos can be doctored. Deepfakes have recently risen with the rise of technological advancement and have allowed nefarious online users to replace one face with a computer generated face of anyone they would like, including important political and cultural figures. Deepfakes are now a tool to be able to spread mass misinformation. There is now an immense need to create models that are able to detect deepfakes and keep them from b…
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
Deep convolutional neural networks have achieved exceptional results on multiple detection and recognition tasks. However, the performance of such detectors are often evaluated in public benchmarks under constrained and non-realistic situations. The impact of conventional distortions and processing operations found in imaging workflows such as compression, noise, and enhancement are not sufficiently studied. Currently, only a few researches have been done to improve the detector robustness to unseen perturbations. This paper proposes a more effective data augmentation scheme based on real-worl…
Deepfake media is becoming widespread nowadays because of the easily available tools and mobile apps which can generate realistic looking deepfake videos/images without requiring any technical knowledge. With further advances in this field of technology in the near future, the quantity and quality of deepfake media is also expected to flourish, while making deepfake media a likely new practical tool to spread mis/disinformation. Because of these concerns, the deepfake media detection tools are becoming a necessity. In this study, we propose a novel hybrid transformer network utilizing early fe…
DeiTFake: Deepfake Detection Model using DeiT Multi-Stage Training
School of Computing, Amrita Vishwa Vidyapeetham, Amaravati Campus（阿姆里塔维瓦达佩特学院）；School of Computer Engineering, Kalinga Institute of Industrial Technology (KIIT) Deemed to be University（卡林加工业技术学院（KIIT）被认定为大学）；School of Engineering, Amrita Vishwa Vidyapeetham, Amaravati Campus（阿姆里塔维瓦达佩特工程学院）；Department of Computer Science and Engineering, Manipal Institute of Technology Bengaluru, Manipal Academy of Higher Education（计算机科学与工程系，班加罗尔Manipal技术学院，Manipal高等教育学院）；Amity Institute of Information Technology, Amity University Jharkhand（阿米蒂信息技术学院，阿米蒂贾坎德大学）
Universitas Mercatorum（梅尔卡托鲁姆大学）；National Inter-University Consortium for Telecommunications (CNIT)（全国校际电信联盟（CNIT））
Deepfake Generation Techniques are evolving at a rapid pace, making it possible to create realistic manipulated images and videos and endangering the serenity of modern society. The continual emergence of new and varied techniques brings with it a further problem to be faced, namely the ability of deepfake detection models to update themselves promptly in order to be able to identify manipulations carried out using even the most recent methods. This is an extremely complex problem to solve, as training a model requires large amounts of data, which are difficult to obtain if the deepfake genera…
Deepfake is the manipulated video made with a generative deep learning technique such as Generative Adversarial Networks (GANs) or Auto Encoder that anyone can utilize. Recently, with the increase of Deepfake videos, some classifiers consisting of the convolutional neural network that can distinguish fake videos as well as deepfake datasets have been actively created. However, the previous studies based on the CNN structure have the problem of not only overfitting, but also considerable misjudging fake video as real ones. In this paper, we propose a Vision Transformer model with distillation m…
Deepfakes are the synthesized digital media in order to create ultra-realistic fake videos to trick the spectator. Deep generative algorithms, such as, Generative Adversarial Networks(GAN) are widely used to accomplish such tasks. This approach synthesizes pseudo-realistic contents that are very difficult to distinguish by traditional detection methods. In most cases, Convolutional Neural Network(CNN) based discriminators are being used for detecting such synthesized media. However, it emphasise primarily on the spatial attributes of individual video frames, thereby fail to learn the temporal …
CYENS Centre of Excellence（CYENS卓越中心）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
Deepfakes are synthetically generated images, videos or audios, which fraudsters use to manipulate legitimate information. Current deepfake detection systems struggle against unseen data. To address this, we employ three different deep Convolutional Neural Network (CNN) models, (1) VGG16, (2) InceptionV3, and (3) XceptionNet to classify fake and real images extracted from videos. We also constructed a fusion of the deep CNN models to improve the robustness and generalisation capability. The proposed technique outperforms state-of-the-art models with 96.5% accuracy, when tested on publicly avai…
The rapid advancement of deep learning models that can generate and synthesis hyper-realistic videos known as Deepfakes and their ease of access to the general public have raised concern from all concerned bodies to their possible malicious intent use. Deep learning techniques can now generate faces, swap faces between two subjects in a video, alter facial expressions, change gender, and alter facial features, to list a few. These powerful video manipulation methods have potential use in many fields. However, they also pose a looming threat to everyone if used for harmful purposes such as iden…
School of Computer Science Carnegie Mellon University Pittsburgh, United States
The proliferation of deepfake technology poses significant challenges to the authenticity and trustworthiness of digital media, necessitating the development of robust detection methods. This study explores the application of Swin Transformers, a state-of-the-art architecture leveraging shifted windows for self-attention, in detecting and classifying deepfake images. Using the Real and Fake Face Detection dataset by Yonsei University's Computational Intelligence Photography Lab, we evaluate the Swin Transformer and hybrid models such as Swin-ResNet and Swin-KNN, focusing on their ability to id…
Multimedia Signal Processing Group (MMSPG)（多媒体信号处理组）
Deep convolutional neural networks have shown remarkable results on multiple detection tasks. Despite the significant progress, the performance of such detectors are often assessed in public benchmarks under non-realistic conditions. Specifically, impact of conventional distortions and processing operations such as compression, noise, and enhancement are not sufficiently studied. This paper proposes a rigorous framework to assess performance of learning-based detectors in more realistic situations. An illustrative example is shown under deepfake detection context. Inspired by the assessment re…
Transcending Forgery Specificity with Latent Space Augmentation for Generalizable Deepfake Detection
The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（香港中文大学（深圳））；University at Buffalo, State University of New York（纽约州立大学布法罗分校）；Nanjing University of Information Science and Technology（南京信息工程大学）
Towards Generalizable Deepfake Image Detection with Vision Transformers
Department of Electronics and Communication Engineering（电子与通信工程系）；Department of Information Technology（信息技术系）；Department of Electrical and Electronics Engineering（电气与电子工程系）；National Institute of Technology Karnataka (NITK), Surathkal - 575025, India（卡纳塔克国家理工学院（NITK），苏拉塔卡尔-575025，印度）
This paper presents our results and findings on the use of temporal images for deepfake detection. We modelled temporal relations that exist in the movement of 468 facial landmarks across frames of a given video as spatial relations by constructing an image (referred to as temporal image) using the pixel values at these facial landmarks. CNNs are capable of recognizing spatial relationships that exist between the pixels of a given image. 10 different ImageNet models were considered for the study.
School of Computing（计算机学院）；Wichita State University（威奇托州立大学）
Facial manipulation by deep fake has caused major security risks and raised severe societal concerns. As a countermeasure, a number of deep fake detection methods have been proposed recently. Most of them model deep fake detection as a binary classification problem using a backbone convolutional neural network (CNN) architecture pretrained for the task. These CNN-based methods have demonstrated very high efficacy in deep fake detection with the Area under the Curve (AUC) as high as 0.99. However, the performance of these methods degrades significantly when evaluated across datasets. In this pa…
Norwegian University of Science and Technology（挪威科技大学）
The accelerated growth in synthetic visual media generation and manipulation has now reached the point of raising significant concerns and posing enormous intimidations towards society. There is an imperative need for automatic detection networks towards false digital content and avoid the spread of dangerous artificial information to contend with this threat. In this paper, we utilize and compare two kinds of handcrafted features(SIFT and HoG) and two kinds of deep features(Xception and CNN+RNN) for the deepfake detection task. We also check the performance of these features when there are mi…
Recent advances in deep learning have enabled realistic digital alterations to videos, known as deepfakes. This technology raises important societal concerns regarding disinformation and authenticity, galvanizing the development of numerous deepfake detection algorithms. At the same time, there are significant differences between training data and in-the-wild video data, which may undermine their practical efficacy. We simulate data corruption techniques and examine the performance of a state-of-the-art deepfake detection algorithm on corrupted variants of the FaceForensics++ dataset. While de…
Amrita School of Computing Amrita Vishwa Vidyapeetham Amaravati(Andhra Pradesh), India；Engineering Thapar Institute of Engineering \& Technology Patiala(Punjab), India
For deepfake detection, video-level detectors have not been explored as extensively as image-level detectors, which do not exploit temporal data. In this paper, we empirically show that existing approaches on image and sequence classifiers generalize poorly to new manipulation techniques. To this end, we propose spatio-temporal features, modeled by 3D CNNs, to extend the generalization capabilities to detect new sorts of deepfake videos. We show that spatial features learn distinct deepfake-method-specific attributes, while spatio-temporal features capture shared attributes between deepfake me…
Video and Image Processing Laboratory (VIPER)（视频与图像处理实验室（VIPER））；School of Electrical Engineering（电气工程学院）；Purdue University（普渡大学）
Altered and manipulated multimedia is increasingly present and widely distributed via social media platforms. Advanced video manipulation tools enable the generation of highly realistic-looking altered multimedia. While many methods have been presented to detect manipulations, most of them fail when evaluated with data outside of the datasets used in research environments. In order to address this problem, the Deepfake Detection Challenge (DFDC) provides a large dataset of videos containing realistic manipulations and an evaluation system that ensures that methods work quickly and accurately, …
DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detection
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University at Buffalo, State University of New York（纽约州立大学布法罗分校）；Nanjing University of Posts and Telecommunications（南京邮电大学）
University of Southern California（南加州大学）；Army Research Laboratory（陆军研究实验室）
On the basis of DefakeHop, an enhanced lightweight Deepfake detector called DefakeHop++ is proposed in this work. The improvements lie in two areas. First, DefakeHop examines three facial regions (i.e., two eyes and mouth) while DefakeHop++ includes eight more landmarks for broader coverage. Second, for discriminant features selection, DefakeHop uses an unsupervised approach while DefakeHop++ adopts a more effective approach with supervision, called the Discriminant Feature Test (DFT). In DefakeHop++, rich spatial and spectral features are first derived from facial regions and landmarks automa…
DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection
The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（香港中文大学（深圳））；Tencent AI Lab（腾讯AI实验室）；University at Buffalo, State University of New York（纽约州立大学布法罗分校）
How Generalizable are Deepfake Image Detectors? An Empirical Study
School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）；College of Computer Science and Technology, Harbin Engineering University（哈尔滨工程大学计算机科学与技术学院）
University of Science and Technology of China（中国科学技术大学）；Microsoft Cloud AI（微软云人工智能）
Face forgery by deepfake is widely spread over the internet and has raised severe societal concerns. Recently, how to detect such forgery contents has become a hot research topic and many deepfake detection methods have been proposed. Most of them model deepfake detection as a vanilla binary classification problem, i.e, first use a backbone network to extract a global feature and then feed it into a binary classifier (real/fake). But since the difference between the real and fake images in this task is often subtle and local, we argue this vanilla solution is not optimal. In this paper, we ins…
The University of Adelaide（阿德莱德大学）；Tencent AI Lab（腾讯人工智能实验室）
Recent studies in deepfake detection have yielded promising results when the training and testing face forgeries are from the same dataset. However, the problem remains challenging when one tries to generalize the detector to forgeries created by unseen methods in the training dataset. This work addresses the generalizable deepfake detection from a simple principle: a generalizable representation should be sensitive to diverse types of forgeries. Following this principle, we propose to enrich the "diversity" of forgeries by synthesizing augmented forgeries with a pool of forgery configurations…
A Multi-Domain Feature Fusion Framework for Generalizable Deepfake Detection Across Different Generators
School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan（巴基斯坦国立科技大学电气工程与计算机科学学院）；Department of Communication, Quality Management and Information Systems, Mid Sweden University, Ostersund Campus, Sweden（瑞典中部大学通信、质量管理和信息系统系，厄斯特松德校区）；Department of Computer Science, Electrical and Space Engineering, Lulea University of Technology, Luleå, Sweden（瑞典吕勒奥理工大学计算机科学、电气与空间工程系）
