Deepfake is the manipulated video made with a generative deep learning technique such as Generative Adversarial Networks (GANs) or Auto Encoder that anyone can utilize. Recently, with the increase of Deepfake videos, some classifiers consisting of the convolutional neural network that can distinguish fake videos as well as deepfake datasets have been actively created. However, the previous studies based on the CNN structure have the problem of not only overfitting, but also considerable misjudging fake video as real ones. In this paper, we propose a Vision Transformer model with distillation methodology for detecting fake videos. We design that a CNN features and patch-based positioning model learns to interact with all positions to find the artifact region for solving false negative problem. Through comparative analysis on Deepfake Detection (DFDC) Dataset, we verify that the proposed scheme with patch embedding as input outperforms the state-of-the-art using the combined CNN features. Without ensemble technique, our model obtains 0.978 of AUC and 91.9 of f1 score, while previous SOTA model yields 0.972 of AUC and 90.6 of f1 score on the same condition.
The University of Hong Kong（香港大学）；Shandong University（山东大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Recently, Deepfake has drawn considerable public attention due to security and privacy concerns in social media digital forensics. As the wildly spreading Deepfake videos on the Internet become more realistic, traditional detection techniques have failed in distinguishing between real and fake. Most existing deep learning methods mainly focus on local features and relations within the face image using convolutional neural networks as a backbone. However, local features and relations are insufficient for model training to learn enough general information for Deepfake detection. Therefore, the e…
CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation
Peking University（北京大学）；Shenzhen Graduate School（深圳研究生院）；University of Science and Technology of China（中国科学技术大学）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
School of Computer Science Carnegie Mellon University Pittsburgh, United States
The proliferation of deepfake technology poses significant challenges to the authenticity and trustworthiness of digital media, necessitating the development of robust detection methods. This study explores the application of Swin Transformers, a state-of-the-art architecture leveraging shifted windows for self-attention, in detecting and classifying deepfake images. Using the Real and Fake Face Detection dataset by Yonsei University's Computational Intelligence Photography Lab, we evaluate the Swin Transformer and hybrid models such as Swin-ResNet and Swin-KNN, focusing on their ability to id…
The widespread dissemination of Deepfakes demands effective approaches that can detect perceptually convincing forged images. In this paper, we aim to capture the subtle manipulation artifacts at different scales using transformer models. In particular, we introduce a Multi-modal Multi-scale TRansformer (M2TR), which operates on patches of different sizes to detect local inconsistencies in images at different spatial levels. M2TR further learns to detect forgery artifacts in the frequency domain to complement RGB information through a carefully designed cross modality fusion block. In addition…
Face forgery by deepfake is widely spread over the internet and this raises severe societal concerns. In this paper, we propose a novel video transformer with incremental learning for detecting deepfake videos. To better align the input face images, we use a 3D face reconstruction method to generate UV texture from a single input face image. The aligned face image can also provide pose, eyes blink and mouth movement information that cannot be perceived in the UV texture image, so we use both face images and their UV texture maps to extract the image features. We present an incremental learning…
Deepfake media is becoming widespread nowadays because of the easily available tools and mobile apps which can generate realistic looking deepfake videos/images without requiring any technical knowledge. With further advances in this field of technology in the near future, the quantity and quality of deepfake media is also expected to flourish, while making deepfake media a likely new practical tool to spread mis/disinformation. Because of these concerns, the deepfake media detection tools are becoming a necessity. In this study, we propose a novel hybrid transformer network utilizing early fe…
Universitas Mercatorum（梅尔卡托鲁姆大学）；National Inter-University Consortium for Telecommunications (CNIT)（全国校际电信联盟（CNIT））
Deepfake Generation Techniques are evolving at a rapid pace, making it possible to create realistic manipulated images and videos and endangering the serenity of modern society. The continual emergence of new and varied techniques brings with it a further problem to be faced, namely the ability of deepfake detection models to update themselves promptly in order to be able to identify manipulations carried out using even the most recent methods. This is an extremely complex problem to solve, as training a model requires large amounts of data, which are difficult to obtain if the deepfake genera…
The rapid advancement of deep learning models that can generate and synthesis hyper-realistic videos known as Deepfakes and their ease of access to the general public have raised concern from all concerned bodies to their possible malicious intent use. Deep learning techniques can now generate faces, swap faces between two subjects in a video, alter facial expressions, change gender, and alter facial features, to list a few. These powerful video manipulation methods have potential use in many fields. However, they also pose a looming threat to everyone if used for harmful purposes such as iden…
Deepfakes are the result of digital manipulation to forge realistic yet fake imagery. With the astonishing advances in deep generative models, fake images or videos are nowadays obtained using variational autoencoders (VAEs) or Generative Adversarial Networks (GANs). These technologies are becoming more accessible and accurate, resulting in fake videos that are very difficult to be detected. Traditionally, Convolutional Neural Networks (CNNs) have been used to perform video deepfake detection, with the best results obtained using methods based on EfficientNet B7. In this study, we focus on vid…
Explicit Correlation Learning for Generalizable Cross-Modal Deepfake Detection
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；University at Buffalo, State University of New York（纽约州立大学布法罗分校）；Institute of Microelectronics, Chinese Academy of Sciences（中国科学院微电子研究所）
University of Science Technology of China（中国科学技术大学）；Microsoft Cloud AI（微软云人工智能）
The fast evolution and widespread of deepfake techniques in real-world scenarios require stronger generalization abilities of face forgery detectors. Some works capture the features that are unrelated to method-specific artifacts, such as clues of blending boundary, accumulated up-sampling, to strengthen the generalization ability. However, the effectiveness of these methods can be easily corrupted by post-processing operations such as compression. Inspired by transfer learning, neural networks pre-trained on other large-scale face-related tasks may provide useful features for deepfake detecti…
Towards Generalizable Deepfake Image Detection with Vision Transformers
Department of Electronics and Communication Engineering（电子与通信工程系）；Department of Information Technology（信息技术系）；Department of Electrical and Electronics Engineering（电气与电子工程系）；National Institute of Technology Karnataka (NITK), Surathkal - 575025, India（卡纳塔克国家理工学院（NITK），苏拉塔卡尔-575025，印度）
Recent advances in face forgery techniques produce nearly visually untraceable deepfake videos, which could be leveraged with malicious intentions. As a result, researchers have been devoted to deepfake detection. Previous studies have identified the importance of local low-level cues and temporal information in pursuit to generalize well across deepfake methods, however, they still suffer from robustness problem against post-processings. In this work, we propose the Local- & Temporal-aware Transformer-based Deepfake Detection (LTTD) framework, which adopts a local-to-global learning protocol …
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；S-Lab, Nanyang Technological University（南洋理工大学S-Lab）
University of Southern California（南加州大学）；Army Research Laboratory（陆军研究实验室）
A light-weight high-performance Deepfake detection method, called DefakeHop, is proposed in this work. State-of-the-art Deepfake detection methods are built upon deep neural networks. DefakeHop extracts features automatically using the successive subspace learning (SSL) principle from various parts of face images. The features are extracted by c/w Saab transform and further processed by our feature distillation module using spatial dimension reduction and soft classification for each channel to get a more concise description of the face. Extensive experiments are conducted to demonstrate the e…
AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection
Social Networks and Human-Centered Computing Program, Taiwan International Graduate Program, Academia Sinica（社会网络与以人为本计算计划、台湾国际研究生计划、中国科学院）；Institute of Information Systems and Applications, National Tsing Hua University（信息系统与应用研究所、国立清华大学）；Department of Computer Science, National Chengchi University（计算机科学系、国立成功大学）；Research Center for Information Technology Innovation, Academia Sinica（信息技术创新研究中心、中国科学院）；Institute of Information Science, Academia Sinica（信息科学研究所、中国科学院）
Amrita School of Computing Amrita Vishwa Vidyapeetham Amaravati(Andhra Pradesh), India；Engineering Thapar Institute of Engineering \& Technology Patiala(Punjab), India
DeepFake-Adapter: Dual-Level Adapter for DeepFake Detection
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；S-Lab, Nanyang Technological University（南洋理工大学S-Lab）
DeiTFake: Deepfake Detection Model using DeiT Multi-Stage Training
School of Computing, Amrita Vishwa Vidyapeetham, Amaravati Campus（阿姆里塔维瓦达佩特学院）；School of Computer Engineering, Kalinga Institute of Industrial Technology (KIIT) Deemed to be University（卡林加工业技术学院（KIIT）被认定为大学）；School of Engineering, Amrita Vishwa Vidyapeetham, Amaravati Campus（阿姆里塔维瓦达佩特工程学院）；Department of Computer Science and Engineering, Manipal Institute of Technology Bengaluru, Manipal Academy of Higher Education（计算机科学与工程系，班加罗尔Manipal技术学院，Manipal高等教育学院）；Amity Institute of Information Technology, Amity University Jharkhand（阿米蒂信息技术学院，阿米蒂贾坎德大学）
Despite significant advancements of deep learning-based forgery detectors for distinguishing manipulated deepfake images, most detection approaches suffer from moderate to significant performance degradation with low-quality compressed deepfake images. Because of the limited information in low-quality images, detecting low-quality deepfake remains an important challenge. In this work, we apply frequency domain learning and optimal transport theory in knowledge distillation (KD) to specifically improve the detection of low-quality compressed deepfake images. We explore transfer learning capabil…
Deepfake Forensics Adapter: A Dual-Stream Network for Generalizable Deepfake Detection
Shenzhen Technology University（深圳技术大学）；Singapore Institute of Technology（新加坡理工学院）；The University of Hong Kong（香港大学）；Nanyang Technological University（南洋理工大学）
AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Deepfake Detection of Frontal Face Videos
Social Networks and Human-Centered Computing Program, Taiwan International Graduate Program, Academia Sinica（学术.sinica.edu.tw 社会网络与人本计算计划，台湾国际研究生计划，台湾.sinica.edu.tw）；Department of Computer Science, National Chengchi University（计算机科学系，国立政治大学）；Institute of Information Systems and Applications, National Tsing Hua University（信息系统与应用研究所，国立清华大学）；Research Center for Information Technology Innovation, Academia Sinica（资讯技术创新研究中心，学术.sinica.edu.tw）；Institute of Information Science, Academia Sinica（资讯研究所，学术.sinica.edu.tw）
As GAN-based video and image manipulation technologies become more sophisticated and easily accessible, there is an urgent need for effective deepfake detection technologies. Moreover, various deepfake generation techniques have emerged over the past few years. While many deepfake detection methods have been proposed, their performance suffers from new types of deepfake methods on which they are not sufficiently trained. To detect new types of deepfakes, the model should learn from additional data without losing its prior knowledge about deepfakes (catastrophic forgetting), especially when new…
University of Macau（澳门大学）；Macau University of Science and Technology（澳门科技大学）
With the continuous research on Deepfake forensics, recent studies have attempted to provide the fine-grained localization of forgeries, in addition to the coarse classification at the video-level. However, the detection and localization performance of existing Deepfake forensic methods still have plenty of room for further improvement. In this work, we propose a Spatial-Temporal Deepfake Detection and Localization (ST-DDL) network that simultaneously explores spatial and temporal features for detecting and localizing forged regions. Specifically, we design a new Anchor-Mesh Motion (AMM) algor…
Zero-Shot to Zero-Lies: Detecting Bengali Deepfake Audio through Transfer Learning
Department of CSE, BRAC University, Bangladesh（布鲁斯大学计算机科学与工程系）；Department of CSE, Ahsanullah University of Science and Technology, Bangladesh（阿沙努拉科技大学计算机科学与工程系）；Enosis Solutions, Bangladesh（恩奥斯解决方案公司）；Department of CSE, Southeast University, Bangladesh（东南大学计算机科学与工程系）
Institute of Information Engineering, Chinese Academy of Science（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；School of Communication and Information Engineering, Shanghai University（上海大学通信与信息工程学院）
While the abuse of deepfake technology has caused serious concerns recently, how to detect deepfake videos is still a challenge due to the high photo-realistic synthesis of each frame. Existing image-level approaches often focus on single frame and ignore the spatiotemporal cues hidden in deepfake videos, resulting in poor generalization and robustness. The key of a video-level detector is to fully exploit the spatiotemporal inconsistency distributed in local facial regions across different frames in deepfake videos. Inspired by that, this paper proposes a simple yet effective patch-level appr…
ARMOR++: Agentic Orchestration of a Multi-Domain Primitive Set for Transferable Attacks on Deepfake Detectors
Department of Informatics, Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学信息学系）；Infocomm Technology Cluster, Singapore Institute of Technology（新加坡科技学院信息通信技术集群）；Department of Applied Physics and Applied Mathematics, Columbia University（哥伦比亚大学应用物理与应用数学系）；Division of Natural and Applied Sciences, Duke Kunshan University（昆山杜克大学自然科学与应用科学部）；Department of Electrical and Computer Engineering, University of Toronto（多伦多大学电气与计算机工程系）
Criminal Investigation Police University of China（中国刑事警察学院）；Chinese University of Hong Kong（香港中文大学）；Dalian University of Technology（大连理工大学）；Northeastern University（东北大学）；Baidu（百度）；Harbin Institute of Technology（哈尔滨工业大学）
