Learning Natural Consistency Representation for Face Forgery Video Detection
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tsinghua University（清华大学）；RealAI
Face Forgery videos have elicited critical social public concerns and various detectors have been proposed. However, fully-supervised detectors may lead to easily overfitting to specific forgery methods or videos, and existing self-supervised detectors are strict on auxiliary tasks, such as requiring audio or multi-modalities, leading to limited generalization and robustness. In this paper, we examine whether we can address this issue by leveraging visual-only real face videos. To this end, we propose to learn the Natural Consistency representation (NACO) of real face videos in a self-supervised manner, which is inspired by the observation that fake videos struggle to maintain the natural spatiotemporal consistency even under unknown forgery methods and different perturbations. Our NACO first extracts spatial features of each frame by CNNs then integrates them into Transformer to learn the long-range spatiotemporal representation, leveraging the advantages of CNNs and Transformer on local spatial receptive field and long-term memory respectively. Furthermore, a Spatial Predictive Module~(SPM) and a Temporal Contrastive Module~(TCM) are introduced to enhance the natural consistency representation learning. The SPM aims to predict random masked spatial features from spatiotemporal representation, and the TCM regularizes the latent distance of spatiotemporal representation by shuffling the natural order to disturb the consistency, which could both force our NACO more sensitive to the natural spatiotemporal consistency. After the representation learning stage, a MLP head is fine-tuned to perform the usual forgery video classification task. Extensive experiments show that our method outperforms other state-of-the-art competitors with impressive generalization and robustness.
Beijing University of Posts and Telecommunications（北京邮电大学）
With the emergence of GAN, face forgery technologies have been heavily abused. Achieving accurate face forgery detection is imminent. Inspired by remote photoplethysmography (rPPG) that PPG signal corresponds to the periodic change of skin color caused by heartbeat in face videos, we observe that despite the inevitable loss of PPG signal during the forgery process, there is still a mixture of PPG signals in the forgery video with a unique rhythmic pattern depending on its generation method. Motivated by this key observation, we propose a framework for face forgery detection and categorization …
University of Science and Technology of China（中国科学技术大学）
Existing face forgery detection methods usually treat face forgery detection as a binary classification problem and adopt deep convolution neural networks to learn discriminative features. The ideal discriminative features should be only related to the real/fake labels of facial images. However, we observe that the features learned by vanilla classification networks are correlated to unnecessary properties, such as forgery methods and facial identities. Such phenomenon would limit forgery detection performance especially for the generalization ability. Motivated by this, we propose a novel met…
Peking University（北京大学）；Microsoft Research Asia（微软亚洲研究院）
In this paper we propose a novel image representation called face X-ray for detecting forgery in face images. The face X-ray of an input face image is a greyscale image that reveals whether the input image can be decomposed into the blending of two images from different sources. It does so by showing the blending boundary for a forged image and the absence of blending for a real image. We observe that most existing face manipulation methods share a common step: blending the altered face into an existing background image. For this reason, face X-ray provides an effective way for detecting forge…
The rapid progress of photorealistic synthesis techniques has reached a critical point where the boundary between real and manipulated images starts to blur. Recently, a mega-scale deep face forgery dataset, ForgeryNet which comprised of 2.9 million images and 221,247 videos has been released. It is by far the largest publicly available in terms of data-scale, manipulations (7 image-level approaches, 8 video-level approaches), perturbations (36 independent and more mixed perturbations), and annotations (6.3 million classification labels, 2.9 million manipulated area annotations, and 221,247 te…
Learning to Discover Forgery Cues for Face Forgery Detection
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；RealAI Inc（RealAI公司）
SenseTime Research（商汤科技研究院）；University of Electronic Science and Technology of China（电子科技大学）；College of Software, Beihang University（北京航空航天大学软件学院）；Northwestern Polytechnical University（西北工业大学）
As realistic facial manipulation technologies have achieved remarkable progress, social concerns about potential malicious abuse of these technologies bring out an emerging research topic of face forgery detection. However, it is extremely challenging since recent advances are able to forge faces beyond the perception ability of human eyes, especially in compressed images and videos. We find that mining forgery patterns with the awareness of frequency could be a cure, as frequency provides a complementary viewpoint where either subtle forgery artifacts or compression errors could be well descr…
Nowadays, forgery faces pose pressing security concerns over fake news, fraud, impersonation, etc. Despite the demonstrated success in intra-domain face forgery detection, existing detection methods lack generalization capability and tend to suffer from dramatic performance drops when deployed to unforeseen domains. To mitigate this issue, this paper designs a more general fake face detection model based on the vision transformer(ViT) architecture. In the training phase, the pretrained ViT weights are freezed, and only the Low-Rank Adaptation(LoRA) modules are updated. Additionally, the Single…
Beijing University of Posts and Telecommunications（北京邮电大学）
Although vanilla Convolutional Neural Network (CNN) based detectors can achieve satisfactory performance on fake face detection, we observe that the detectors tend to seek forgeries on a limited region of face, which reveals that the detectors is short of understanding of forgery. Therefore, we propose an attention-based data augmentation framework to guide detector refine and enlarge its attention. Specifically, our method tracks and occludes the Top-N sensitive facial regions, encouraging the detector to mine deeper into the regions ignored before for more representative forgery. Especially,…
COMICS: End-to-end Bi-grained Contrastive Learning for Multi-face Forgery Detection
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Pengcheng Laboratory（鹏城实验室）；College of Computer Science and Technology, Ocean University of China（中国海洋大学计算机科学与技术学院）；University at Buffalo, SUNY（纽约州立大学布法罗分校）
University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）
The remarkable success in face forgery techniques has received considerable attention in computer vision due to security concerns. We observe that up-sampling is a necessary step of most face forgery techniques, and cumulative up-sampling will result in obvious changes in the frequency domain, especially in the phase spectrum. According to the property of natural images, the phase spectrum preserves abundant frequency components that provide extra information and complement the loss of the amplitude spectrum. To this end, we present a novel Spatial-Phase Shallow Learning (SPSL) method, which c…
FedForgery: Generalized Face Forgery Detection with Residual Federated Learning
Xidian University（西安电子科技大学）；Shanghai Key Laboratory of Computer Software Evaluating and Testing（上海市计算机软件评测重点实验室）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
With the rapid development of facial forgery techniques, forgery detection has attracted more and more attention due to security concerns. Existing approaches attempt to use frequency information to mine subtle artifacts under high-quality forged faces. However, the exploitation of frequency information is coarse-grained, and more importantly, their vanilla learning process struggles to extract fine-grained forgery traces. To address this issue, we propose a progressive enhancement learning framework to exploit both the RGB and fine-grained frequency clues. Specifically, we perform a fine-grai…
Various facial manipulation techniques have drawn serious public concerns in morality, security, and privacy. Although existing face forgery classifiers achieve promising performance on detecting fake images, these methods are vulnerable to adversarial examples with injected imperceptible perturbations on the pixels. Meanwhile, many face forgery detectors always utilize the frequency diversity between real and fake faces as a crucial clue. In this paper, instead of injecting adversarial perturbations into the spatial domain, we propose a frequency adversarial attack method against face forgery…
With various facial manipulation techniques arising, face forgery detection has drawn growing attention due to security concerns. Previous works always formulate face forgery detection as a classification problem based on cross-entropy loss, which emphasizes category-level differences rather than the essential discrepancies between real and fake faces, limiting model generalization in unseen domains. To address this issue, we propose a novel face forgery detection framework, named Dual Contrastive Learning (DCL), which specially constructs positive and negative paired data and performs designe…
MFVLR: Multi-domain Fine-grained Vision-Language Reconstruction for Generalizable Diffusion Face Forgery Detection and Localization
Faculty of Computer Science and Technology, Qilu University of Technology (Shandong Academy of Sciences)（计算机科学与技术学院，齐鲁工业大学（山东省科学院））；School of Computing, National University of Singapore（国立新加坡大学计算机学院）；Shandong Artificial Intelligence Institute, Qilu University of Technology (Shandong Academy of Sciences)（山东省人工智能研究院，齐鲁工业大学（山东省科学院））；Key Laboratory of Computer Vision and System, Ministry of Education, Tianjin University of Technology（教育部计算机视觉与系统重点实验室，天津工业大学）
Contrastive Desensitization Learning for Cross Domain Face Forgery Detection
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics & MIIT Key Laboratory of Pattern Analysis and Machine Intelligence（计算机科学与技术学院，南京航空航天大学及信息通信技术部模式分析与机器智能重点实验室）
With the rapid development of facial manipulation techniques, face forgery detection has received considerable attention in digital media forensics due to security concerns. Most existing methods formulate face forgery detection as a classification problem and utilize binary labels or manipulated region masks as supervision. However, without considering the correlation between local regions, these global supervisions are insufficient to learn a generalized feature and prone to overfitting. To address this issue, we propose a novel perspective of face forgery detection via local relation learni…
InfoDense: Density-Aware Regional Decisive Replay for Memory-Efficient Incremental Face Forgery Detection
Huazhong Agricultural University（华中农业大学）；Peking University（北京大学）；National University of Singapore（新加坡国立大学）；Nantong University（南通大学）；Wuhan University（武汉大学）
One of the most pressing challenges for the detection of face-manipulated videos is generalising to forgery methods not seen during training while remaining effective under common corruptions such as compression. In this paper, we examine whether we can tackle this issue by harnessing videos of real talking faces, which contain rich information on natural facial appearance and behaviour and are readily available in large quantities online. Our method, termed RealForensics, consists of two stages. First, we exploit the natural correspondence between the visual and auditory modalities in real vi…
Nanjing University of Information Science and Technology（南京信息工程大学）；Jinan University（暨南大学）；Nanjing University（南京大学）
In this work, we propose a novel method to improve the generalization ability of CNN-based face forgery detectors. Our method considers the feature anomalies of forged faces caused by the prevalent blending operations in face forgery algorithms. Specifically, we propose a weakly supervised Second Order Local Anomaly (SOLA) learning module to mine anomalies in local regions using deep feature maps. SOLA first decomposes the neighborhood of local features by different directions and distances and then calculates the first and second order local anomaly maps which provide more general forgery tra…
Currently, many face forgery detection methods aggregate spatial and frequency features to enhance the generalization ability and gain promising performance under the cross-dataset scenario. However, these methods only leverage one level frequency information which limits their expressive ability. To overcome these limitations, we propose a multi-scale wavelet transformer framework for face forgery detection. Specifically, to take full advantage of the multi-scale and multi-frequency wavelet representation, we gradually aggregate the multi-scale wavelet representation at different stages of th…
MFCLIP: Multi-modal Fine-grained CLIP for Generalizable Diffusion Face Forgery Detection
Faculty of Computer Science and Technology, Qilu University of Technology (Shandong Academy of Sciences)（计算机科学与技术学院，齐鲁工业大学（山东科学院））；School of Computing, National University of Singapore（computing 学院，新加坡国立大学）；School of Computing and Information Technology, Great Bay University（computing 与信息学院，大湾大学）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（大数据系统计算技术国家工程实验室，深圳大学）
Xiamen University（厦门大学）；Microsoft Research Asia（微软亚洲研究院）
Although current face manipulation techniques achieve impressive performance regarding quality and controllability, they are struggling to generate temporal coherent face videos. In this work, we explore to take full advantage of the temporal coherence for video face forgery detection. To achieve this, we propose a novel end-to-end framework, which consists of two major stages. The first stage is a fully temporal convolution network (FTCN). The key insight of FTCN is to reduce the spatial convolution kernel size to 1, while maintaining the temporal convolution kernel size unchanged. We surpris…
Exploring Specular Reflection Inconsistency for Generalizable Face Forgery Detection
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）；WeChat AI, Tencent Inc（腾讯公司微信AI）
Counterfactual Explanations for Face Forgery Detection via Adversarial Removal of Artifacts
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Nanjing University of Information Science and Technology（南京信息工程大学）
Low-rank Orthogonal Subspace Intervention for Generalizable Face Forgery Detection
Engineering Research Center of Digital Forensics, Ministry of Education, Nanjing University of Information Science and Technology（数字取证工程研究中心，教育部，南京信息科技大学）
National Institute of Informatics（国立信息学研究所）；The Graduate University for Advanced Studies (SOKENDAI)（综合研究大学院大学（SOKENDAI））；University of Tokyo（东京大学）
The proliferation of deepfake media is raising concerns among the public and relevant authorities. It has become essential to develop countermeasures against forged faces in social media. This paper presents a comprehensive study on two new countermeasure tasks: multi-face forgery detection and segmentation in-the-wild. Localizing forged faces among multiple human faces in unrestricted natural scenes is far more challenging than the traditional deepfake recognition task. To promote these new tasks, we have created the first large-scale dataset posing a high level of challenges that is designed…
MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detection
Rapid-Rich Object Search (ROSE) Lab, School of Electrical and Electronic Engineering, Nanyang Technology University（快速丰富目标搜索（ROSE）实验室，电气与电子工程学院，南洋理工大学）；School of Computing and Artificial Intelligence, Jiangxi University of Finance and Economics（计算机与人工智能学院，江西财经大学）；Department of Electrical and Engineering, City University of Hong Kong（电气与工程系，香港城市大学）；Department of Computer Science and Computing, Zhejiang University City College（计算机科学与计算系，浙江工业大学城市学院）；Department of Computer Science, City University of Hong Kong（计算机科学系，香港城市大学）
With the rapid development of deep learning technology, more and more face forgeries by deepfake are widely spread on social media, causing serious social concern. Face forgery detection has become a research hotspot in recent years, and many related methods have been proposed until now. For those images with low quality and/or diverse sources, however, the detection performances of existing methods are still far from satisfactory. In this paper, we propose an improved Xception with dual attention mechanism and feature fusion for face forgery detection. Different from the middle flow in origin…
DevFD: Developmental Face Forgery Detection by Learning Shared and Orthogonal LoRA Subspaces
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS部门）；China Mobile Financial Technology Co., Ltd.（中国移动金融科技有限公司）；CAIR, HKSIS, Chinese Academy of Sciences（中国科学院CAIR部门，香港创新技术研究所）；School of Computer Science and Engineering, the Faculty of Innovation Engineering, M.U.S.T（澳门科技大学计算机科学与工程学院，创新工程学院）
Shanghai Jiao Tong University（上海交通大学）；Tencent AI Lab（腾讯AI实验室）；Tencent Data Platform（腾讯数据平台）
Current face forgery detection methods achieve high accuracy under the within-database scenario where training and testing forgeries are synthesized by the same algorithm. However, few of them gain satisfying performance under the cross-database scenario where training and testing forgeries are synthesized by different algorithms. In this paper, we find that current CNN-based detectors tend to overfit to method-specific color textures and thus fail to generalize. Observing that image noises remove color textures and expose discrepancies between authentic and tampered regions, we propose to uti…
Stacking Brick by Brick: Aligned Feature Isolation for Incremental Face Forgery Detection
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School（北京大学深圳研究生院电子与计算机工程学院）；WeChat Vision, Tencent Inc.（腾讯公司微信视觉部）
