Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues
SenseTime Research（商汤科技研究院）；University of Electronic Science and Technology of China（电子科技大学）；College of Software, Beihang University（北京航空航天大学软件学院）；Northwestern Polytechnical University（西北工业大学）
Yuyang Qian, Guojun Yin, Lu Sheng, Zixuan Chen, Jing Shao
备注：
21 pages, 9 figures, accepted as a POSTER at ECCV2020, UPDATE the appendix of the paper
As realistic facial manipulation technologies have achieved remarkable progress, social concerns about potential malicious abuse of these technologies bring out an emerging research topic of face forgery detection. However, it is extremely challenging since recent advances are able to forge faces beyond the perception ability of human eyes, especially in compressed images and videos. We find that mining forgery patterns with the awareness of frequency could be a cure, as frequency provides a complementary viewpoint where either subtle forgery artifacts or compression errors could be well described. To introduce frequency into the face forgery detection, we propose a novel Frequency in Face Forgery Network (F3-Net), taking advantages of two different but complementary frequency-aware clues, 1) frequency-aware decomposed image components, and 2) local frequency statistics, to deeply mine the forgery patterns via our two-stream collaborative learning framework. We apply DCT as the applied frequency-domain transformation. Through comprehensive studies, we show that the proposed F3-Net significantly outperforms competing state-of-the-art methods on all compression qualities in the challenging FaceForensics++ dataset, especially wins a big lead upon low-quality media.
Xidian University（西安电子科技大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
In recent years, with the rapid development of face editing and generation, more and more fake videos are circulating on social media, which has caused extreme public concerns. Existing face forgery detection methods based on frequency domain find that the GAN forged images have obvious grid-like visual artifacts in the frequency spectrum compared to the real images. But for synthesized videos, these methods only confine to single frame and pay little attention to the most discriminative part and temporal frequency clue among different frames. To take full advantage of the rich information in …
With the rapid development of facial forgery techniques, forgery detection has attracted more and more attention due to security concerns. Existing approaches attempt to use frequency information to mine subtle artifacts under high-quality forged faces. However, the exploitation of frequency information is coarse-grained, and more importantly, their vanilla learning process struggles to extract fine-grained forgery traces. To address this issue, we propose a progressive enhancement learning framework to exploit both the RGB and fine-grained frequency clues. Specifically, we perform a fine-grai…
Videos are prone to tampering attacks that alter the meaning and deceive the audience. Previous video forgery detection schemes find tiny clues to locate the tampered areas. However, attackers can successfully evade supervision by destroying such clues using video compression or blurring. This paper proposes a video watermarking network for tampering localization. We jointly train a 3D-UNet-based watermark embedding network and a decoder that predicts the tampering mask. The perturbation made by watermark embedding is close to imperceptible. Considering that there is no off-the-shelf different…
Enhancing Frequency Forgery Clues for Diffusion-Generated Image Detection
School of Computer and Communication Sciences, EPFL（瑞士联邦理工学院计算机与通信科学学院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）
SUMI-IFL: An Information-Theoretic Framework for Image Forgery Localization with Sufficiency and Minimality Constraints
Sun Yat-sen University（中山大学）；State Key Laboratory of Mathematical Engineering and Advanced Computing（数学工程与先进计算国家重点实验室）；University of Macau（澳门大学）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）
Image forgery localization is a very active and open research field for the difficulty to handle the large variety of manipulations a malicious user can perform by means of more and more sophisticated image editing tools. Here, we propose a localization framework based on the fusion of three very different tools, based, respectively, on sensor noise, patch-matching, and machine learning. The binary masks provided by these tools are finally fused based on some suitable reliability indexes. According to preliminary experiments on the training set, the proposed framework provides often a very goo…
ForgerySleuth: Empowering Multimodal Large Language Models for Image Manipulation Detection
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University（上海智能信息处理关键实验室，复旦大学计算机学院）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）
Latent Spatiotemporal Adaptation for Generalized Face Forgery Video Detection
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tsinghua University（清华大学）；RealAI
MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detection
Rapid-Rich Object Search (ROSE) Lab, School of Electrical and Electronic Engineering, Nanyang Technology University（快速丰富目标搜索（ROSE）实验室，电气与电子工程学院，南洋理工大学）；School of Computing and Artificial Intelligence, Jiangxi University of Finance and Economics（计算机与人工智能学院，江西财经大学）；Department of Electrical and Engineering, City University of Hong Kong（电气与工程系，香港城市大学）；Department of Computer Science and Computing, Zhejiang University City College（计算机科学与计算系，浙江工业大学城市学院）；Department of Computer Science, City University of Hong Kong（计算机科学系，香港城市大学）
DFIL: Deepfake Incremental Learning by Exploiting Domain-invariant Forgery Clues
Zhejiang University（浙江大学）；ZJU-Hangzhou Global Scientific and Technological Innovation Center（浙江大学杭州国际科创中心）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；University College London（伦敦大学学院）
Various facial manipulation techniques have drawn serious public concerns in morality, security, and privacy. Although existing face forgery classifiers achieve promising performance on detecting fake images, these methods are vulnerable to adversarial examples with injected imperceptible perturbations on the pixels. Meanwhile, many face forgery detectors always utilize the frequency diversity between real and fake faces as a crucial clue. In this paper, instead of injecting adversarial perturbations into the spatial domain, we propose a frequency adversarial attack method against face forgery…
University of Notre Dame（圣母大学）；Politecnico di Milano（米兰理工大学）；University of Campinas（坎皮纳斯大学）
As image tampering becomes ever more sophisticated and commonplace, the need for image forensics algorithms that can accurately and quickly detect forgeries grows. In this paper, we revisit the ideas of image querying and retrieval to provide clues to better localize forgeries. We propose a method to perform large-scale image forensics on the order of one million images using the help of an image search algorithm and database to gather contextual clues as to where tampering may have taken place. In this vein, we introduce five new strongly invariant image comparison methods and test their effe…
University of Science and Technology of China（中国科学技术大学）；Kuaishou Technology（快手科技）
Face forgery detection is raising ever-increasing interest in computer vision since facial manipulation technologies cause serious worries. Though recent works have reached sound achievements, there are still unignorable problems: a) learned features supervised by softmax loss are separable but not discriminative enough, since softmax loss does not explicitly encourage intra-class compactness and interclass separability; and b) fixed filter banks and hand-crafted features are insufficient to capture forgery patterns of frequency from diverse inputs. To compensate for such limitations, a novel …
Open-Set Deepfake Detection: A Parameter-Efficient Adaptation Method with Forgery Style Mixture
Rapid-Rich Object Search (ROSE) Lab, School of Electrical and Electronic Engineering, Nanyang Technology University（快速丰富对象搜索实验室（ROSE实验室），电子与电气工程学院，南洋理工大学）；School of Computing and Artificial Intelligence, Jiangxi University of Finance and Economics（计算机与人工智能学院，江西财经大学）；Department of Computer Science, Hong Kong Baptist University（计算机科学系，香港 Baptist大学）；Department of Electrical Engineering, City University of Hong Kong（电气工程系，香港城市大学）；Department of Computer Science and Computing, Zhejiang University City College（计算机科学与计算系，浙江大学城市学院）；Artificial Intelligence Lab. ( Recod.ai ) at the University of Campinas（Campinas大学的人工智能实验室（Recod.ai））
As the remarkable development of facial manipulation technologies is accompanied by severe security concerns, face forgery detection has become a recent research hotspot. Most existing detection methods train a binary classifier under global supervision to judge real or fake. However, advanced manipulations only perform small-scale tampering, posing challenges to comprehensively capture subtle and local forgery artifacts, especially in high compression settings and cross-dataset scenarios. To address such limitations, we propose a novel framework named Multi-modal Contrastive Classification by…
Tsinghua University（清华大学）；Alibaba Security AI Lab（阿里巴巴安全人工智能实验室）
Face forgery has attracted increasing attention in recent applications of computer vision. Existing detection techniques using the two-branch framework benefit a lot from a frequency perspective, yet are restricted by their fixed frequency decomposition and transform. In this paper, we propose to Adaptively learn Frequency information in the two-branch Detection framework, dubbed AFD. To be specific, we automatically learn decomposition in the frequency domain by introducing heterogeneity constraints, and propose an attention-based module to adaptively incorporate frequency features into spati…
SHIELD : An Evaluation Benchmark for Face Spoofing and Forgery Detection with Multimodal Large Language Models
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；Department of Artificial Intelligence, Xiamen University（厦门大学人工智能系）；School of Information Science and Technology, Shijiazhuang Tiedao University（石家庄铁道大学信息科学与技术学院）；Electrical Engineering Department, UCLA（加州大学洛杉矶分校电子工程系）；New Laboratory of Pattern Recognition(NLPR), Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别新实验室）；College of Electronics and Information Engineering, Shenzhen University（深圳大学电子与信息工程学院）；School of Computing and Information Technology, Great Bay University（广东东莞Great Bay大学计算与信息科技学院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络科学与技术学院）
Generalized Face Forgery Detection via Adaptive Learning for Pre-trained Vision Transformer
School of Information Technology, Jiangxi University of Finance and Economics（江西财经大学信息技术学院）；School of Computer Science and Engineering, Sun Yat-Sen University（中山大学计算机科学与工程学院）；Rapid-Rich Object Search (ROSE) Lab, School of Electrical and Electronic Engineering, Nanyang Technology University（南洋理工大学电机与电子工程学院快速目标搜索实验室）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（深圳大学大数据系统计算技术国家工程实验室）；China-Singapore International Joint Research Institute（中新国际联合研究院）
University of Shanghai for Science and Technology（上海理工大学）；Fudan University（复旦大学）
Visual noise is often regarded as a disturbance in image quality, whereas it can also provide a crucial clue for image-based forensic tasks. Conventionally, noise is assumed to comprise an additive Gaussian model to be estimated and then used to reveal anomalies. However, for real sensor noise, it should be modeled as signal-dependent noise (SDN). In this work, we apply SDN to splicing forgery localization tasks. Through statistical analysis of the SDN model, we assume that noise can be modeled as a Gaussian approximation for a certain brightness and propose a likelihood model for a noise leve…
FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Models
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology（广东省超高清沉浸式媒体技术重点实验室）；School of Future Technology, South China University of Technology（华南理工大学未来技术学院）
Towards Generalizable Deepfake Detection via Forgery-aware Audio-Visual Adaptation: A Variational Bayesian Approach
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Department of New Networks, Pengcheng Laboratory（鹏城实验室网络部）；School of Cyber Science and Technology, Sun Yat-sen University（中山大学网络安全科学与技术学院）；School of Computer Science and Cyber Engineering, Guangzhou University（广州大学计算机科学与网络工程学院）；School of Cyberspace Science, Harbin Institute of Technology（哈尔滨工业大学网络空间科学学院）
ED$^4$: Explicit Data-level Debiasing for Deepfake Detection
School of computer science, Wuhan University（武汉大学计算机学院）；WeChat, Tencent Inc.（腾讯公司）；School of electronic and computer engineering, Peking University Shenzhen Graduate School（北京大学深圳研究生院电子与计算机工程学院）
HydraPrompt: An Adaptive and Asymmetric Framework of Vision-Language Models for Synthetic Image Detection
Beijing University of Posts and Telecommunications（北京邮电大学）；School of Advanced Interdisciplinary Sciences (SAIS), University of Chinese Academy of Sciences（中国科学院大学先进交叉学科学院）；Shenzhen Institute of Advanced Technology (SIAT), Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS）；University of Barcelona（巴塞罗那大学）
Pixel-Inconsistency Modeling for Image Manipulation Localization
Nanyang Technology University（南洋理工大学）；School of Electrical and Electronic Engineering（电子电气工程学院）；Rapid-Rich Object Search (ROSE) Lab（快速富对象搜索实验室）；School of Computer Science and Engineering（计算机科学与工程学院）；Sun Yat-sen University（中山大学）；City University of Hong Kong（香港城市大学）；University of Campinas（坎皮纳斯大学）；Institute of Computing（计算机研究所）；Artificial Intelligence Laboratory (Recod.ai)（人工智能实验室（Recod.ai））
Image splicing is a common form of image forgery. Such alterations may leave no visual clues of tampering. In recent works camera characteristics consistency across the image has been used to establish the authenticity and integrity of digital images. Such constant camera characteristic properties are inherent from camera manufacturing processes and are unique. The majority of digital cameras are equipped with spherical lens and this introduces radial distortions on images. This aberration is often disturbed and fails to be consistent across the image, when an image is spliced. This paper desc…
University of Science Technology of China（中国科学技术大学）；Microsoft Cloud AI（微软云人工智能）
The fast evolution and widespread of deepfake techniques in real-world scenarios require stronger generalization abilities of face forgery detectors. Some works capture the features that are unrelated to method-specific artifacts, such as clues of blending boundary, accumulated up-sampling, to strengthen the generalization ability. However, the effectiveness of these methods can be easily corrupted by post-processing operations such as compression. Inspired by transfer learning, neural networks pre-trained on other large-scale face-related tasks may provide useful features for deepfake detecti…
Unveiling Deepfakes: A Frequency-Aware Triple Branch Network for Deepfake Detection
State Key Laboratory of Blockchain and Data Security, Zhejiang University（区块链与数据安全国家重点实验室，浙江大学）；College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；State Grid Blockchain Technology (Beijing) Co., Ltd.（国家电网区块链技术（北京）有限公司）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）；Hangzhou City University（杭州城市学院）；College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）
