With the rapid development of facial manipulation techniques, face forgery has received considerable attention in multimedia and computer vision community due to security concerns. Existing methods are mostly designed for single-frame detection trained with precise image-level labels or for video-level prediction by only modeling the inter-frame inconsistency, leaving potential high risks for DeepFake attackers. In this paper, we introduce a new problem of partial face attack in DeepFake video, where only video-level labels are provided but not all the faces in the fake videos are manipulated. We address this problem by multiple instance learning framework, treating faces and input video as instances and bag respectively. A sharp MIL (S-MIL) is proposed which builds direct mapping from instance embeddings to bag prediction, rather than from instance embeddings to instance prediction and then to bag prediction in traditional MIL. Theoretical analysis proves that the gradient vanishing in traditional MIL is relieved in S-MIL. To generate instances that can accurately incorporate the partially manipulated faces, spatial-temporal encoded instance is designed to fully model the intra-frame and inter-frame inconsistency, which further helps to promote the detection performance. We also construct a new dataset FFPMS for partially attacked DeepFake video detection, which can benefit the evaluation of different methods at both frame and video levels. Experiments on FFPMS and the widely used DFDC dataset verify that S-MIL is superior to other counterparts for partially attacked DeepFake video detection. In addition, S-MIL can also be adapted to traditional DeepFake image detection tasks and achieve state-of-the-art performance on single-frame datasets.
National Institute of Informatics（国立信息学研究所）；The Graduate University for Advanced Studies (SOKENDAI)（综合研究大学院大学（SOKENDAI））；University of Tokyo（东京大学）
The proliferation of deepfake media is raising concerns among the public and relevant authorities. It has become essential to develop countermeasures against forged faces in social media. This paper presents a comprehensive study on two new countermeasure tasks: multi-face forgery detection and segmentation in-the-wild. Localizing forged faces among multiple human faces in unrestricted natural scenes is far more challenging than the traditional deepfake recognition task. To promote these new tasks, we have created the first large-scale dataset posing a high level of challenges that is designed…
Adversarially Robust Deepfake Detection via Adversarial Feature Similarity Learning
Research Center for Information Technology Innovation, Academia Sinica（中央研究院資訊科技創新研究中心）；National Chengchi University（國立政治大學）；National Taiwan University（國立臺灣大學）
CrossDF: Improving Cross-Domain Deepfake Detection with Deep Information Decomposition
Chengdu University of Information Technology（成都信息工程大学）；University at Buffalo, SUNY（纽约州立大学布法罗分校）；Purdue University（普渡大学）；Microsoft Research Asia（微软亚洲研究院）；University at Albany, SUNY（纽约州立大学奥尔巴尼分校）
Whether, Which, and Whose: Solving the Triple Challenge of Deepfake Proactive Forensics in Multi-Face Scenarios
College of Computer Science and Technology（计算机科学与技术学院）；Xinjiang University（新疆大学）；College of Computer Science and Electronic Engineering（计算机科学与电子工程学院）；Hunan University（湖南大学）
Northwood High School（诺斯伍德高中）；Irvine Unified School District（尔湾联合学区）；Fowler School of Engineering（福勒工程学院）；Chapman University（查普曼大学）
Deep generative networks in recent years have reinforced the need for caution while consuming various modalities of digital information. One avenue of deepfake creation is aligned with injection and removal of tumors from medical scans. Failure to detect medical deepfakes can lead to large setbacks on hospital resources or even loss of life. This paper attempts to address the detection of such attacks with a structured case study. Specifically, we evaluate eight different machine learning algorithms, which including three conventional machine learning methods, support vector machine, random fo…
Beijing University of Posts and Telecommunications（北京邮电大学）；University of California, Berkeley（加州大学伯克利分校）
The divergence between labeled training data and unlabeled testing data is a significant challenge for recent deep learning models. Unsupervised domain adaptation (UDA) attempts to solve such a problem. Recent works show that self-training is a powerful approach to UDA. However, existing methods have difficulty in balancing scalability and performance. In this paper, we propose an instance adaptive self-training framework for UDA on the task of semantic segmentation. To effectively improve the quality of pseudo-labels, we develop a novel pseudo-label generation strategy with an instance adapti…
Deep learning has enabled realistic face manipulation (i.e., deepfake), which poses significant concerns over the integrity of the media in circulation. Most existing deep learning techniques for deepfake detection can achieve promising performance in the intra-dataset evaluation setting (i.e., training and testing on the same dataset), but are unable to perform satisfactorily in the inter-dataset evaluation setting (i.e., training on one dataset and testing on another). Most of the previous methods use the backbone network to extract global features for making predictions and only employ bina…
In this paper, we introduce MINTIME, a video deepfake detection approach that captures spatial and temporal anomalies and handles instances of multiple people in the same video and variations in face sizes. Previous approaches disregard such information either by using simple a-posteriori aggregation schemes, i.e., average or max operation, or using only one identity for the inference, i.e., the largest one. On the contrary, the proposed approach builds on a Spatio-Temporal TimeSformer combined with a Convolutional Neural Network backbone to capture spatio-temporal anomalies from the face sequ…
Learning Real Facial Concepts for Independent Deepfake Detection
School of Software, Shandong University（山东大学软件学院）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；Nanyang Technological University（南洋理工大学）
Recent advances in autoencoders and generative models have given rise to effective video forgery methods, used for generating so-called "deepfakes". Mitigation research is mostly focused on post-factum deepfake detection and not on prevention. We complement these efforts by introducing a novel class of adversarial attacks---training-resistant attacks---which can disrupt face-swapping autoencoders whether or not its adversarial images have been included in the training set of said autoencoders. We propose the Oscillating GAN (OGAN) attack, a novel attack optimized to be training-resistant, whic…
SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model
University of Liverpool, UK（利物浦大学）；Nanyang Technological University, SG（南洋理工大学）；WMG, University of Warwick（沃里克大学工程学院）；The Chinese University of Hong Kong, Shenzhen, Guangdong, China（香港中文大学（深圳））
LAKAN: Landmark-assisted Adaptive Kolmogorov-Arnold Network for Face Forgery Detection
School of Cyber Science and Technology, University of Science and Technology of China（计算机科学与技术学院，中国科学技术大学）；Anhui Province Key Laboratory of Digital Security（安徽省数字安全重点实验室）
Nagoya Institute of Technology（名古屋工业大学）；Chubu University（中部大学）
In this paper we propose an extension of the Attention Branch Network (ABN) by using instance segmentation for generating sharper attention maps for action recognition. Methods for visual explanation such as Grad-CAM usually generate blurry maps which are not intuitive for humans to understand, particularly in recognizing actions of people in videos. Our proposed method, Object-ABN, tackles this issue by introducing a new mask loss that makes the generated attention maps close to the instance segmentation result. Further the PC loss and multiple attention maps are introduced to enhance the sha…
Learning to Aggregate Multi-Scale Context for Instance Segmentation in Remote Sensing Images
The Hong Kong Polytechnic University（香港理工大学）；Wuhan University（武汉大学）；Changjiang Spatial Information Technology Engineering Co., Ltd.（长江空间信息技术工程有限公司）；Peng Cheng Laboratory（鹏城实验室）
Recent deep learning based video synthesis approaches, in particular with applications that can forge identities such as "DeepFake", have raised great security concerns. Therefore, corresponding deep forensic methods are proposed to tackle this problem. However, existing methods are either based on unexplainable deep networks which greatly degrades the principal interpretability factor to media forensic, or rely on fragile image statistics such as noise pattern, which in real-world scenarios can be easily deteriorated by data compression. In this paper, we propose an fully-interpretable video …
Bimodal SegNet: Instance Segmentation Fusing Events and RGB Frames for Robotic Grasping
Faculty of Science, Engineering and Computing, Kingston University（科学、工程与计算学院，金斯顿大学）；Aerospace Engineering Department, Khalifa University of Science and Technology（航空航天工程系，哈利法大学科技学院）；Dubai Future Labs - RLab（迪拜未来实验室-RLab）
CustomTex: High-fidelity Indoor Scene Texturing via Multi-Reference Customization
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）
Topological Inductive Bias fosters Multiple Instance Learning in Data-Scarce Scenarios
Helmholtz Munich Technical University of Munich（海德堡-慕尼黑应用技术大学）；Ludwig-Maximilian-University Hospital University of Munich（路德维希-马克西米利安大学医院慕尼黑大学）；DKTK, German Cancer Consortium（德国癌症联合会）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Helmholtz Munich University of Fribourg（海德堡-慕尼黑应用技术大学弗里堡大学）
