This paper presents a novel Region-Aware Face Swapping (RAFSwap) network to achieve identity-consistent harmonious high-resolution face generation in a local-global manner: \textbf{1)} Local Facial Region-Aware (FRA) branch augments local identity-relevant features by introducing the Transformer to effectively model misaligned cross-scale semantic interaction. \textbf{2)} Global Source Feature-Adaptive (SFA) branch further complements global identity-relevant cues for generating identity-consistent swapped faces. Besides, we propose a \textit{Face Mask Predictor} (FMP) module incorporated with StyleGAN2 to predict identity-relevant soft facial masks in an unsupervised manner that is more practical for generating harmonious high-resolution faces. Abundant experiments qualitatively and quantitatively demonstrate the superiority of our method for generating more identity-consistent high-resolution swapped faces over SOTA methods, \eg, obtaining 96.70 ID retrieval that outperforms SOTA MegaFS by 5.87$\uparrow$.
The Open University of Israel（以色列开放大学）；USC（南加州大学）；Information Sciences Institute, USC（南加州大学信息科学研究所）
We show that even when face images are unconstrained and arbitrarily paired, face swapping between them is actually quite simple. To this end, we make the following contributions. (a) Instead of tailoring systems for face segmentation, as others previously proposed, we show that a standard fully convolutional network (FCN) can achieve remarkably fast and accurate segmentations, provided that it is trained on a rich enough example set. For this purpose, we describe novel data collection and generation routines which provide challenging segmented face examples. (b) We use our segmentations to en…
Assessing the Use of Face Swapping Methods as Face Anonymizers in Videos
Department of Computer Engineering, Istanbul Technical University, Istanbul, Turkiye（伊斯坦布尔技术大学计算机工程系）；Division of Engineering, NYU Abu Dhabi, Abu Dhabi, UAE（纽约大学阿布扎比分校工程系）
University of Macau（澳门大学）；University of Chinese Academy of Sciences（中国科学院大学）
Face swapping has both positive applications such as entertainment, human-computer interaction, etc., and negative applications such as DeepFake threats to politics, economics, etc. Nevertheless, it is necessary to understand the scheme of advanced methods for high-quality face swapping and generate enough and representative face swapping images to train DeepFake detection algorithms. This paper proposes the first Megapixel level method for one shot Face Swapping (or MegaFS for short). Firstly, MegaFS organizes face representation hierarchically by the proposed Hierarchical Representation Face…
Waseda University（早稻田大学）；Waseda Research Institute of Science and Engineering（早稻田大学科学与工程研究院）
This paper presents FSNet, a deep generative model for image-based face swapping. Traditionally, face-swapping methods are based on three-dimensional morphable models (3DMMs), and facial textures are replaced between the estimated three-dimensional (3D) geometries in two images of different individuals. However, the estimation of 3D geometries along with different lighting conditions using 3DMMs is still a difficult task. We herein represent the face region with a latent variable that is assigned with the proposed deep neural network (DNN) instead of facial textures. The proposed DNN synthesiz…
IIIT Hyderabad（海得拉巴国际信息技术研究所）；University of Bath（巴斯大学）
Doubles play an indispensable role in the movie industry. They take the place of the actors in dangerous stunt scenes or scenes where the same actor plays multiple characters. The double's face is later replaced with the actor's face and expressions manually using expensive CGI technology, costing millions of dollars and taking months to complete. An automated, inexpensive, and fast way can be to use face-swapping techniques that aim to swap an identity from a source face video (or an image) to a target face video. However, such methods cannot preserve the source expressions of the actor impor…
Zhejiang University（浙江大学）；Georgia Institute of Technology（佐治亚理工学院）
Recently, face swapping has been developing rapidly and achieved a surprising reality, raising concerns about fake content. As a countermeasure, various detection approaches have been proposed and achieved promising performance. However, most existing detectors struggle to maintain performance on unseen face swapping methods and low-quality images. Apart from the generalization problem, current detection approaches have been shown vulnerable to evasion attacks crafted by detection-aware manipulators. Lack of robustness under adversary scenarios leaves threats for applying face swapping detecti…
In this work, we propose a novel two-stage framework, called FaceShifter, for high fidelity and occlusion aware face swapping. Unlike many existing face swapping works that leverage only limited information from the target image when synthesizing the swapped face, our framework, in its first stage, generates the swapped face in high-fidelity by exploiting and integrating the target attributes thoroughly and adaptively. We propose a novel attributes encoder for extracting multi-level target face attributes, and a new generator with carefully designed Adaptive Attentional Denormalization (AAD) l…
Tencent AI Lab（腾讯人工智能实验室）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；South China University of Technology（华南理工大学）
We present a novel paradigm for high-fidelity face swapping that faithfully preserves the desired subtle geometry and texture details. We rethink face swapping from the perspective of fine-grained face editing, \textit{i.e., ``editing for swapping'' (E4S)}, and propose a framework that is based on the explicit disentanglement of the shape and texture of facial components. Following the E4S principle, our framework enables both global and local swapping of facial features, as well as controlling the amount of partial swapping specified by the user. Furthermore, the E4S paradigm is inherently ca…
In this work, we propose a high fidelity face swapping method, called HifiFace, which can well preserve the face shape of the source face and generate photo-realistic results. Unlike other existing face swapping works that only use face recognition model to keep the identity similarity, we propose 3D shape-aware identity to control the face shape with the geometric supervision from 3DMM and 3D face reconstruction method. Meanwhile, we introduce the Semantic Facial Fusion module to optimize the combination of encoder and decoder features and make adaptive blending, which makes the results more …
In this paper, we propose a diffusion-based face swapping framework for the first time, called DiffFace, composed of training ID conditional DDPM, sampling with facial guidance, and a target-preserving blending. In specific, in the training process, the ID conditional DDPM is trained to generate face images with the desired identity. In the sampling process, we use the off-the-shelf facial expert models to make the model transfer source identity while preserving target attributes faithfully. During this process, to preserve the background of the target image and obtain the desired face swappin…
We propose an efficient framework, called Simple Swap (SimSwap), aiming for generalized and high fidelity face swapping. In contrast to previous approaches that either lack the ability to generalize to arbitrary identity or fail to preserve attributes like facial expression and gaze direction, our framework is capable of transferring the identity of an arbitrary source face into an arbitrary target face while preserving the attributes of the target face. We overcome the above defects in the following two ways. First, we present the ID Injection Module (IIM) which transfers the identity informa…
Learning Disentangled Representation for One-shot Progressive Face Swapping
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of Macau（澳门大学）；University of California, Merced（加州大学默塞德分校）；Yonsei University（延世大学）
E4S: Fine-grained Face Swapping via Editing With Regional GAN Inversion
Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；South China University of Technology（华南理工大学）；The University of Hong Kong（香港大学）；Tencent AI Lab（腾讯AI实验室）
Baidu Inc.（百度公司）；South China University of Technology（华南理工大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Advanced face swapping methods have achieved appealing results. However, most of these methods have many parameters and computations, which makes it challenging to apply them in real-time applications or deploy them on edge devices like mobile phones. In this work, we propose a lightweight Identity-aware Dynamic Network (IDN) for subject-agnostic face swapping by dynamically adjusting the model parameters according to the identity information. In particular, we design an efficient Identity Injection Module (IIM) by introducing two dynamic neural network techniques, including the weights predic…
Safeguarding Facial Identity against Diffusion-based Face Swapping via Cascading Pathway Disruption
MoE Key Laboratory of Information Technology, Sun Yat-sen University, Guangzhou, China（信息科技教育部重点实验室，中山大学，广州，中国）；State Key Laboratory of Mathematical Engineering and Advanced Computing, Zhengzhou, China（数学工程与先进计算国家重点实验室，郑州，中国）
In this work, we present a new single-stage method for subject agnostic face swapping and identity transfer, named FaceDancer. We have two major contributions: Adaptive Feature Fusion Attention (AFFA) and Interpreted Feature Similarity Regularization (IFSR). The AFFA module is embedded in the decoder and adaptively learns to fuse attribute features and features conditioned on identity information without requiring any additional facial segmentation process. In IFSR, we leverage the intermediate features in an identity encoder to preserve important attributes such as head pose, facial expressio…
Taking full advantage of the excellent performance of StyleGAN, style transfer-based face swapping methods have been extensively investigated recently. However, these studies require separate face segmentation and blending modules for successful face swapping, and the fixed selection of the manipulated latent code in these works is reckless, thus degrading face swapping quality, generalizability, and practicability. This paper proposes a novel and end-to-end integrated framework for high resolution and attribute preservation face swapping via Adaptive Latent Representation Learning. Specifical…
In this paper, we present an integrated system for automatically generating and editing face images through face swapping, attribute-based editing, and random face parts synthesis. The proposed system is based on a deep neural network that variationally learns the face and hair regions with large-scale face image datasets. Different from conventional variational methods, the proposed network represents the latent spaces individually for faces and hairs. We refer to the proposed network as region-separative generative adversarial network (RSGAN). The proposed network independently handles face …
CanonSwap: High-Fidelity and Consistent Video Face Swapping via Canonical Space Modulation
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；International Digital Economy Academy (IDEA)（国际数字经济学院）；Xi’an Jiaotong University（西安交通大学）
In this work, we investigate into the performance of mainstream neural generative models on the very task of swapping faces. We have experimented on CVAE, CGAN, CVAE-GAN, and conditioned diffusion models. Existing finely trained models have already managed to produce fake faces (Facke) indistinguishable to the naked eye as well as achieve high objective metrics. We perform a comparison among them and analyze their pros and cons. Furthermore, we proposed some promising tricks though they do not apply to this task.
Hanoi University of Science and Technology（河内科技大学）；Applied Artificial Intelligence Institute (A2I2), Deakin University（迪肯大学应用人工智能研究所（A2I2））
We propose a novel high-fidelity face swapping method called "Arithmetic Face Swapping" (AFS) that explicitly disentangles the intermediate latent space W+ of a pretrained StyleGAN into the "identity" and "style" subspaces so that a latent code in W+ is the sum of an "identity" code and a "style" code in the corresponding subspaces. Via our disentanglement, face swapping (FS) can be regarded as a simple arithmetic operation in W+, i.e., the summation of a source "identity" code and a target "style" code. This makes AFS more intuitive and elegant than other FS methods. In addition, our method c…
Disruptive Attacks on Face Swapping via Low-Frequency Perceptual Perturbations
Department of Mathematics and Artificial Intelligence（数学与人工智能系）；Shandong Artificial Intelligence Institute（山东省人工智能研究所）；Qilu University of Technology（齐鲁大学）
Robust Identity Perceptual Watermark Against Deepfake Face Swapping
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；Shandong Artificial Intelligence Institute, Qilu University of Technology (Shandong Academy of Sciences)（山东人工智能研究院，齐鲁工业大学（山东科学院））；School of Cyber Security, Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东科学院）网络安全学院）；Key Laboratory of Computing Power Network and Information Security, Ministry of Education, China（教育部计算能力网络与信息安全重点实验室）
Netease Fuxi AI Lab（网易伏羲人工智能实验室）；University of Technology Sydney（悉尼科技大学）；Zhejiang University（浙江大学）
In this work, we propose a semantic flow-guided two-stage framework for shape-aware face swapping, namely FlowFace. Unlike most previous methods that focus on transferring the source inner facial features but neglect facial contours, our FlowFace can transfer both of them to a target face, thus leading to more realistic face swapping. Concretely, our FlowFace consists of a face reshaping network and a face swapping network. The face reshaping network addresses the shape outline differences between the source and target faces. It first estimates a semantic flow (i.e., face shape differences) be…
Towards High Fidelity Face Swapping: A Comprehensive Survey and New Benchmark
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Nanyang Technological University（南洋理工大学）；Yonsei University（延世大学）
Face-swapping models have been drawing attention for their compelling generation quality, but their complex architectures and loss functions often require careful tuning for successful training. We propose a new face-swapping model called `Smooth-Swap', which excludes complex handcrafted designs and allows fast and stable training. The main idea of Smooth-Swap is to build smooth identity embedding that can provide stable gradients for identity change. Unlike the one used in previous models trained for a purely discriminative task, the proposed embedding is trained with a supervised contrastive…
Numerous attempts have been made to the task of person-agnostic face swapping given its wide applications. While existing methods mostly rely on tedious network and loss designs, they still struggle in the information balancing between the source and target faces, and tend to produce visible artifacts. In this work, we introduce a concise and effective framework named StyleSwap. Our core idea is to leverage a style-based generator to empower high-fidelity and robust face swapping, thus the generator's advantage can be adopted for optimizing identity similarity. We identify that with only minim…
Face swapping aims at injecting a source image's identity (i.e., facial features) into a target image, while strictly preserving the target's attributes, which are irrelevant to identity. However, we observed that previous approaches still suffer from source attribute leakage, where the source image's attributes interfere with the target image's. In this paper, we analyze the latent space of StyleGAN and find the adequate combination of the latents geared for face swapping task. Based on the findings, we develop a simple yet robust face swapping model, RobustSwap, which is resistant to the pot…
South China University of Technology（华南理工大学）；Tencent（腾讯）；Cardiff University（卡迪夫大学）；The University of Hong Kong（香港大学）
We present a novel high-resolution face swapping method using the inherent prior knowledge of a pre-trained GAN model. Although previous research can leverage generative priors to produce high-resolution results, their quality can suffer from the entangled semantics of the latent space. We explicitly disentangle the latent semantics by utilizing the progressive nature of the generator, deriving structure attributes from the shallow layers and appearance attributes from the deeper ones. Identity and pose information within the structure attributes are further separated by introducing a landmark…
To study the preference of infants for contingency of movements and familiarity of faces during self-recognition task, we built, as an accurate and instantaneous imitator, a real-time face- swapper for videos. We present a non-constraint face-swapper based on 3D visual tracking that achieves real-time performance through parallel computing. Our imitator system is par- ticularly suited for experiments involving children with Autistic Spectrum Disorder who are often strongly disturbed by the constraints of other methods.
We present Face Swapping GAN (FSGAN) for face swapping and reenactment. Unlike previous work, we offer a subject agnostic swapping scheme that can be applied to pairs of faces without requiring training on those faces. We derive a novel iterative deep learning--based approach for face reenactment which adjusts significant pose and expression variations that can be applied to a single image or a video sequence. For video sequences, we introduce a continuous interpolation of the face views based on reenactment, Delaunay Triangulation, and barycentric coordinates. Occluded face regions are handle…
Bar-Ilan University（巴伊兰大学）；The Open University of Israel（以色列开放大学）
We present Face Swapping GAN (FSGAN) for face swapping and reenactment. Unlike previous work, FSGAN is subject agnostic and can be applied to pairs of faces without requiring training on those faces. To this end, we describe a number of technical contributions. We derive a novel recurrent neural network (RNN)-based approach for face reenactment which adjusts for both pose and expression variations and can be applied to a single image or a video sequence. For video sequences, we introduce continuous interpolation of the face views based on reenactment, Delaunay Triangulation, and barycentric co…
