Progressive Semantic-Aware Style Transformation for Blind Face Restoration
The University of Hong Kong（香港大学）；Harbin Institute of Technology（哈尔滨工业大学）；DAMO Academy Alibaba Group（阿里巴巴达摩院）；Peking University（北京大学）；The Hong Kong Polytechnic University（香港理工大学）
Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, Kwan-Yee K. Wong
Face restoration is important in face image processing, and has been widely studied in recent years. However, previous works often fail to generate plausible high quality (HQ) results for real-world low quality (LQ) face images. In this paper, we propose a new progressive semantic-aware style transformation framework, named PSFR-GAN, for face restoration. Specifically, instead of using an encoder-decoder framework as previous methods, we formulate the restoration of LQ face images as a multi-scale progressive restoration procedure through semantic-aware style transformation. Given a pair of LQ face image and its corresponding parsing map, we first generate a multi-scale pyramid of the inputs, and then progressively modulate different scale features from coarse-to-fine in a semantic-aware style transfer way. Compared with previous networks, the proposed PSFR-GAN makes full use of the semantic (parsing maps) and pixel (LQ images) space information from different scales of input pairs. In addition, we further introduce a semantic aware style loss which calculates the feature style loss for each semantic region individually to improve the details of face textures. Finally, we pretrain a face parsing network which can generate decent parsing maps from real-world LQ face images. Experiment results show that our model trained with synthetic data can not only produce more realistic high-resolution results for synthetic LQ inputs and but also generalize better to natural LQ face images compared with state-of-the-art methods. Codes are available at https://github.com/chaofengc/PSFRGAN.
Applied Research Center (ARC), Tencent PCG（腾讯平台与内容事业群应用研究中心）
Blind face restoration usually relies on facial priors, such as facial geometry prior or reference prior, to restore realistic and faithful details. However, very low-quality inputs cannot offer accurate geometric prior while high-quality references are inaccessible, limiting the applicability in real-world scenarios. In this work, we propose GFP-GAN that leverages rich and diverse priors encapsulated in a pretrained face GAN for blind face restoration. This Generative Facial Prior (GFP) is incorporated into the face restoration process via novel channel-split spatial feature transform layers,…
DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；The Hong Kong Polytechnic University（香港理工大学）
Blind face restoration (BFR) from severely degraded face images in the wild is a very challenging problem. Due to the high illness of the problem and the complex unknown degradation, directly training a deep neural network (DNN) usually cannot lead to acceptable results. Existing generative adversarial network (GAN) based methods can produce better results but tend to generate over-smoothed restorations. In this work, we propose a new method by first learning a GAN for high-quality face image generation and embedding it into a U-shaped DNN as a prior decoder, then fine-tuning the GAN prior emb…
AuthFace: Towards Authentic Blind Face Restoration with Face-oriented Generative Diffusion Prior
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；vivo Mobile Communication Co., Ltd（vivo移动通信有限公司）；Nanyang Technological University（南洋理工大学）
InterLCM: Low-Quality Images as Intermediate States of Latent Consistency Models for Effective Blind Face Restoration
Nankai University（南开大学）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Linkoping University（林雪平大学）；SB Intuitions, SoftBank（软银SB Intuitions）；Nankai International Advanced Research Institute (Shenzhen Futian), Nankai University（南开大学深圳福田国际先进研究院）
LAFR: Efficient Diffusion-based Blind Face Restoration via Latent Codebook Alignment Adapter
School of Electronic and Computer Engineering, Peking University（电子与计算机工程学院，北京大学）；Computer Vision Lab, CAIDAS & IFI, University of Würzburg（计算机视觉实验室，乌尔姆大学）
3D Priors-Guided Diffusion for Blind Face Restoration
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Yunnan Key Laboratory of Smart Education（云南省智慧教育重点实验室）；Tencent Youtu Lab（腾讯优图实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent Cloud Architecture Platform（腾讯云架构平台）
School of Mathematical Sciences, Dalian University of Technology（大连理工大学数学科学学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
Image inpainting is a challenging problem as it needs to fill the information of the corrupted regions. Most of the existing inpainting algorithms assume that the positions of the corrupted regions are known. Different from the existing methods that usually make some assumptions on the corrupted regions, we present an efficient blind image inpainting algorithm to directly restore a clear image from a corrupted input. Our algorithm is motivated by the residual learning algorithm which aims to learn the missing infor- mation in corrupted regions. However, directly using exist- ing residual learn…
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Alibaba Group（阿里巴巴集团）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Recent generative models show impressive results in photo-realistic image generation. However, artifacts often inevitably appear in the generated results, leading to downgraded user experience and reduced performance in downstream tasks. This work aims to develop a plugin post-processing module for diverse generative models, which can faithfully restore images from diverse generative artifacts. This is challenging because: (1) Unlike traditional degradation patterns, generative artifacts are non-linear and the transformation function is highly complex. (2) There are no readily available artifa…
Tokyo Institute of Technology（东京工业大学）；National Institute of Advanced Industrial Science and Technology（日本产业技术综合研究所）
Blind image restoration processors based on convolutional neural network (CNN) are intensively researched because of their high performance. However, they are too sensitive to the perturbation of the degradation model. They easily fail to restore the image whose degradation model is slightly different from the trained degradation model. In this paper, we propose a non-blind CNN-based image restoration processor, aiming to be robust against a perturbation of the degradation model compared to the blind restoration processor. Experimental comparisons demonstrate that the proposed non-blind CNN-ba…
Enhancing Diffusion Model Stability for Image Restoration via Gradient Management
College of Computer Science, Sichuan University（四川大学计算机学院）；National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University（合成视觉基础科学国家重点实验室（四川大学））；Engineering Research Center of Machine Learning（机器学习工程研究中心）
Echocardiography to Cardiac MRI View Transformation for Real-Time Blind Restoration
Tampere University（坦佩雷大学）；Qatar University（卡塔尔大学）；VTT Technical Research Centre of Finland（芬兰国家技术研究中心）；Hamad Medical Corporation Hospital（哈马德医疗公司医院）；Hayatabad Medical Complex（海亚特阿巴德医疗中心）
Many image restoration techniques are highly dependent on the degradation used during training, and their performance declines significantly when applied to slightly different input. Blind and universal techniques attempt to mitigate this by producing a trained model that can adapt to varying conditions. However, blind techniques to date require prior knowledge of the degradation process, and assumptions regarding its parameter-space. In this paper we present the Self-Normalization Side-Chain (SCNC), a novel approach to blind universal restoration in which no prior knowledge of the degradation…
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
Blind face restoration usually encounters with diverse scale face inputs, especially in the real world. However, most of the current works support specific scale faces, which limits its application ability in real-world scenarios. In this work, we propose a novel scale-aware blind face restoration framework, named FaceFormer, which formulates facial feature restoration as scale-aware transformation. The proposed Facial Feature Up-sampling (FFUP) module dynamically generates upsampling filters based on the original scale-factor priors, which facilitate our network to adapt to arbitrary face sca…
Face restoration is an inherently ill-posed problem, where additional prior constraints are typically considered crucial for mitigating such pathology. However, real-world image prior are often hard to simulate with precise mathematical models, which inevitably limits the performance and generalization ability of existing prior-regularized restoration methods. In this paper, we study the problem of face restoration under a more practical ``dual blind'' setting, i.e., without prior assumptions or hand-crafted regularization terms on the degradation profile or image contents. To this end, a nove…
Towards Real-World Blind Face Restoration with Generative Diffusion Prior
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；College of Engineering and Computer Science, Australian National University（澳大利亚国立大学工程与计算机科学学院）
Harbin Institute of Technology（哈尔滨工业大学）；The Hong Kong Polytechnic University（香港理工大学）；Nanyang Technological University（南洋理工大学）；Peng Cheng Lab（鹏城实验室）
To improve the performance of blind face restoration, recent works mainly treat the two aspects, i.e., generic and specific restoration, separately. In particular, generic restoration attempts to restore the results through general facial structure prior, while on the one hand, cannot generalize to real-world degraded observations due to the limited capability of direct CNNs' mappings in learning blind restoration, and on the other hand, fails to exploit the identity-specific details. On the contrary, specific restoration aims to incorporate the identity features from the reference of the same…
Survey on Deep Face Restoration: From Non-blind to Blind and Beyond
Beijing University of Posts and Telecommunications（北京邮电大学）；ETH Zürich（苏黎世联邦理工学院）；Shanghai University（上海大学）；Nanyang Technological University（南洋理工大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；National Tsing Hua University（国立清华大学）
Blind face restoration is a highly ill-posed problem that often requires auxiliary guidance to 1) improve the mapping from degraded inputs to desired outputs, or 2) complement high-quality details lost in the inputs. In this paper, we demonstrate that a learned discrete codebook prior in a small proxy space largely reduces the uncertainty and ambiguity of restoration mapping by casting blind face restoration as a code prediction task, while providing rich visual atoms for generating high-quality faces. Under this paradigm, we propose a Transformer-based prediction network, named CodeFormer, to…
Institute of Information Science, Academia Sinica（中央研究院信息科学研究所）
Blind image restoration is a non-convex problem which involves restoration of images from an unknown blur kernel. The factors affecting the performance of this restoration are how much prior information about an image and a blur kernel are provided and what algorithm is used to perform the restoration task. Prior information on images is often employed to restore the sharpness of the edges of an image. By contrast, no consensus is still present regarding what prior information to use in restoring from a blur kernel due to complex image blurring processes. In this paper, we propose modelling of…
Enhancing Blind Face Restoration through Online Reinforcement Learning
Institute of Information Science, Beijing Jiaotong University（信息科学学院，北京交通大学）；Visual Intelligence +X International Cooperation Joint Laboratory of MOE（教育部视觉智能+X国际合作联合实验室）；Kuaishou Technology（快手科技）
The University of Hong Kong（香港大学）；SenseTime Research（商汤科技研究院）
Blind face restoration is to recover a high-quality face image from unknown degradations. As face image contains abundant contextual information, we propose a method, RestoreFormer, which explores fully-spatial attentions to model contextual information and surpasses existing works that use local operators. RestoreFormer has several benefits compared to prior arts. First, unlike the conventional multi-head self-attention in previous Vision Transformers (ViTs), RestoreFormer incorporates a multi-head cross-attention layer to learn fully-spatial interactions between corrupted queries and high-qu…
Harbin Institute of Technology（哈尔滨工业大学）；Sun Yat-sen University（中山大学）；University of Kentucky（肯塔基大学）
This paper studies the problem of blind face restoration from an unconstrained blurry, noisy, low-resolution, or compressed image (i.e., degraded observation). For better recovery of fine facial details, we modify the problem setting by taking both the degraded observation and a high-quality guided image of the same identity as input to our guided face restoration network (GFRNet). However, the degraded observation and guided image generally are different in pose, illumination and expression, thereby making plain CNNs (e.g., U-Net) fail to recover fine and identity-aware facial details. To tac…
BIR-Adapter: A parameter-efficient diffusion adapter for blind image restoration
Chair of Media Technology, Munich Institute of Robotics and Machine Intelligence（媒体技术教授会，慕尼黑机器人与机器智能研究所）；School of Computation, Information and Technology, Technical University of Munich（计算、信息与技术学院，慕尼黑技术大学）
Beijing Institute of Technology（北京理工大学）；Australian National University（澳大利亚国立大学）；Sun Yat-sen University（中山大学）
Blind Face Restoration (BFR) aims to construct a high-quality (HQ) face image from its corresponding low-quality (LQ) input. Recently, many BFR methods have been proposed and they have achieved remarkable success. However, these methods are trained or evaluated on privately synthesized datasets, which makes it infeasible for the subsequent approaches to fairly compare with them. To address this problem, we first synthesize two blind face restoration benchmark datasets called EDFace-Celeb-1M (BFR128) and EDFace-Celeb-150K (BFR512). State-of-the-art methods are benchmarked on them under five set…
