Face hallucination using cascaded super-resolution and identity priors
University of Ljubljana, Faculty of Electrical Engineering（卢布尔雅那大学电气工程学院）；University of Notre Dame, Department of Computer Science and Engineering（圣母大学计算机科学与工程系）
Klemen Grm, Simon Dobrišek, Walter J. Scheirer, Vitomir Štruc
In this paper we address the problem of hallucinating high-resolution facial images from unaligned low-resolution inputs at high magnification factors. We approach the problem with convolutional neural networks (CNNs) and propose a novel (deep) face hallucination model that incorporates identity priors into the learning procedure. The model consists of two main parts: i) a cascaded super-resolution network that upscales the low-resolution images, and ii) an ensemble of face recognition models that act as identity priors for the super-resolution network during training. Different from competing super-resolution approaches that typically rely on a single model for upscaling (even with large magnification factors), our network uses a cascade of multiple SR models that progressively upscale the low-resolution images using steps of $2\times$. This characteristic allows us to apply supervision signals (target appearances) at different resolutions and incorporate identity constraints at multiple-scales. Our model is able to upscale (very) low-resolution images captured in unconstrained conditions and produce visually convincing results. We rigorously evaluate the proposed model on a large datasets of facial images and report superior performance compared to the state-of-the-art.
Over the past few decades, numerous attempts have been made to address the problem of recovering a high-resolution (HR) facial image from its corresponding low-resolution (LR) counterpart, a task commonly referred to as face hallucination. Despite the impressive performance achieved by position-patch and deep learning-based methods, most of these techniques are still unable to recover identity-specific features of faces. The former group of algorithms often produces blurry and oversmoothed outputs particularly in the presence of higher levels of degradation, whereas the latter generates faces …
University of California, Merced（加州大学默塞德分校）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；The Chinese University of Hong Kong（香港中文大学）
We present a novel framework for hallucinating faces of unconstrained poses and with very low resolution (face size as small as 5pxIOD). In contrast to existing studies that mostly ignore or assume pre-aligned face spatial configuration (e.g. facial landmarks localization or dense correspondence field), we alternatingly optimize two complementary tasks, namely face hallucination and dense correspondence field estimation, in a unified framework. In addition, we propose a new gated deep bi-network that contains two functionality-specialized branches to recover different levels of texture details…
University of Maryland（马里兰大学）；Academia Sinica（中央研究院）；Johns Hopkins University（约翰斯·霍普金斯大学）
Face super-resolution is a challenging and highly ill-posed problem since a low-resolution (LR) face image may correspond to multiple high-resolution (HR) ones during the hallucination process and cause a dramatic identity change for the final super-resolved results. Thus, to address this problem, we propose an end-to-end progressive learning framework incorporating facial attributes and enforcing additional supervision from multi-scale discriminators. By incorporating facial attributes into the learning process and progressively resolving the facial image, the mapping between LR and HR images…
Halt the Hallucination: Decoupling Signal and Semantic OOD Detection Based on Cascaded Early Rejection
School of Computer and Electronic Information, Nanjing Normal University, China（南京师范大学计算机与电子信息学院）；School of Artificial Intelligence and Information Technology, Nanjing University of Chinese Medicine, China（南京中医药大学人工智能与信息技术学院）
National Taiwan University（国立台湾大学）；SenseTime Group Limited（商汤集团有限公司）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Shenzhen Institutes of Advanced Technology, CAS（中国科学院深圳先进技术研究院）；Tencent AI Lab（腾讯人工智能实验室）
Face hallucination is a generative task to super-resolve the facial image with low resolution while human perception of face heavily relies on identity information. However, previous face hallucination approaches largely ignore facial identity recovery. This paper proposes Super-Identity Convolutional Neural Network (SICNN) to recover identity information for generating faces closed to the real identity. Specifically, we define a super-identity loss to measure the identity difference between a hallucinated face and its corresponding high-resolution face within the hypersphere identity metric s…
National Pingtung University of Science and Technology（国立屏东科技大学）；National Tsing Hua University（国立清华大学）；National Institute of Informatics（国立信息学研究所）
Despite generative adversarial networks (GANs) can hallucinate photo-realistic high-resolution (HR) faces from low-resolution (LR) faces, they cannot guarantee preserving the identities of hallucinated HR faces, making the HR faces poorly recognizable. To address this problem, we propose a Siamese GAN (SiGAN) to reconstruct HR faces that visually resemble their corresponding identities. On top of a Siamese network, the proposed SiGAN consists of a pair of two identical generators and one discriminator. We incorporate reconstruction error and identity label information in the loss function of S…
There are many factors affecting visual face recognition, such as low resolution images, aging, illumination and pose variance, etc. One of the most important problem is low resolution face images which can result in bad performance on face recognition. Most of the general face recognition algorithms usually assume a sufficient resolution for the face images. However, in practice many applications often do not have sufficient image resolutions. The modern face hallucination models demonstrate reasonable performance to reconstruct high-resolution images from its corresponding low resolution ima…
ByteDance Inc（字节跳动公司）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanghai AI Laboratory（上海人工智能实验室）
Face image super resolution (face hallucination) usually relies on facial priors to restore realistic details and preserve identity information. Recent advances can achieve impressive results with the help of GAN prior. They either design complicated modules to modify the fixed GAN prior or adopt complex training strategies to finetune the generator. In this work, we propose a generative and controllable face SR framework, called GCFSR, which can reconstruct images with faithful identity information without any additional priors. Generally, GCFSR has an encoder-generator architecture. Two modu…
In this paper, we address the problem of face hallucination by proposing a novel multi-scale generative adversarial network (GAN) architecture optimized for face verification. First, we propose a multi-scale generator architecture for face hallucination with a high up-scaling ratio factor, which has multiple intermediate outputs at different resolutions. The intermediate outputs have the growing goal of synthesizing small to large images. Second, we incorporate a face verifier with the original GAN discriminator and propose a novel discriminator which learns to discriminate different identitie…
Deep models have achieved impressive performance for face hallucination tasks. However, we observe that directly feeding the hallucinated facial images into recog- nition models can even degrade the recognition performance despite the much better visualization quality. In this paper, we address this problem by jointly learning a deep model for two tasks, i.e. face hallucination and recognition. In particular, we design an end-to-end deep convolution network with hallucination sub-network cascaded by recognition sub-network. The recognition sub- network are responsible for producing discriminat…
LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models
Adobe Research, Romania（罗马尼亚Adobe研究院）；Department of Computer Science, University of Bucharest, Romania（罗马尼亚布加勒斯特大学计算机科学系）；International Computer High School of Bucharest, Romania（罗马尼亚布加勒斯特国际计算机高中）
Hong Kong Polytechnic University（香港理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））
This paper presents a Deep convolutional network model for Identity-Aware Transfer (DIAT) of facial attributes. Given the source input image and the reference attribute, DIAT aims to generate a facial image that owns the reference attribute as well as keeps the same or similar identity to the input image. In general, our model consists of a mask network and an attribute transform network which work in synergy to generate a photo-realistic facial image with the reference attribute. Considering that the reference attribute may be only related to some parts of the image, the mask network is intro…
Xi’an Jiaotong University（西安交通大学）；National University of Singapore（新加坡国立大学）；National University of Defense Technology（中国人民解放军国防科技大学）；Pensees Pte Ltd（Pensees私人有限公司）；University of Electronic Science and Technology of China（电子科技大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Recent deep learning based face recognition methods have achieved great performance, but it still remains challenging to recognize very low-resolution query face like 28x28 pixels when CCTV camera is far from the captured subject. Such face with very low-resolution is totally out of detail information of the face identity compared to normal resolution in a gallery and hard to find corresponding faces therein. To this end, we propose a Resolution Invariant Model (RIM) for addressing such cross-resolution face recognition problems, with three distinct novelties. First, RIM is a novel and unified…
SynMind: Reducing Semantic Hallucination in fMRI-Based Image Reconstruction
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（人工智能学院，北京邮电大学）；SketchX Lab, Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey（SketchX实验室，视觉、语音和信号处理中心（CVSSP）， Surrey大学）
PASDiff: Physics-Aware Semantic Guidance for Joint Real-World Low-Light Face Enhancement and Restoration
Nanjing University of Posts and Telecommunications（南京邮电大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；PCA Lab, Nanjing University of Science and Technology（南京理工大学PCA实验室）；East China Normal University（华东师范大学）
School of Cyber Science and Engineering, Southeast University（东南大学网络空间科学与工程学院）；Engineering Research Center of Blockchain Application, Supervision and Management (Southeast University), Ministry of Education（教育部区块链应用与监管管理工程研究中心（东南大学））；Purple Mountain Laboratories（紫金山实验室）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；Lee Kong Chian School of Medicine, Nanyang Technological University（南洋理工大学李光前医学院）；Faculty of Engineering, Shenzhen MSU-BIT University（深圳北理莫斯科大学工程学院）；VinUniversity；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Rutgers University（罗格斯大学）；IBM T. J. Watson Research Center（IBM托马斯·J·沃森研究中心）；Intellifusion（智融）
We propose a novel method for real-time face alignment in videos based on a recurrent encoder-decoder network model. Our proposed model predicts 2D facial point heat maps regularized by both detection and regression loss, while uniquely exploiting recurrent learning at both spatial and temporal dimensions. At the spatial level, we add a feedback loop connection between the combined output response map and the input, in order to enable iterative coarse-to-fine face alignment using a single network model, instead of relying on traditional cascaded model ensembles. At the temporal level, we first…
South China University of Technology（华南理工大学）；Tencent（腾讯）；Cardiff University（卡迪夫大学）；The University of Hong Kong（香港大学）
We present a novel high-resolution face swapping method using the inherent prior knowledge of a pre-trained GAN model. Although previous research can leverage generative priors to produce high-resolution results, their quality can suffer from the entangled semantics of the latent space. We explicitly disentangle the latent semantics by utilizing the progressive nature of the generator, deriving structure attributes from the shallow layers and appearance attributes from the deeper ones. Identity and pose information within the structure attributes are further separated by introducing a landmark…
Due to the massive explanation of artificial intelligence, machine learning technology is being used in various areas of our day-to-day life. In the world, there are a lot of scenarios where a simple crime can be prevented before it may even happen or find the person responsible for it. A face is one distinctive feature that we have and can differentiate easily among many other species. But not just different species, it also plays a significant role in determining someone from the same species as us, humans. Regarding this critical feature, a single problem occurs most often nowadays. When th…
