This paper describes a simple image-based method that applies engraving stylisation to portraits using ordered dithering. Face detection is used to estimate a rough proxy geometry of the head consisting of a cylinder, which is used to warp the dither matrix, causing the engraving lines to curve around the face for better stylisation. Finally, an application of the approach to colour engraving is demonstrated.
A portrait is a painting, drawing, photograph, or engraving of a person, especially one depicting only the face or head and shoulders. In the digital world the portrait of a person is captured by having the person as a subject in the image and capturing the image of the person such that the background is blurred. DSLRs generally do it by reducing the aperture to focus on very close regions of interest and automatically blur the background. In this paper I have come up with a novel approach to replicate the portrait mode from DSLR using any smartphone to generate high quality portrait images.
In one of his paintings, the School of Athens, Raphael is depicting Leonardo da Vinci as the philosopher Plato. Some image processing tools can help us in comparing this portrait with two Leonardo's portraits, considered as self-portraits.
We present a method for estimating Neural Radiance Fields (NeRF) from a single headshot portrait. While NeRF has demonstrated high-quality view synthesis, it requires multiple images of static scenes and thus impractical for casual captures and moving subjects. In this work, we propose to pretrain the weights of a multilayer perceptron (MLP), which implicitly models the volumetric density and colors, with a meta-learning framework using a light stage portrait dataset. To improve the generalization to unseen faces, we train the MLP in the canonical coordinate space approximated by 3D face morph…
Portrait images often suffer from undesirable shadows cast by casual objects or even the face itself. While existing methods for portrait shadow removal require training on a large-scale synthetic dataset, we propose the first unsupervised method for portrait shadow removal without any training data. Our key idea is to leverage the generative facial priors embedded in the off-the-shelf pretrained StyleGAN2. To achieve this, we formulate the shadow removal task as a layer decomposition problem: a shadowed portrait image is constructed by the blending of a shadow image and a shadow-free image. W…
One of the most famous drawings by Leonardo da Vinci is a self-portrait in red chalk, where he looks quite old. In fact, there is a sketch in one of his notebooks, partially covered by written notes, that can be a self-portrait of the artist when he was young. The use of image processing, to remove the handwritten text and improve the image, allows a comparison of the two portraits.
PortraitGen: Exemplar-Driven GRPO with Dual-Reward Guidance for Photorealistic Portrait Generation
Dalian University of Technology（大连理工大学）；University of Electronic Science and Technology of China（电子科技大学）；Zhejiang University（浙江大学）；WeChat, Tencent Inc.（腾讯微信）
A key challenge for novel view synthesis of monocular portrait images is 3D consistency under continuous pose variations. Most existing methods rely on 2D generative models which often leads to obvious 3D inconsistency artifacts. We present a 3D-consistent novel view synthesis approach for monocular portrait images based on a recent proposed 3D-aware GAN, namely Generative Radiance Manifolds (GRAM), which has shown strong 3D consistency at multiview image generation of virtual subjects via the radiance manifolds representation. However, simply learning an encoder to map a real image into the l…
Australian National University（澳大利亚国立大学）；Data61/CSIRO（Data61/联邦科学与工业研究组织）
Recovering a photorealistic face from an artistic portrait is a challenging task since crucial facial details are often distorted or completely lost in artistic compositions. To handle this loss, we propose an Attribute-guided Face Recovery from Portraits (AFRP) that utilizes a Face Recovery Network (FRN) and a Discriminative Network (DN). FRN consists of an autoencoder with residual block-embedded skip-connections and incorporates facial attribute vectors into the feature maps of input portraits at the bottleneck of the autoencoder. DN has multiple convolutional and fully-connected layers, an…
Estimating correspondence between two images and extracting the foreground object are two challenges in computer vision. With dual-lens smart phones, such as iPhone 7Plus and Huawei P9, coming into the market, two images of slightly different views provide us new information to unify the two topics. We propose a joint method to tackle them simultaneously via a joint fully connected conditional random field (CRF) framework. The regional correspondence is used to handle textureless regions in matching and make our CRF system computationally efficient. Our method is evaluated over 2,000 new image…
Current image harmonization methods consider the entire background as the guidance for harmonization. However, this may limit the capability for user to choose any specific object/person in the background to guide the harmonization. To enable flexible interaction between user and harmonization, we introduce interactive harmonization, a new setting where the harmonization is performed with respect to a selected \emph{region} in the reference image instead of the entire background. A new flexible framework that allows users to pick certain regions of the background image and use it to guide the …
Cardiff University（卡迪夫大学）；Carleton University（卡尔顿大学）；Tsinghua University（清华大学）；The Interdisciplinary Center Herzliya（赫兹利亚跨学科中心）；Pohang University of Science and Technology（浦项科技大学）；Lambda Labs, Inc.（Lambda Labs公司）；Hasso Plattner Institute, University of Potsdam（波茨坦大学哈索·普拉特纳研究院）；Samsung Advanced Institute of Technology（三星高等技术研究院）
Despite the recent upsurge of activity in image-based non-photorealistic rendering (NPR), and in particular portrait image stylisation, due to the advent of neural style transfer, the state of performance evaluation in this field is limited, especially compared to the norms in the computer vision and machine learning communities. Unfortunately, the task of evaluating image stylisation is thus far not well defined, since it involves subjective, perceptual and aesthetic aspects. To make progress towards a solution, this paper proposes a new structured, three level, benchmark dataset for the eval…
PaAgent: Portrait-Aware Image Restoration Agent via Subjective-Objective Reinforcement Learning
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；Shenzhen Research Institute of Northwestern Polytechnical University（西北工业大学深圳研究院）；State Key Laboratory of Internet of Things for Smart City, University of Macau（澳门大学智慧城市物联网国家重点实验室）；National-Local Joint Engineering Research Center of Biodiagnosis and Biotherapy, the Second Affiliated Hospital of Xi’an Jiaotong University（西安交通大学生物诊断与生物治疗国家地方联合工程研究中心）；College of Information and Control Engineering, Xi’an University of Architecture and Technology（西安建筑科技大学信息与控制工程学院）
University of Science and Technology of China（中国科学技术大学）；Snap Inc（斯奈普公司）；Microsoft Cloud AI（微软云人工智能）；City University of Hong Kong（香港城市大学）
Despite the recent success of face image generation with GANs, conditional hair editing remains challenging due to the under-explored complexity of its geometry and appearance. In this paper, we present MichiGAN (Multi-Input-Conditioned Hair Image GAN), a novel conditional image generation method for interactive portrait hair manipulation. To provide user control over every major hair visual factor, we explicitly disentangle hair into four orthogonal attributes, including shape, structure, appearance, and background. For each of them, we design a corresponding condition module to represent, pr…
Imperial College London（帝国理工学院）；Huawei Technologies（华为技术有限公司）
We present Dynamic Neural Portraits, a novel approach to the problem of full-head reenactment. Our method generates photo-realistic video portraits by explicitly controlling head pose, facial expressions and eye gaze. Our proposed architecture is different from existing methods that rely on GAN-based image-to-image translation networks for transforming renderings of 3D faces into photo-realistic images. Instead, we build our system upon a 2D coordinate-based MLP with controllable dynamics. Our intuition to adopt a 2D-based representation, as opposed to recent 3D NeRF-like systems, stems from t…
Data61/CSIRO（联邦科学与工业研究组织Data61）；Australian National University（澳大利亚国立大学）
Given an artistic portrait, recovering the latent photorealistic face that preserves the subject's identity is challenging because the facial details are often distorted or fully lost in artistic portraits. We develop an Identity-preserving Face Recovery from Portraits (IFRP) method that utilizes a Style Removal network (SRN) and a Discriminative Network (DN). Our SRN, composed of an autoencoder with residual block-embedded skip connections, is designed to transfer feature maps of stylized images to the feature maps of the corresponding photorealistic faces. Owing to the Spatial Transformer Ne…
National Lab of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；North China University of Technology（北方工业大学）
Image matting plays an important role in image and video editing. However, the formulation of image matting is inherently ill-posed. Traditional methods usually employ interaction to deal with the image matting problem with trimaps and strokes, and cannot run on the mobile phone in real-time. In this paper, we propose a real-time automatic deep matting approach for mobile devices. By leveraging the densely connected blocks and the dilated convolution, a light full convolutional network is designed to predict a coarse binary mask for portrait images. And a feathering block, which is edge-preser…
Multimedia Laboratory, The Chinese University of Hong Kong（香港中文大学多媒体实验室）；Adobe NextCam；Shanghai AI Laboratory（上海人工智能实验室）；CPII under InnoHK（创新香港研发平台下的CPII）
City University of Hong Kong（香港城市大学）；Dalian University of Technology（大连理工大学）
To address the challenging portrait video matting problem more precisely, existing works typically apply some matting priors that require additional user efforts to obtain, such as annotated trimaps or background images. In this work, we observe that instead of asking the user to explicitly provide a background image, we may recover it from the input video itself. To this end, we first propose a novel background restoration module (BRM) to recover the background image dynamically from the input video. BRM is extremely lightweight and can be easily integrated into existing matting models. By co…
Australian National University（澳大利亚国立大学）；Data61/CSIRO（数据61/联邦科学与工业研究组织）
Recovering the latent photorealistic faces from their artistic portraits aids human perception and facial analysis. However, a recovery process that can preserve identity is challenging because the fine details of real faces can be distorted or lost in stylized images. In this paper, we present a new Identity-preserving Face Recovery from Portraits (IFRP) to recover latent photorealistic faces from unaligned stylized portraits. Our IFRP method consists of two components: Style Removal Network (SRN) and Discriminative Network (DN). The SRN is designed to transfer feature maps of stylized images…
Beijing Institute of Technology（北京理工大学）；Microsoft Research Asia（微软亚洲研究院）；Tsinghua University（清华大学）
In this paper, we present a learning-based approach for recovering the 3D geometry of human head from a single portrait image. Our method is learned in an unsupervised manner without any ground-truth 3D data. We represent the head geometry with a parametric 3D face model together with a depth map for other head regions including hair and ear. A two-step geometry learning scheme is proposed to learn 3D head reconstruction from in-the-wild face images, where we first learn face shape on single images using self-reconstruction and then learn hair and ear geometry using pairs of images in a stereo…
Lite2Relight: 3D-aware Single Image Portrait Relighting
MPI for Informatics, SIC & VIA Research Center（MPI信息学研究所SIC与VIA研究中心）；Google Inc.（谷歌公司）；Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)（埃尔朗根-纽伦堡弗里德里希-亚历山大大学（FAU））；IST Austria（奥地利科学技术研究所）；ETH Zürich（苏黎世联邦理工学院）；Harvard University（哈佛大学）；MIT（麻省理工学院）；MPI for Informatics（MPI信息学研究所）
Semantic 3D-aware Portrait Synthesis and Manipulation Based on Compositional Neural Radiance Field
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；ByteDance Ltd（字节跳动公司）；Nanjing University（南京大学）
FaceSnap: Enhanced ID-fidelity Network for Tuning-free Portrait Customization
Vision AI System Lab, Nanjing University, Nanjing, China（南京大学视觉人工智能系统实验室）；Research and Development Department, Wonxing Technology, Shanghai, China（Wonxing技术有限公司研发部）
