Existing face relighting methods often struggle with two problems: maintaining the local facial details of the subject and accurately removing and synthesizing shadows in the relit image, especially hard shadows. We propose a novel deep face relighting method that addresses both problems. Our method learns to predict the ratio (quotient) image between a source image and the target image with the desired lighting, allowing us to relight the image while maintaining the local facial details. During training, our model also learns to accurately modify shadows by using estimated shadow masks to emphasize on the high-contrast shadow borders. Furthermore, we introduce a method to use the shadow mask to estimate the ambient light intensity in an image, and are thus able to leverage multiple datasets during training with different global lighting intensities. With quantitative and qualitative evaluations on the Multi-PIE and FFHQ datasets, we demonstrate that our proposed method faithfully maintains the local facial details of the subject and can accurately handle hard shadows while achieving state-of-the-art face relighting performance.
Michigan State University（密歇根州立大学）；Qualcomm Technologies Inc.（高通技术公司）
Most face relighting methods are able to handle diffuse shadows, but struggle to handle hard shadows, such as those cast by the nose. Methods that propose techniques for handling hard shadows often do not produce geometrically consistent shadows since they do not directly leverage the estimated face geometry while synthesizing them. We propose a novel differentiable algorithm for synthesizing hard shadows based on ray tracing, which we incorporate into training our face relighting model. Our proposed algorithm directly utilizes the estimated face geometry to synthesize geometrically consistent…
Image light source transfer (LLST), as the most challenging task in the domain of image relighting, has attracted extensive attention in recent years. In the latest research, LLST is decomposed three sub-tasks: scene reconversion, shadow estimation, and image re-rendering, which provides a new paradigm for image relighting. However, many problems for scene reconversion and shadow estimation tasks, including uncalibrated feature information and poor semantic information, are still unresolved, thereby resulting in insufficient feature representation. In this paper, we propose novel down-sampling…
Computer Vision Center Universitat Autònoma de Barcelona（Autonomous University of Barcelona计算机视觉中心）；Johns Hopkins University（约翰霍普金斯大学）；Universidad Politécnica de Madrid（马德里理工大学）；Université Laval（拉瓦尔大学）
Beyond Shadows: A Large-Scale Benchmark and Multi-Stage Framework for High-Fidelity Facial Shadow Removal
New York Institute of Technology（纽约理工学院）；Shanghai University（上海大学）；Central South University（中南大学）；Xi'an Jiaotong-Liverpool University（西安交通大学利物浦大学）；Huizhou University（惠州大学）
Tsinghua University（清华大学）；Meta Reality Labs（Meta现实实验室）；Guangdong OPPO Mobile Telecommunications Corp., Ltd（广东OPPO移动通信有限公司）
Single-image human relighting aims to relight a target human under new lighting conditions by decomposing the input image into albedo, shape and lighting. Although plausible relighting results can be achieved, previous methods suffer from both the entanglement between albedo and lighting and the lack of hard shadows, which significantly decrease the realism. To tackle these two problems, we propose a geometry-aware single-image human relighting framework that leverages single-image geometry reconstruction for joint deployment of traditional graphics rendering and neural rendering techniques. F…
School of Computer and Communication Sciences, EPFL（洛桑联邦理工学院计算机与通信科学学院）
Deep image relighting is gaining more interest lately, as it allows photo enhancement through illumination-specific retouching without human effort. Aside from aesthetic enhancement and photo montage, image relighting is valuable for domain adaptation, whether to augment datasets for training or to normalize input test data. Accurate relighting is, however, very challenging for various reasons, such as the difficulty in removing and recasting shadows and the modeling of different surfaces. We present a novel dataset, the Virtual Image Dataset for Illumination Transfer (VIDIT), in an effort to …
Illumination effects in images, specifically cast shadows and shading, have been shown to decrease the performance of deep neural networks on a large number of vision-based detection, recognition and segmentation tasks in urban driving scenes. A key factor that contributes to this performance gap is the lack of `time-of-day' diversity within real, labeled datasets. There have been impressive advances in the realm of image to image translation in transferring previously unseen visual effects into a dataset, specifically in day to night translation. However, it is not easy to constrain what visu…
Adobe Research（奥多比研究院）；University College London（伦敦大学学院）
We propose a relighting method for outdoor images. Our method mainly focuses on predicting cast shadows in arbitrary novel lighting directions from a single image while also accounting for shading and global effects such the sun light color and clouds. Previous solutions for this problem rely on reconstructing occluder geometry, e.g. using multi-view stereo, which requires many images of the scene. Instead, in this work we make use of a noisy off-the-shelf single-image depth map estimation as a source of geometry. Whilst this can be a good guide for some lighting effects, the resulting depth m…
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）
Previous image based relighting methods require capturing multiple images to acquire high frequency lighting effect under different lighting conditions, which needs nontrivial effort and may be unrealistic in certain practical use scenarios. While such approaches rely entirely on cleverly sampling the color images under different lighting conditions, little has been done to utilize geometric information that crucially influences the high-frequency features in the images, such as glossy highlight and cast shadow. We therefore propose a framework for image relighting from a single flash photogra…
SARU: A Shadow-Aware and Removal Unified Framework for Remote Sensing Images with New Benchmarks
MOE Key Lab of ICSP, Anhui Provincial Key Lab of Multimodal Cognitive Computation, IMIS Lab of Anhui Province, School of Computer Science and Technology, Anhui University, Hefei, China（教育部信息与通信系统重点实验室、安徽省多模态认知计算重点实验室、安徽省IMIS实验室、计算机科学与技术学院、安徽大学、合肥，中国）；Graduate School of Frontier Sciences, The University of Tokyo, Chiba, 277-8561, Japan（前沿科学研究生院、东京大学、千叶，日本）
SRUG: Shadow-Guided Relightable Urban Scene with Generation Model
College of Computer Science, Nankai University（南开大学计算机科学学院）；Nankai University（南开大学）；Nanjing University（南京大学）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
The Hong Kong Polytechnic University（香港理工大学）；LIX, Ecole Polytechnique, CNRS, IP Paris（巴黎综合理工学院计算机实验室、法国国家科学研究中心、巴黎理工学院）
Manipulating the light source of given images is an interesting task and useful in various applications, including photography and cinematography. Existing methods usually require additional information like the geometric structure of the scene, which may not be available for most images. In this paper, we formulate the single image relighting task and propose a novel Deep Relighting Network (DRN) with three parts: 1) scene reconversion, which aims to reveal the primary scene structure through a deep auto-encoder network, 2) shadow prior estimation, to predict light effect from the new light d…
AeroDeshadow: Physics-Guided Shadow Synthesis and Penumbra-Aware Deshadowing for Aerospace Imagery
MOE Key Laboratory of ICSP, IMIS Laboratory of Anhui, Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, Zenmorn-AHU AI Joint Laboratory, School of Computer Science and Technology, Anhui University（教育部ICSP重点实验室、安徽IMIS实验室、安徽省多模态认知计算重点实验室、Zenmorn-AHU人工智能联合实验室、安徽大学计算机科学与技术学院）
Shadows are common aspect of images and when left undetected can hinder scene understanding and visual processing. We propose a simple yet effective approach based on reflectance to detect shadows from single image. An image is first segmented and based on the reflectance, illumination and texture characteristics, segments pairs are identified as shadow and non-shadow pairs. The proposed method is tested on two publicly available and widely used datasets. Our method achieves higher accuracy in detecting shadows compared to previous reported methods despite requiring fewer parameters. We also s…
Relighting as a Probe of Visual Priors via Augmented Latent Intrinsics
UvA-Bosch Delta Lab, University of Amsterdam, Amsterdam, Netherlands（乌得勒支大学阿姆斯特丹分校博世Delta实验室）；The University of Chicago, Chicago, USA（芝加哥大学）；Johns Hopkins University, Baltimore, USA（约翰霍普金斯大学）
Image relighting has emerged as a problem of significant research interest inspired by augmented reality applications. Physics-based traditional methods, as well as black box deep learning models, have been developed. The existing deep networks have exploited training to achieve a new state of the art; however, they may perform poorly when training is limited or does not represent problem phenomenology, such as the addition or removal of dense shadows. We propose a model which enriches neural networks with physical insight. More precisely, our method generates the relighted image with new illu…
FlowPortal: Residual-Corrected Flow for Training-Free Video Relighting and Background Replacement
Wangxuan Institute of Computer Technology, State Key Laboratory of Multimedia Information Processing, Peking University, Beijing, China（王轩计算机技术研究所，多媒体信息处理国家重点实验室，北京大学，北京，中国）
University of California, San Diego（加利福尼亚大学圣地亚哥分校）；Massachusetts Institute of Technology（麻省理工学院）；Google（谷歌）；Google Research（谷歌研究院）
The light stage has been widely used in computer graphics for the past two decades, primarily to enable the relighting of human faces. By capturing the appearance of the human subject under different light sources, one obtains the light transport matrix of that subject, which enables image-based relighting in novel environments. However, due to the finite number of lights in the stage, the light transport matrix only represents a sparse sampling on the entire sphere. As a consequence, relighting the subject with a point light or a directional source that does not coincide exactly with one of t…
RelightVid: Temporal-Consistent Diffusion Model for Video Relighting
Shanghai AI Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；Shanghai Jiao Tong University（上海交通大学）；Zhejiang University（浙江大学）；Stanford University（斯坦福大学）；The Chinese University of Hong Kong（香港中文大学）
RelightMaster: Precise Video Relighting with Multi-plane Light Images
Multimedia Laboratory, The Chinese University of Hong Kong（香港中文大学多媒体实验室）；Kling Team, Kuaishou Technology（快手技术 Kling 团队）；CPII under InnoHK（创新香港 CPII）；Zhejiang University（浙江大学）；Dalian University of Technology（大连理工大学）
A unique challenge in creating high-quality animatable and relightable 3D avatars of people is modeling human eyes. The challenge of synthesizing eyes is multifold as it requires 1) appropriate representations for the various components of the eye and the periocular region for coherent viewpoint synthesis, capable of representing diffuse, refractive and highly reflective surfaces, 2) disentangling skin and eye appearance from environmental illumination such that it may be rendered under novel lighting conditions, and 3) capturing eyeball motion and the deformation of the surrounding skin to en…
We show how to relight a scene, depicted in a single image, such that (a) the overall shading has changed and (b) the resulting image looks like a natural image of that scene. Applications for such a procedure include generating training data and building authoring environments. Naive methods for doing this fail. One reason is that shading and albedo are quite strongly related; for example, sharp boundaries in shading tend to appear at depth discontinuities, which usually apparent in albedo. The same scene can be lit in different ways, and established theory shows the different lightings form …
MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；Université Laval（拉瓦尔大学）；Epic Games；Borealis AI
Relighting is an essential step in realistically transferring objects from a captured image into another environment. For example, authentic telepresence in Augmented Reality requires faces to be displayed and relit consistent with the observer's scene lighting. We investigate end-to-end deep learning architectures that both de-light and relight an image of a human face. Our model decomposes the input image into intrinsic components according to a diffuse physics-based image formation model. We enable non-diffuse effects including cast shadows and specular highlights by predicting a residual c…
Structure-Guided Diffusion Models for High-Fidelity Portrait Shadow Removal
School of Computer Science and Engineering, Sun Yat-sen University, China（中山大学计算机科学与工程学院）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China（教育部人工智能与先进计算重点实验室）
ShanghaiTech University（上海科技大学）；University of Toronto（多伦多大学）
In this work, we propose NARRATE, a novel pipeline that enables simultaneously editing portrait lighting and perspective in a photorealistic manner. As a hybrid neural-physical face model, NARRATE leverages complementary benefits of geometry-aware generative approaches and normal-assisted physical face models. In a nutshell, NARRATE first inverts the input portrait to a coarse geometry and employs neural rendering to generate images resembling the input, as well as producing convincing pose changes. However, inversion step introduces mismatch, bringing low-quality images with less facial detai…
