We present the first system for physically based, neural inverse rendering from multi-viewpoint videos of propagating light. Our approach relies on a time-resolved extension of neural radiance caching -- a technique that accelerates inverse rendering by storing infinite-bounce radiance arriving at any point from any direction. The resulting model accurately accounts for direct and indirect light transport effects and, when applied to captured measurements from a flash lidar system, enables state-of-the-art 3D reconstruction in the presence of strong indirect light. Further, we demonstrate view synthesis of propagating light, automatic decomposition of captured measurements into direct and indirect components, as well as novel capabilities such as multi-view time-resolved relighting of captured scenes.
GAINS: Gaussian-based Inverse Rendering from Sparse Multi-View Captures
University Kaiserslautern-Landau（凯撒斯劳滕-兰道大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；College of William & Mary（威廉与玛丽学院）
A polarization camera has great potential for 3D reconstruction since the angle of polarization (AoP) of reflected light is related to an object's surface normal. In this paper, we propose a novel 3D reconstruction method called Polarimetric Multi-View Inverse Rendering (Polarimetric MVIR) that effectively exploits geometric, photometric, and polarimetric cues extracted from input multi-view color polarization images. We first estimate camera poses and an initial 3D model by geometric reconstruction with a standard structure-from-motion and multi-view stereo pipeline. We then refine the initia…
Recent advances in implicit neural representations and differentiable rendering make it possible to simultaneously recover the geometry and materials of an object from multi-view RGB images captured under unknown static illumination. Despite the promising results achieved, indirect illumination is rarely modeled in previous methods, as it requires expensive recursive path tracing which makes the inverse rendering computationally intractable. In this paper, we propose a novel approach to efficiently recovering spatially-varying indirect illumination. The key insight is that indirect illuminatio…
GS-IR: 3D Gaussian Splatting for Inverse Rendering
South China University of Technology（华南理工大学）；Tencent AI Lab（腾讯人工智能实验室）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
GUS-IR: Gaussian Splatting with Unified Shading for Inverse Rendering
School of Electronic and Information Engineering, South China University of Technology（华南理工大学电子与信息工程学院）；Australian National University（澳大利亚国立大学）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；VIVO
A polarization camera has great potential for 3D reconstruction since the angle of polarization (AoP) and the degree of polarization (DoP) of reflected light are related to an object's surface normal. In this paper, we propose a novel 3D reconstruction method called Polarimetric Multi-View Inverse Rendering (Polarimetric MVIR) that effectively exploits geometric, photometric, and polarimetric cues extracted from input multi-view color-polarization images. We first estimate camera poses and an initial 3D model by geometric reconstruction with a standard structure-from-motion and multi-view ster…
The University of Hong Kong（香港大学）；Nanyang Technological University（南洋理工大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Traditional multi-view photometric stereo (MVPS) methods are often composed of multiple disjoint stages, resulting in noticeable accumulated errors. In this paper, we present a neural inverse rendering method for MVPS based on implicit representation. Given multi-view images of a non-Lambertian object illuminated by multiple unknown directional lights, our method jointly estimates the geometry, materials, and lights. Our method first employs multi-light images to estimate per-view surface normal maps, which are used to regularize the normals derived from the neural radiance field. It then join…
GeoSplatting: Towards Geometry Guided Gaussian Splatting for Physically-based Inverse Rendering
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Wangxuan Institute of Computer Technology, Peking University（北京大学王轩计算机技术研究所）；State Key Laboratory of Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
Indoor scenes typically exhibit complex, spatially-varying appearance from global illumination, making inverse rendering a challenging ill-posed problem. This work presents an end-to-end, learning-based inverse rendering framework incorporating differentiable Monte Carlo raytracing with importance sampling. The framework takes a single image as input to jointly recover the underlying geometry, spatially-varying lighting, and photorealistic materials. Specifically, we introduce a physically-based differentiable rendering layer with screen-space ray tracing, resulting in more realistic specular …
FlexDrive: Toward Trajectory Flexibility in Driving Scene Reconstruction and Rendering
Multimedia Laboratory, The Chinese University of Hong Kong（香港中文大学多媒体实验室）；Centre for Perceptual and Interactive Intelligence, Hong Kong（香港感知与交互智能中心）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beihang University（北京航空航天大学）
Holistic Inverse Rendering of Complex Facade via Aerial 3D Scanning
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Zhejianglab（之江实验室）；Zhejiang University（浙江大学）；Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
FIRe: Fast Inverse Rendering using Directional and Signed Distance Functions
TU Munich（慕尼黑工业大学）；MCML（慕尼黑计算数学与机器学习实验室）；Massachusetts Institute of Technology（麻省理工学院）；University of Bonn（波恩大学）；MPI Informatics（马克斯·普朗克信息学研究所）；SIC（萨尔信息中心）
In this paper a semi-supervised deep framework is proposed for the problem of 3D shape inverse rendering from a single 2D input image. The main structure of proposed framework consists of unsupervised pre-trained components which significantly reduce the need to labeled data for training the whole framework. using labeled data has the advantage of achieving to accurate results without the need to predefined assumptions about image formation process. Three main components are used in the proposed network: an encoder which maps 2D input image to a representation space, a 3D decoder which decodes…
TransparentGS: Fast Inverse Rendering of Transparent Objects with Gaussians
State Key Lab for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；TMCC, College of Computer Science, Nankai University（天津大学计算机学院）；State Key Lab of CAD & CG, Zhejiang University（浙江大学计算机辅助设计与图形学国家重点实验室）；Institute of Hangzhou Holographic Intelligent Technology（杭州全息智能技术研究院）
We present a method for differentiable rendering of 3D surfaces that supports both explicit and implicit representations, provides derivatives at occlusion boundaries, and is fast and simple to implement. The method first samples the surface using non-differentiable rasterization, then applies differentiable, depth-aware point splatting to produce the final image. Our approach requires no differentiable meshing or rasterization steps, making it efficient for large 3D models and applicable to isosurfaces extracted from implicit surface definitions. We demonstrate the effectiveness of our method…
KAUST（阿卜杜拉国王科技大学）；University of Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）
In this paper, we present Neural Adaptive Tomography (NeAT), the first adaptive, hierarchical neural rendering pipeline for multi-view inverse rendering. Through a combination of neural features with an adaptive explicit representation, we achieve reconstruction times far superior to existing neural inverse rendering methods. The adaptive explicit representation improves efficiency by facilitating empty space culling and concentrating samples in complex regions, while the neural features act as a neural regularizer for the 3D reconstruction. The NeAT framework is designed specifically for the …
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Google Research（谷歌研究院）
We introduce a novel framework for solving inverse problems using NeRF-style generative models. We are interested in the problem of 3-D scene reconstruction given a single 2-D image and known camera parameters. We show that naively optimizing the latent space leads to artifacts and poor novel view rendering. We attribute this problem to volume obstructions that are clear in the 3-D geometry and become visible in the renderings of novel views. We propose a novel radiance field regularization method to obtain better 3-D surfaces and improved novel views given single view observations. Our method…
We show how to train a fully convolutional neural network to perform inverse rendering from a single, uncontrolled image. The network takes an RGB image as input, regresses albedo and normal maps from which we compute lighting coefficients. Our network is trained using large uncontrolled image collections without ground truth. By incorporating a differentiable renderer, our network can learn from self-supervision. Since the problem is ill-posed we introduce additional supervision: 1. We learn a statistical natural illumination prior, 2. Our key insight is to perform offline multiview stereo (M…
Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Rendering
University of California, Irvine（加州大学伊维特分校）；Stony Brook University（石溪大学）；Huazhong University of Science and Technology（华中科技大学）；University of Florida（佛罗里达大学）
We propose a neural inverse rendering pipeline called IRON that operates on photometric images and outputs high-quality 3D content in the format of triangle meshes and material textures readily deployable in existing graphics pipelines. Our method adopts neural representations for geometry as signed distance fields (SDFs) and materials during optimization to enjoy their flexibility and compactness, and features a hybrid optimization scheme for neural SDFs: first, optimize using a volumetric radiance field approach to recover correct topology, then optimize further using edgeaware physics-based…
