A polarization camera has great potential for 3D reconstruction since the angle of polarization (AoP) of reflected light is related to an object's surface normal. In this paper, we propose a novel 3D reconstruction method called Polarimetric Multi-View Inverse Rendering (Polarimetric MVIR) that effectively exploits geometric, photometric, and polarimetric cues extracted from input multi-view color polarization images. We first estimate camera poses and an initial 3D model by geometric reconstruction with a standard structure-from-motion and multi-view stereo pipeline. We then refine the initial model by optimizing photometric and polarimetric rendering errors using multi-view RGB and AoP images, where we propose a novel polarimetric rendering cost function that enables us to effectively constrain each estimated surface vertex's normal while considering four possible ambiguous azimuth angles revealed from the AoP measurement. Experimental results using both synthetic and real data demonstrate that our Polarimetric MVIR can reconstruct a detailed 3D shape without assuming a specific polarized reflection depending on the material.
A polarization camera has great potential for 3D reconstruction since the angle of polarization (AoP) and the degree of polarization (DoP) of reflected light are related to an object's surface normal. In this paper, we propose a novel 3D reconstruction method called Polarimetric Multi-View Inverse Rendering (Polarimetric MVIR) that effectively exploits geometric, photometric, and polarimetric cues extracted from input multi-view color-polarization images. We first estimate camera poses and an initial 3D model by geometric reconstruction with a standard structure-from-motion and multi-view ster…
PolGS: Polarimetric Gaussian Splatting for Fast Reflective Surface Reconstruction
Beijing University of Posts and Telecommunications（北京邮电大学）；Xiong’an Aerospace Information Research Institute（雄安航空航天信息研究所）；State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（信息处理国家重点实验室，北京大学计算机学院）；National Engineering Research Center of Visual Technology, School of Computer Science, Peking University（视觉技术国家工程研究中心，北京大学计算机学院）
GAINS: Gaussian-based Inverse Rendering from Sparse Multi-View Captures
University Kaiserslautern-Landau（凯撒斯劳滕-兰道大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；College of William & Mary（威廉与玛丽学院）
Recent advances in implicit neural representations and differentiable rendering make it possible to simultaneously recover the geometry and materials of an object from multi-view RGB images captured under unknown static illumination. Despite the promising results achieved, indirect illumination is rarely modeled in previous methods, as it requires expensive recursive path tracing which makes the inverse rendering computationally intractable. In this paper, we propose a novel approach to efficiently recovering spatially-varying indirect illumination. The key insight is that indirect illuminatio…
GS-IR: 3D Gaussian Splatting for Inverse Rendering
South China University of Technology（华南理工大学）；Tencent AI Lab（腾讯人工智能实验室）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
GUS-IR: Gaussian Splatting with Unified Shading for Inverse Rendering
School of Electronic and Information Engineering, South China University of Technology（华南理工大学电子与信息工程学院）；Australian National University（澳大利亚国立大学）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；VIVO
GeoSplatting: Towards Geometry Guided Gaussian Splatting for Physically-based Inverse Rendering
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Wangxuan Institute of Computer Technology, Peking University（北京大学王轩计算机技术研究所）；State Key Laboratory of Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
The University of Hong Kong（香港大学）；Nanyang Technological University（南洋理工大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Traditional multi-view photometric stereo (MVPS) methods are often composed of multiple disjoint stages, resulting in noticeable accumulated errors. In this paper, we present a neural inverse rendering method for MVPS based on implicit representation. Given multi-view images of a non-Lambertian object illuminated by multiple unknown directional lights, our method jointly estimates the geometry, materials, and lights. Our method first employs multi-light images to estimate per-view surface normal maps, which are used to regularize the normals derived from the neural radiance field. It then join…
In this work, we propose a novel method for the detailed reconstruction of transparent objects by exploiting polarimetric cues. Most of the existing methods usually lack sufficient constraints and suffer from the over-smooth problem. Hence, we introduce polarization information as a complementary cue. We implicitly represent the object's geometry as a neural network, while the polarization render is capable of rendering the object's polarization images from the given shape and illumination configuration. Direct comparison of the rendered polarization images to the real-world captured images wi…
Indoor scenes typically exhibit complex, spatially-varying appearance from global illumination, making inverse rendering a challenging ill-posed problem. This work presents an end-to-end, learning-based inverse rendering framework incorporating differentiable Monte Carlo raytracing with importance sampling. The framework takes a single image as input to jointly recover the underlying geometry, spatially-varying lighting, and photorealistic materials. Specifically, we introduce a physically-based differentiable rendering layer with screen-space ray tracing, resulting in more realistic specular …
Holistic Inverse Rendering of Complex Facade via Aerial 3D Scanning
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Zhejianglab（之江实验室）；Zhejiang University（浙江大学）；Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
FIRe: Fast Inverse Rendering using Directional and Signed Distance Functions
TU Munich（慕尼黑工业大学）；MCML（慕尼黑计算数学与机器学习实验室）；Massachusetts Institute of Technology（麻省理工学院）；University of Bonn（波恩大学）；MPI Informatics（马克斯·普朗克信息学研究所）；SIC（萨尔信息中心）
In this paper a semi-supervised deep framework is proposed for the problem of 3D shape inverse rendering from a single 2D input image. The main structure of proposed framework consists of unsupervised pre-trained components which significantly reduce the need to labeled data for training the whole framework. using labeled data has the advantage of achieving to accurate results without the need to predefined assumptions about image formation process. Three main components are used in the proposed network: an encoder which maps 2D input image to a representation space, a 3D decoder which decodes…
TransparentGS: Fast Inverse Rendering of Transparent Objects with Gaussians
State Key Lab for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；TMCC, College of Computer Science, Nankai University（天津大学计算机学院）；State Key Lab of CAD & CG, Zhejiang University（浙江大学计算机辅助设计与图形学国家重点实验室）；Institute of Hangzhou Holographic Intelligent Technology（杭州全息智能技术研究院）
We show how to train a fully convolutional neural network to perform inverse rendering from a single, uncontrolled image. The network takes an RGB image as input, regresses albedo and normal maps from which we compute lighting coefficients. Our network is trained using large uncontrolled image collections without ground truth. By incorporating a differentiable renderer, our network can learn from self-supervision. Since the problem is ill-posed we introduce additional supervision: 1. We learn a statistical natural illumination prior, 2. Our key insight is to perform offline multiview stereo (M…
Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Rendering
University of California, Irvine（加州大学伊维特分校）；Stony Brook University（石溪大学）；Huazhong University of Science and Technology（华中科技大学）；University of Florida（佛罗里达大学）
We propose a neural inverse rendering pipeline called IRON that operates on photometric images and outputs high-quality 3D content in the format of triangle meshes and material textures readily deployable in existing graphics pipelines. Our method adopts neural representations for geometry as signed distance fields (SDFs) and materials during optimization to enjoy their flexibility and compactness, and features a hybrid optimization scheme for neural SDFs: first, optimize using a volumetric radiance field approach to recover correct topology, then optimize further using edgeaware physics-based…
University of California, San Diego（加州大学圣地亚哥分校）；Adobe Research（Adobe研究院）
We propose a deep inverse rendering framework for indoor scenes. From a single RGB image of an arbitrary indoor scene, we create a complete scene reconstruction, estimating shape, spatially-varying lighting, and spatially-varying, non-Lambertian surface reflectance. To train this network, we augment the SUNCG indoor scene dataset with real-world materials and render them with a fast, high-quality, physically-based GPU renderer to create a large-scale, photorealistic indoor dataset. Our inverse rendering network incorporates physical insights -- including a spatially-varying spherical Gaussian …
