Inverse rendering of an object under entirely unknown capture conditions is a fundamental challenge in computer vision and graphics. Neural approaches such as NeRF have achieved photorealistic results on novel view synthesis, but they require known camera poses. Solving this problem with unknown camera poses is highly challenging as it requires joint optimization over shape, radiance, and pose. This problem is exacerbated when the input images are captured in the wild with varying backgrounds and illuminations. Standard pose estimation techniques fail in such image collections in the wild due to very few estimated correspondences across images. Furthermore, NeRF cannot relight a scene under any illumination, as it operates on radiance (the product of reflectance and illumination). We propose a joint optimization framework to estimate the shape, BRDF, and per-image camera pose and illumination. Our method works on in-the-wild online image collections of an object and produces relightable 3D assets for several use-cases such as AR/VR. To our knowledge, our method is the first to tackle this severely unconstrained task with minimal user interaction. Project page: https://markboss.me/publication/2022-samurai/ Video: https://youtu.be/LlYuGDjXp-8
We present a learning-based method for synthesizing novel views of complex scenes using only unstructured collections of in-the-wild photographs. We build on Neural Radiance Fields (NeRF), which uses the weights of a multilayer perceptron to model the density and color of a scene as a function of 3D coordinates. While NeRF works well on images of static subjects captured under controlled settings, it is incapable of modeling many ubiquitous, real-world phenomena in uncontrolled images, such as variable illumination or transient occluders. We introduce a series of extensions to NeRF to address …
Seg-Wild: Interactive Segmentation based on 3D Gaussian Splatting for Unconstrained Image Collections
College of Computer Science and Engineering, Shandong University of Science and Technology（山东科技大学计算机科学与工程学院）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）
提出基于3D Gaussian Splatting的交互式分割方法Seg-Wild，通过多维特征嵌入和Spiky 3D Gaussian Cutter处理无约束图像中的瞬态遮挡，实现更优的野外场景分割与重建质量。
Taking Language Embedded 3D Gaussian Splatting into the Wild
State Key Laboratory of Virtual Reality Technology and Systems（虚拟现实技术与系统国家重点实验室）；School of Computer Science and Engineering（计算机科学与工程学院）；Beihang University（北航大学）
Cross-Ray Neural Radiance Fields for Novel-view Synthesis from Unconstrained Image Collections
South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）；Guangzhou Shiyuan Electronics Co., Ltd（广州视源电子股份有限公司）；Key Laboratory of Big Data and Intelligent Robot, Ministry of Education（教育部大数据与智能机器人重点实验室）
Micro-macro Wavelet-based Gaussian Splatting for 3D Reconstruction from Unconstrained Images
State Key Laboratory of Complex and Critical Software Environment（复杂与关键软件环境国家重点实验室）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Views
University of Science and Technology of China（科学技术大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Brown University（布朗大学）；Shanghai Jiao Tong University（交通大学）；The University of Hong Kong（香港大学）
Recent work has shown the possibility of training generative models of 3D content from 2D image collections on small datasets corresponding to a single object class, such as human faces, animal faces, or cars. However, these models struggle on larger, more complex datasets. To model diverse and unconstrained image collections such as ImageNet, we present VQ3D, which introduces a NeRF-based decoder into a two-stage vector-quantized autoencoder. Our Stage 1 allows for the reconstruction of an input image and the ability to change the camera position around the image, and our Stage 2 allows for t…
Tsinghua University（清华大学）；USTC（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）
This paper deals with the challenging task of synthesizing novel views for in-the-wild photographs. Existing methods have shown promising results leveraging monocular depth estimation and color inpainting with layered depth representations. However, these methods still have limited capability to handle scenes with complex 3D geometry. We propose a new method based on the multiplane image (MPI) representation. To accommodate diverse scene layouts in the wild and tackle the difficulty in producing high-dimensional MPI contents, we design a network structure that consists of two novel modules, on…
University of Tübingen（图宾根大学）；Google Research（谷歌研究院）
Decomposing a scene into its shape, reflectance, and illumination is a challenging but important problem in computer vision and graphics. This problem is inherently more challenging when the illumination is not a single light source under laboratory conditions but is instead an unconstrained environmental illumination. Though recent work has shown that implicit representations can be used to model the radiance field of an object, most of these techniques only enable view synthesis and not relighting. Additionally, evaluating these radiance fields is resource and time-intensive. We propose a ne…
Toyota Research Institute（丰田研究院）；Massachusetts Institute of Technology（麻省理工学院）
Understanding where people are looking is an informative social cue. In this work, we present Gaze360, a large-scale gaze-tracking dataset and method for robust 3D gaze estimation in unconstrained images. Our dataset consists of 238 subjects in indoor and outdoor environments with labelled 3D gaze across a wide range of head poses and distances. It is the largest publicly available dataset of its kind by both subject and variety, made possible by a simple and efficient collection method. Our proposed 3D gaze model extends existing models to include temporal information and to directly output a…
Tencent Youtu Lab（腾讯优图实验室）；Nanjing University of Science and Technology（南京理工大学）
Non-parametric face modeling aims to reconstruct 3D face only from images without shape assumptions. While plausible facial details are predicted, the models tend to over-depend on local color appearance and suffer from ambiguous noise. To address such problem, this paper presents a novel Learning to Aggregate and Personalize (LAP) framework for unsupervised robust 3D face modeling. Instead of using controlled environment, the proposed method implicitly disentangles ID-consistent and scene-specific face from unconstrained photo set. Specifically, to learn ID-consistent face, LAP adaptively agg…
Semi-Supervised Unconstrained Head Pose Estimation in the Wild
School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Chongqing Academy of Science and Technology（重庆科学研究院）；Lenovo Research and Technology（联想研究与技术）；Department of Computer Science and Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）
University Paris-Est Créteil（巴黎东克雷泰伊大学）；University Paris City（巴黎城市大学）；Istanbul Technical University（伊斯坦布尔理工大学）；National Institute of Technology Warangal（瓦朗加尔国家技术学院）；The National Engineering School of Sfax（斯法克斯国家工程师学院）；French National Centre for Scientific Research（法国国家科学研究中心）
POST-traumatic stress disorder (PTSD) is a chronic and debilitating mental condition that is developed in response to catastrophic life events, such as military combat, sexual assault, and natural disasters. PTSD is characterized by flashbacks of past traumatic events, intrusive thoughts, nightmares, hypervigilance, and sleep disturbance, all of which affect a person's life and lead to considerable social, occupational, and interpersonal dysfunction. The diagnosis of PTSD is done by medical professionals using self-assessment questionnaire of PTSD symptoms as defined in the Diagnostic and Stat…
Computer Vision Group, University of Bonn（波恩大学计算机视觉组）；National University of Science and Technology（国立科技大学）；Gokhale Method Institute（戈卡莱方法研究所）；Multimedia, Simulation, Virtual Reality Group, University of Bonn（波恩大学多媒体、仿真与虚拟现实组）
In this work we address the challenging problem of 3D human pose estimation from single images. Recent approaches learn deep neural networks to regress 3D pose directly from images. One major challenge for such methods, however, is the collection of training data. Specifically, collecting large amounts of training data containing unconstrained images annotated with accurate 3D poses is infeasible. We therefore propose to use two independent training sources. The first source consists of accurate 3D motion capture data, and the second source consists of unconstrained images with annotated 2D po…
University of Bologna（博洛尼亚大学）；Google Inc.（谷歌公司）；Technische Universität München（慕尼黑工业大学）
Novel view synthesis from a single image has recently achieved remarkable results, although the requirement of some form of 3D, pose, or multi-view supervision at training time limits the deployment in real scenarios. This work aims at relaxing these assumptions enabling training of conditional generative models for novel view synthesis in a completely unsupervised manner. We first pre-train a purely generative decoder model using a 3D-aware GAN formulation while at the same time train an encoder network to invert the mapping from latent space to images. Then, we swap encoder and decoder and t…
