We present a method for composing photorealistic scenes from captured images of objects. Our work builds upon neural radiance fields (NeRFs), which implicitly model the volumetric density and directionally-emitted radiance of a scene. While NeRFs synthesize realistic pictures, they only model static scenes and are closely tied to specific imaging conditions. This property makes NeRFs hard to generalize to new scenarios, including new lighting or new arrangements of objects. Instead of learning a scene radiance field as a NeRF does, we propose to learn object-centric neural scattering functions (OSFs), a representation that models per-object light transport implicitly using a lighting- and view-dependent neural network. This enables rendering scenes even when objects or lights move, without retraining. Combined with a volumetric path tracing procedure, our framework is capable of rendering both intra- and inter-object light transport effects including occlusions, specularities, shadows, and indirect illumination. We evaluate our approach on scene composition and show that it generalizes to novel illumination conditions, producing photorealistic, physically accurate renderings of multi-object scenes.
Omni-Scene: Omni-Gaussian Representation for Ego-Centric Sparse-View Scene Reconstruction
School of Engineering, Westlake University（西湖大学工学院）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖高等研究院先进技术研究院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
OccluGaussian: Occlusion-Aware Gaussian Splatting for Large Scene Reconstruction and Rendering
Huawei Noah’s Ark Lab（华为诺亚实验室）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shenzhen Institutes of Advanced Technology（深圳先进技术研究院）；The University of Hong Kong（香港大学）；Huawei Embodied Intelligence Lab（华为具身智能实验室）
University of Tübingen（图宾根大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Snap Inc.（Snap公司）
We present a novel method for performing flexible, 3D-aware image content manipulation while enabling high-quality novel view synthesis. While NeRF-based approaches are effective for novel view synthesis, such models memorize the radiance for every point in a scene within a neural network. Since these models are scene-specific and lack a 3D scene representation, classical editing such as shape manipulation, or combining scenes is not possible. Hence, editing and combining NeRF-based scenes has not been demonstrated. With the aim of obtaining interpretable and controllable scene representations…
University of California, San Diego（加州大学圣地亚哥分校）；Adobe Research（Adobe研究院）
While NeRF has shown great success for neural reconstruction and rendering, its limited MLP capacity and long per-scene optimization times make it challenging to model large-scale indoor scenes. In contrast, classical 3D reconstruction methods can handle large-scale scenes but do not produce realistic renderings. We propose NeRFusion, a method that combines the advantages of NeRF and TSDF-based fusion techniques to achieve efficient large-scale reconstruction and photo-realistic rendering. We process the input image sequence to predict per-frame local radiance fields via direct network inferen…
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering
School of CS, Huazhong University of Science and Technology（华中科技大学计算机学院）；School of EIC, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；Huawei Inc.（华为公司）
State Key Lab of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）；Google（谷歌）；The Chinese University of Hong Kong（香港中文大学）
Implicit neural rendering techniques have shown promising results for novel view synthesis. However, existing methods usually encode the entire scene as a whole, which is generally not aware of the object identity and limits the ability to the high-level editing tasks such as moving or adding furniture. In this paper, we present a novel neural scene rendering system, which learns an object-compositional neural radiance field and produces realistic rendering with editing capability for a clustered and real-world scene. Specifically, we design a novel two-pathway architecture, in which the scene…
STDR: Spatio-Temporal Decoupling for Real-Time Dynamic Scene Rendering
Institute of Computing Technology, Chinese Academy of Sciences, ICT（中国科学院计算技术研究所）；University of Chinese Academy of Sciences, UCAS（中国科学院大学）；The University of Queensland（昆士兰大学）；University of California, Merced（加州大学梅尔塞德斯分校）
Indoor scenes typically exhibit complex, spatially-varying appearance from global illumination, making inverse rendering a challenging ill-posed problem. This work presents an end-to-end, learning-based inverse rendering framework incorporating differentiable Monte Carlo raytracing with importance sampling. The framework takes a single image as input to jointly recover the underlying geometry, spatially-varying lighting, and photorealistic materials. Specifically, we introduce a physically-based differentiable rendering layer with screen-space ray tracing, resulting in more realistic specular …
CityGaussian: Real-time High-quality Large-Scale Scene Rendering with Gaussians
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Shandong University（山东大学）；TuSimple（图森未来）；University of Science and Technology Beijing（北京科技大学）
We tackle the problem of object-centric learning on point clouds, which is crucial for high-level relational reasoning and scalable machine intelligence. In particular, we introduce a framework, SPAIR3D, to factorize a 3D point cloud into a spatial mixture model where each component corresponds to one object. To model the spatial mixture model on point clouds, we derive the Chamfer Mixture Loss, which fits naturally into our variational training pipeline. Moreover, we adopt an object-specification scheme that describes each object's location relative to its local voxel grid cell. Such a scheme…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Facebook AI Research（Facebook人工智能研究院）；National University of Singapore（新加坡国立大学）
Photo-realistic free-viewpoint rendering of real-world scenes using classical computer graphics techniques is challenging, because it requires the difficult step of capturing detailed appearance and geometry models. Recent studies have demonstrated promising results by learning scene representations that implicitly encode both geometry and appearance without 3D supervision. However, existing approaches in practice often show blurry renderings caused by the limited network capacity or the difficulty in finding accurate intersections of camera rays with the scene geometry. Synthesizing high-reso…
HUNTER: Unsupervised Human-centric 3D Detection via Transferring Knowledge from Synthetic Instances to Real Scenes
ShanghaiTech University（上海科技大学）；University of Chinese Academy of Sciences（中国科学院大学）；Innovation Academy for Microsatellites, Chinese Academy of Sciences（中国科学院微小卫星创新研究院）；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）
DASH: 4D Hash Encoding with Self-Supervised Decomposition for Real-Time Dynamic Scene Rendering
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合国家科学中心人工智能研究院）
Weather-Magician: Reconstruction and Rendering Framework for 4D Weather Synthesis In Real Time
SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, Shanghai, 200240, China（上海交通大学巴黎精英技术学院，上海交通大学，上海，200240，中国）；Department of Automation, Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai Jiao Tong University, Shanghai 200240, China（自动化系，系统控制与信息处理重点实验室，中华人民共和国教育部，上海交通大学，上海，200240，中国）
FlexDrive: Toward Trajectory Flexibility in Driving Scene Reconstruction and Rendering
Multimedia Laboratory, The Chinese University of Hong Kong（香港中文大学多媒体实验室）；Centre for Perceptual and Interactive Intelligence, Hong Kong（香港感知与交互智能中心）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beihang University（北京航空航天大学）
SPVC: Structured and Panoptic Video Fixing for Cross-Dataset Driving Scene Rendering
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Zhejiang University（浙江大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Great Wall Motor Company Limited（长城汽车股份有限公司）；Shanghai Jiao Tong University（上海交通大学）；The Hong Kong University of Science and Technology（香港科技大学）
Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Science and Technology of China（中国科学技术大学）；Brown University（布朗大学）；The Chinese University of Hong Kong（香港中文大学）；Tongji University（同济大学）
GSDF: 3DGS Meets SDF for Improved Rendering and Reconstruction
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；University of Science and Technology of China（中国科学技术大学）；Cornell University（康奈尔大学）
Zhejiang University（浙江大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Adobe Research（奥多比研究中心）；Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
Purely MLP-based neural radiance fields (NeRF-based methods) often suffer from underfitting with blurred renderings on large-scale scenes due to limited model capacity. Recent approaches propose to geographically divide the scene and adopt multiple sub-NeRFs to model each region individually, leading to linear scale-up in training costs and the number of sub-NeRFs as the scene expands. An alternative solution is to use a feature grid representation, which is computationally efficient and can naturally scale to a large scene with increased grid resolutions. However, the feature grid tends to be…
