Accurate 3D scene representation and panoptic understanding are essential for applications such as virtual reality, robotics, and autonomous driving. However, challenges persist with existing methods, including precise 2D-to-3D mapping, handling complex scene characteristics like boundary ambiguity and varying scales, and mitigating noise in panoptic pseudo-labels. This paper introduces a novel perceptual-prior-guided 3D scene representation and panoptic understanding method, which reformulates panoptic understanding within neural radiance fields as a linear assignment problem involving 2D semantics and instance recognition. Perceptual information from pre-trained 2D panoptic segmentation models is incorporated as prior guidance, thereby synchronizing the learning processes of appearance, geometry, and panoptic understanding within neural radiance fields. An implicit scene representation and understanding model is developed to enhance generalization across indoor and outdoor scenes by extending the scale-encoded cascaded grids within a reparameterized domain distillation framework. This model effectively manages complex scene attributes and generates 3D-consistent scene representations and panoptic understanding outcomes for various scenes. Experiments and ablation studies under challenging conditions, including synthetic and real-world scenes, demonstrate the proposed method's effectiveness in enhancing 3D scene representation and panoptic segmentation accuracy.
Google Research（谷歌研究院）；Stanford University（斯坦福大学）；University of California, San Diego（加州大学圣地亚哥分校）
We address efficient and structure-aware 3D scene representation from images. Nerflets are our key contribution -- a set of local neural radiance fields that together represent a scene. Each nerflet maintains its own spatial position, orientation, and extent, within which it contributes to panoptic, density, and radiance reconstructions. By leveraging only photometric and inferred panoptic image supervision, we can directly and jointly optimize the parameters of a set of nerflets so as to form a decomposed representation of the scene, where each object instance is represented by a group of ner…
Google Research（谷歌研究院）；Stanford University（斯坦福大学）；Simon Fraser University（西蒙弗雷泽大学）；Georgia Tech（佐治亚理工学院）
We present Panoptic Neural Fields (PNF), an object-aware neural scene representation that decomposes a scene into a set of objects (things) and background (stuff). Each object is represented by an oriented 3D bounding box and a multi-layer perceptron (MLP) that takes position, direction, and time and outputs density and radiance. The background stuff is represented by a similar MLP that additionally outputs semantic labels. Each object MLPs are instance-specific and thus can be smaller and faster than previous object-aware approaches, while still leveraging category-specific priors incorporate…
Hamilton Institute Maynooth University（汉密尔顿研究所梅努斯大学）；Valeo Vision Systems（法雷奥视觉系统公司）；Maynooth University（梅努斯大学）
Generating a detailed near-field perceptual model of the environment is an important and challenging problem in both self-driving vehicles and autonomous mobile robotics. A Bird Eye View (BEV) map, providing a panoptic representation, is a commonly used approach that provides a simplified 2D representation of the vehicle surroundings with accurate semantic level segmentation for many downstream tasks. Current state-of-the art approaches to generate BEV-maps employ a Convolutional Neural Network (CNN) backbone to create feature-maps which are passed through a spatial transformer to project the …
EPRecon: An Efficient Framework for Real-Time Panoptic 3D Reconstruction from Monocular Video
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Samsung AI Centre Toronto（多伦多三星人工智能中心）；University of Toronto（多伦多大学）；York University（约克大学）；Vector Institute for AI（向量人工智能研究所）
Neural Radiance Fields (NeRFs) have emerged as a popular approach for novel view synthesis. While NeRFs are quickly being adapted for a wider set of applications, intuitively editing NeRF scenes is still an open challenge. One important editing task is the removal of unwanted objects from a 3D scene, such that the replaced region is visually plausible and consistent with its context. We refer to this task as 3D inpainting. In 3D, solutions must be both consistent across multiple views and geometrically valid. In this paper, we propose a novel 3D inpainting method that addresses these challenge…
Dalian University of Technology（大连理工大学）；ZMO AI Inc.（ZMO人工智能公司）
In this work, we focus on synthesizing high-fidelity novel view images for arbitrary human performers, given a set of sparse multi-view images. It is a challenging task due to the large variation among articulated body poses and heavy self-occlusions. To alleviate this, we introduce an effective generalizable framework Generalizable Model-based Neural Radiance Fields (GM-NeRF) to synthesize free-viewpoint images. Specifically, we propose a geometry-guided attention mechanism to register the appearance code from multi-view 2D images to a geometry proxy which can alleviate the misalignment betwe…
PLGS: Robust Panoptic Lifting with 3D Gaussian Splatting
School of Automation Science and Electrical Engineering, Beihang University（北京航空航天大学自动化科学与电气工程学院）；Institute of Software, Chinese Academy of Sciences & University of Chinese Academy of Sciences（中国科学院软件研究所）；Peking University（北京大学）
Humans have the remarkable ability to perceive objects as a whole, even when parts of them are occluded. This ability of amodal perception forms the basis of our perceptual and cognitive understanding of our world. To enable robots to reason with this capability, we formulate and propose a novel task that we name amodal panoptic segmentation. The goal of this task is to simultaneously predict the pixel-wise semantic segmentation labels of the visible regions of stuff classes and the instance segmentation labels of both the visible and occluded regions of thing classes. To facilitate research o…
The Chinese University of Hong Kong（香港中文大学）；SmartMore（思谋科技）
Current 3D scene stylization methods transfer textures and colors as styles using arbitrary style references, lacking meaningful semantic correspondences. We introduce Reference-Based Non-Photorealistic Radiance Fields (Ref-NPR) to address this limitation. This controllable method stylizes a 3D scene using radiance fields with a single stylized 2D view as a reference. We propose a ray registration process based on the stylized reference view to obtain pseudo-ray supervision in novel views. Then we exploit semantic correspondences in content images to fill occluded regions with perceptually sim…
Korea Institute of Science and Technology (KIST)（韩国科学技术院）；Yonsei University（延世大学）
Neural Radiance Field (NeRF) has exhibited outstanding three-dimensional (3D) reconstruction quality via the novel view synthesis from multi-view images and paired calibrated camera parameters. However, previous NeRF-based systems have been demonstrated under strictly controlled settings, with little attention paid to less ideal scenarios, including with the presence of noise such as exposure, illumination changes, and blur. In particular, though blur frequently occurs in real situations, NeRF that can handle blurred images has received little attention. The few studies that have investigated …
Neural Radiance Fields (NeRFs), despite their outstanding performance on novel view synthesis, often need dense input views. Many papers train one model for each scene respectively and few of them explore incorporating multi-modal data into this problem. In this paper, we focus on a rarely discussed but important setting: can we train one model that can represent multiple scenes, with 360$^\circ $ insufficient views and RGB-D images? We refer insufficient views to few extremely sparse and almost non-overlapping views. To deal with it, X-NeRF, a fully explicit approach which learns a general sc…
This paper proposes Sparse View Synthesis. This is a view synthesis problem where the number of reference views is limited, and the baseline between target and reference view is significant. Under these conditions, current radiance field methods fail catastrophically due to inescapable artifacts such 3D floating blobs, blurring and structural duplication, whenever the number of reference views is limited, or the target view diverges significantly from the reference views. Advances in network architecture and loss regularisation are unable to satisfactorily remove these artifacts. The occlusion…
Triangle Splatting for Real-Time Radiance Field Rendering
University of Liège（列日大学）；KAUST（阿卜杜拉国王科技大学）；University of Oxford（牛津大学）；Simon Fraser University（西蒙弗雷泽大学）；University of Toronto（多伦多大学）；Google DeepMind（谷歌DeepMind）
Shanghai Jiao Tong University（上海交通大学）；New York University（纽约大学）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）；Peng Cheng Laboratory（鹏城实验室）
Virtual Reality (VR) is becoming ubiquitous with the rise of consumer displays and commercial VR platforms. Such displays require low latency and high quality rendering of synthetic imagery with reduced compute overheads. Recent advances in neural rendering showed promise of unlocking new possibilities in 3D computer graphics via image-based representations of virtual or physical environments. Specifically, the neural radiance fields (NeRF) demonstrated that photo-realistic quality and continuous view changes of 3D scenes can be achieved without loss of view-dependent effects. While NeRF can s…
RePaintGS: Reference-Guided Gaussian Splatting for Realistic and View-Consistent 3D Scene Inpainting
Intelligence and Interaction Research Center, Korea Institute of Science and Technology（智能交互研究所，韩国科学技术院）；Department of Computer Science and Engineering, Korea University（韩国大学计算机科学与工程系）；AI-Robotics, KIST School, Korea National University of Science and Technology（人工智能机器人，KIST学院，韩国科学技术大学）
ASSIST: Interactive Scene Nodes for Scalable and Realistic Indoor Simulation
Institute for AI Industry Research (AIR), Tsinghua University（清华大学智能产业研究院（AIR））；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Beijing University of Civil Engineering and Architecture（北京建筑大学）；University of Science and Technology of China（中国科学技术大学）；McGill University（麦吉尔大学）；Beijing Institute（北京研究所）
Recently, differentiable volume rendering in neural radiance fields (NeRF) has gained a lot of popularity, and its variants have attained many impressive results. However, existing methods usually assume the scene is a homogeneous volume so that a ray is cast along the straight path. In this work, the scene is instead a heterogeneous volume with a piecewise-constant refractive index, where the path will be curved if it intersects the different refractive indices. For novel view synthesis of refractive objects, our NeRF-based framework aims to optimize the radiance fields of bounded volume and …
We extend neural radiance fields (NeRFs) to dynamic large-scale urban scenes. Prior work tends to reconstruct single video clips of short durations (up to 10 seconds). Two reasons are that such methods (a) tend to scale linearly with the number of moving objects and input videos because a separate model is built for each and (b) tend to require supervision via 3D bounding boxes and panoptic labels, obtained manually or via category-specific models. As a step towards truly open-world reconstructions of dynamic cities, we introduce two key innovations: (a) we factorize the scene into three separ…
Supercharging Thermal Gaussian Splatting with Depth Estimation
Technical University of Munich（技术大学慕尼黑）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与人工智能研究所）；Human-Centered Computing and Extended Reality Lab（以人为本计算与扩展现实实验室）；TUM University Hospital（技术大学慕尼黑医院）
JADE-GS: Joint Allocation of Deblurring Evidence for Event-Assisted 3D Gaussian Splatting
School of Mechatronic Engineering and Automation, Shanghai University（上海大学机电工程与自动化学院）；Shanghai Baoshan Shangda General Intelligent Robotics Research Institute（上海宝山尚大通用智能机器人研究院）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）
Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Peng Cheng Laboratory（鹏城实验室）；Georgia Institute of Technology（佐治亚理工学院）；Picsart AI Research (PAIR)（Picsart人工智能研究院）
