Camera-based 3D semantic scene completion (SSC) provides dense geometric and semantic perception for autonomous driving and robotic navigation. However, existing methods rely on a coupled encoder to deliver both semantic and geometric priors, which forces the model to make a trade-off between conflicting demands and limits its overall performance. To tackle these challenges, we propose FoundationSSC, a novel framework that performs dual decoupling at both the source and pathway levels. At the source level, we introduce a foundation encoder that provides rich semantic feature priors for the semantic branch and high-fidelity stereo cost volumes for the geometric branch. At the pathway level, these priors are refined through specialised, decoupled pathways, yielding superior semantic context and depth distributions. Our dual-decoupling design produces disentangled and refined inputs, which are then utilised by a hybrid view transformation to generate complementary 3D features. Additionally, we introduce a novel Axis-Aware Fusion (AAF) module that addresses the often-overlooked challenge of fusing these features by anisotropically merging them into a unified representation. Extensive experiments demonstrate the advantages of FoundationSSC, achieving simultaneous improvements in both semantic and geometric metrics, surpassing prior bests by +0.23 mIoU and +2.03 IoU on SemanticKITTI. Additionally, we achieve state-of-the-art performance on SSCBench-KITTI-360, with 21.78 mIoU and 48.61 IoU.
MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigation
Fujian Key Laboratory of Urban Intelligent Sensing and Computing（福建智能感知与计算重点实验室）；Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China（多媒体可信感知与高效计算重点实验室）；Beihang University（北航）；Nanyang Technological University（南洋理工大学）；University of Chinese Academy of Sciences（中国科学院大学）；Zhongguancun Academy（中关村学院）
Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学智能产业研究院（AIR））；University of Chinese Academy of Sciences（中国科学院大学）
Scene completion refers to obtaining dense scene representation from an incomplete perception of complex 3D scenes. This helps robots detect multi-scale obstacles and analyse object occlusions in scenarios such as autonomous driving. Recent advances show that implicit representation learning can be leveraged for continuous scene completion and achieved through physical constraints like Eikonal equations. However, former Eikonal completion methods only demonstrate results on watertight meshes at a scale of tens of meshes. None of them are successfully done for non-watertight LiDAR point clouds …
Learning Temporal 3D Semantic Scene Completion via Optical Flow Guidance
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机科学与电子工程学院）；State Key Laboratory of Advanced Design and Manufacturing Technology for Vehicle, Hunan University（车辆先进设计与制造技术国家重点实验室）
VOIC: Visible-Occluded Integrated Guidance for 3D Semantic Scene Completion
Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology（可信自主系统研究院，南方科技大学）；Department of Computer Science and Engineering, Southern University of Science and Technology（计算机科学与工程系，南方科技大学）；School of Computer Science, University of Nottingham Ningbo China（宁波大学计算机学院）；Department of Electronic and Information Engineering, Changchun University（电子与信息工程学院，长春大学）
Bridging Stereo Geometry and BEV Representation with Reliable Mutual Interaction for Semantic Scene Completion
Shanghai Jiao Tong University（上海交通大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（宁波数字孪生研究院（东方理工大学））；Tokyo Institute of Technology（东京工业大学）；PhiGent Robotics（PhiGent机器人公司）；Chinese Academy of Sciences（中国科学院）
This paper focuses on visual semantic navigation, the task of producing actions for an active agent to navigate to a specified target object category in an unknown environment. To complete this task, the algorithm should simultaneously locate and navigate to an instance of the category. In comparison to the traditional point goal navigation, this task requires the agent to have a stronger contextual prior to indoor environments. We introduce SSCNav, an algorithm that explicitly models scene priors using a confidence-aware semantic scene completion module to complete the scene and guide the age…
3D Semantic Scene Completion (SSC) can provide dense geometric and semantic scene representations, which can be applied in the field of autonomous driving and robotic systems. It is challenging to estimate the complete geometry and semantics of a scene solely from visual images, and accurate depth information is crucial for restoring 3D geometry. In this paper, we propose the first stereo SSC method named OccDepth, which fully exploits implicit depth information from stereo images (or RGBD images) to help the recovery of 3D geometric structures. The Stereo Soft Feature Assignment (Stereo-SFA) …
CurriFlow: Curriculum-Guided Depth Fusion with Optical Flow-Based Temporal Alignment for 3D Semantic Scene Completion
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Ministry of Education（教育部）；Shandong Computer Science Center (National Supercomputer Center in Jinan)（山东省计算机科学中心（济南国家超算中心））；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing（山东省计算电力互联网与服务计算重点实验室）；Shandong Fundamental Research Center for Computer Science（山东省计算机科学基础研究中心）
One Step Closer: Creating the Future to Boost Monocular Semantic Scene Completion
School of Computer Science and Technology（计算机科学与技术学院）；Xi'an Jiaotong University（西安交通大学）；The Institute of Computing（计算研究所）；China Academy of Railway Sciences Corporation（中国铁路科学研究院）
Unleashing Network Potentials for Semantic Scene Completion
School of Computer Science and Engineering, Nanjing University of Science & Technology（南京理工大学计算机科学与工程学院）；Singapore Management University（新加坡管理大学）；The Hong Kong University of Science & Technology（香港科技大学）
College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
Zhejiang University（浙江大学）；Hunan University（湖南大学）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Shanghai SUPREMIND Technology Company Ltd.（上海思普睿明科技有限公司）
Beijing Advanced Innovation Center for Intelligent Robots and Systems（北京智能机器人与系统高精尖创新中心）；PEK-SenseTime Joint Laboratory of Machine Vision（北京大学-商汤科技机器视觉联合实验室）
Semantic Scene Completion (SSC) aims to simultaneously predict the volumetric occupancy and semantic category of a 3D scene. In this paper, we propose a real-time semantic scene completion method with a feature aggregation strategy and conditioned prediction module. Feature aggregation fuses feature with different receptive fields and gathers context to improve scene completion performance. And the conditioned prediction module adopts a two-step prediction scheme that takes volumetric occupancy as a condition to enhance semantic completion prediction. We conduct experiments on three recognized…
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）；Samsung Research Institute China - Beijing（三星中国研究院（北京））；Xidian University（西安电子科技大学）
Semantic Scene Completion aims at reconstructing a complete 3D scene with precise voxel-wise semantics from a single-view depth or RGBD image. It is a crucial but challenging problem for indoor scene understanding. In this work, we present a novel framework named Scene-Instance-Scene Network (\textit{SISNet}), which takes advantages of both instance and scene level semantic information. Our method is capable of inferring fine-grained shape details as well as nearby objects whose semantic categories are easily mixed-up. The key insight is that we decouple the instances from a coarsely completed…
Scene Completion is the task of completing missing geometry from a partial scan of a scene. Most previous methods compute an implicit representation from range data using a Truncated Signed Distance Function (T-SDF) computed on a 3D grid as input to neural networks. The truncation decreases but does not remove the border errors introduced by the sign of SDF for open surfaces. As an alternative, we present an Unsigned Distance Function (UDF) as an input representation to scene completion neural networks. The proposed UDF is simple, and efficient as a geometry representation, and can be computed…
This paper focuses on semantic scene completion, a task for producing a complete 3D voxel representation of volumetric occupancy and semantic labels for a scene from a single-view depth map observation. Previous work has considered scene completion and semantic labeling of depth maps separately. However, we observe that these two problems are tightly intertwined. To leverage the coupled nature of these two tasks, we introduce the semantic scene completion network (SSCNet), an end-to-end 3D convolutional network that takes a single depth image as input and simultaneously outputs occupancy and s…
Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval
School of Cyber Science and Engineering, Nanjing University of Science and Technology（南京理工大学网络空间安全学院）；State Key Laboratory of Media Convergence and Communication, Communication University of China（中国传媒大学媒体融合与传播国家重点实验室）；Lenovo Research（联想研究院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Laboratory for Advanced Computing and Intelligence Engineering（先进计算与智能工程实验室）；College of Computer Science, Nankai University（南开大学计算机学院）
Universidade de Brasília（巴西利亚大学）；University of Surrey（萨里大学）
Semantic scene completion is the task of producing a complete 3D voxel representation of volumetric occupancy with semantic labels for a scene from a single-view observation. We built upon the recent work of Song et al. (CVPR 2017), who proposed SSCnet, a method that performs scene completion and semantic labelling in a single end-to-end 3D convolutional network. SSCnet uses only depth maps as input, even though depth maps are usually obtained from devices that also capture colour information, such as RGBD sensors and stereo cameras. In this work, we investigate the potential of the RGB colour…
Institute of Cyber-Systems and Control, Zhejiang University（浙江大学控制科学与工程学院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Outdoor scene completion is a challenging issue in 3D scene understanding, which plays an important role in intelligent robotics and autonomous driving. Due to the sparsity of LiDAR acquisition, it is far more complex for 3D scene completion and semantic segmentation. Since semantic features can provide constraints and semantic priors for completion tasks, the relationship between them is worth exploring. Therefore, we propose an end-to-end semantic segmentation-assisted scene completion network, including a 2D completion branch and a 3D semantic segmentation branch. Specifically, the network …
ShanghaiTech University（上海科技大学）；Chinese University of Hong Kong（香港中文大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉成像工程研究中心）
Real scans always miss partial geometries of objects due to the self-occlusions, external-occlusions, and limited sensor resolutions. Point cloud completion aims to refer the complete shapes for incomplete 3D scans of objects. Current deep learning-based approaches rely on large-scale complete shapes in the training process, which are usually obtained from synthetic datasets. It is not applicable for real-world scans due to the domain gap. In this paper, we propose a self-supervised point cloud completion method (TraPCC) for vehicles in real traffic scenes without any complete data. Based on t…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；UT Austin（德克萨斯大学奥斯汀分校）；Phillips Academy Andover（安多弗菲利普斯学院）；Snap Research（Snap研究）；Zhejiang University（浙江大学）；Facebook AI Research（Facebook人工智能研究院）
Estimating the relative rigid pose between two RGB-D scans of the same underlying environment is a fundamental problem in computer vision, robotics, and computer graphics. Most existing approaches allow only limited maximum relative pose changes since they require considerable overlap between the input scans. We introduce a novel deep neural network that extends the scope to extreme relative poses, with little or even no overlap between the input scans. The key idea is to infer more complete scene information about the underlying environment and match on the completed scans. In particular, ins…
University of Brasilia（巴西利亚大学）；University of Surrey（萨里大学）
Semantic scene completion is the task of predicting a complete 3D representation of volumetric occupancy with corresponding semantic labels for a scene from a single point of view. Previous works on Semantic Scene Completion from RGB-D data used either only depth or depth with colour by projecting the 2D image into the 3D volume resulting in a sparse data representation. In this work, we present a new strategy to encode colour information in 3D space using edge detection and flipped truncated signed distance. We also present EdgeNet, a new end-to-end neural network architecture capable of hand…
