Open-vocabulary panoptic reconstruction is essential for advanced robotics perception and simulation. However, existing methods based on 3D Gaussian Splatting (3DGS) often struggle to simultaneously achieve geometric accuracy, coherent panoptic understanding, and real-time inference frequency in large-scale scenes. In this paper, we propose a comprehensive framework that integrates geometric reinforcement, end-to-end panoptic learning, and efficient rendering. First, to ensure physical realism in large-scale environments, we leverage LiDAR data to construct plane-constrained multimodal Gaussian Mixture Models (GMMs) and employ 2D Gaussian surfels as the map representation, enabling high-precision surface alignment and continuous geometric supervision. Building upon this, to overcome the error accumulation and cumbersome cross-frame association inherent in traditional multi-stage panoptic segmentation pipelines, we design a query-guided end-to-end learning architecture. By utilizing a local cross-attention mechanism within the view frustum, the system lifts 2D mask features directly into 3D space, achieving globally consistent panoptic understanding. Finally, addressing the computational bottlenecks caused by high-dimensional semantic features, we introduce Precise Tile Intersection and a Top-K Hard Selection strategy to optimize the rendering pipeline. Experimental results demonstrate that our system achieves superior geometric and panoptic reconstruction quality in large-scale scenes while maintaining an inference rate exceeding 40 FPS, meeting the real-time requirements of robotic control loops.
Mapping the Unseen: Unified Promptable Panoptic Mapping with Dynamic Labeling using Foundation Models
1 Applied AI Institute, Moscow 121205 , Russia;(R.S.) 2 School of Computer Science, University of Lincoln, Lincoln LN6 7TS , UK 3 Independent Researcher, Dubai 500001, United Arab Emirates
PanoSLAM: Panoptic 3D Scene Reconstruction via Gaussian SLAM
The University of Sydney(悉尼大学);The University of Hong Kong(香港大学);ShanghaiTech University(上海科技大学);The University of Melbourne(墨尔本大学);Texas A&M University(德克萨斯农工大学)
RAZER: Robust Accelerated Zero-Shot 3D Open-Vocabulary Panoptic Reconstruction with Spatio-Temporal Aggregation
Control/Robotics Research Laboratory (CRRL), Department of Electrical and Computer Engineering, NYU Tandon School of Engineering(控制/机器人研究实验室(CRRL),电气与计算机工程系,纽约大学Tandon工程学院)
Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);University of Chinese Academy of Sciences(中国科学院大学);The Chinese University of Hong Kong(香港中文大学)
Deep-Learning-Aided Path Planning and Map Construction for Expediting Indoor Mapping
Hebrew University of Jerusalem(耶路撒冷希伯来大学);Technion - Israel Institute of Technology(以色列理工学院);Bar-Ilan University(巴伊兰大学);Holon Institute of Technology(霍隆理工学院)
STAMICS: Splat, Track And Map with Integrated Consistency and Semantics for Dense RGB-D SLAM
University of Science and Technology of China(中国科学技术大学);University of Science and Technology Liaoning(辽宁科技大学);Beijing Institute of Technology(北京理工大学);Beihang University(北京航空航天大学)
Towards Localizing Structural Elements: Merging Geometrical Detection with Semantic Verification in RGB-D Data
University of Luxembourg(卢森堡大学);Interdisciplinary Centre for Security, Reliability, and Trust (SnT)(安全、可靠性与跨学科中心(SnT));Institute for Advanced Studies (IAS)(高等研究院(IAS));Faculty of Science, Technology, and Medicine(科学、技术与医学院)