Humans are able to form a complex mental model of the environment they move in. This mental model captures geometric and semantic aspects of the scene, describes the environment at multiple levels of abstractions (e.g., objects, rooms, buildings), includes static and dynamic entities and their relations (e.g., a person is in a room at a given time). In contrast, current robots' internal representations still provide a partial and fragmented understanding of the environment, either in the form of a sparse or dense set of geometric primitives (e.g., points, lines, planes, voxels) or as a collection of objects. This paper attempts to reduce the gap between robot and human perception by introducing a novel representation, a 3D Dynamic Scene Graph(DSG), that seamlessly captures metric and semantic aspects of a dynamic environment. A DSG is a layered graph where nodes represent spatial concepts at different levels of abstraction, and edges represent spatio-temporal relations among nodes. Our second contribution is Kimera, the first fully automatic method to build a DSG from visual-inertial data. Kimera includes state-of-the-art techniques for visual-inertial SLAM, metric-semantic 3D reconstruction, object localization, human pose and shape estimation, and scene parsing. Our third contribution is a comprehensive evaluation of Kimera in real-life datasets and photo-realistic simulations, including a newly released dataset, uHumans2, which simulates a collection of crowded indoor and outdoor scenes. Our evaluation shows that Kimera achieves state-of-the-art performance in visual-inertial SLAM, estimates an accurate 3D metric-semantic mesh model in real-time, and builds a DSG of a complex indoor environment with tens of objects and humans in minutes. Our final contribution shows how to use a DSG for real-time hierarchical semantic path-planning. The core modules in Kimera are open-source.
Massachusetts Institute of Technology（麻省理工学院）；Laboratory for Information & Decision Systems（信息与决策系统实验室）
We provide an open-source C++ library for real-time metric-semantic visual-inertial Simultaneous Localization And Mapping (SLAM). The library goes beyond existing visual and visual-inertial SLAM libraries (e.g., ORB-SLAM, VINS- Mono, OKVIS, ROVIO) by enabling mesh reconstruction and semantic labeling in 3D. Kimera is designed with modularity in mind and has four key components: a visual-inertial odometry (VIO) module for fast and accurate state estimation, a robust pose graph optimizer for global trajectory estimation, a lightweight 3D mesher module for fast mesh reconstruction, and a dense 3D…
Laboratory for Information & Decision Systems, Massachusetts Institute of Technology（麻省理工学院信息与决策系统实验室）；U.S. Army Combat Capabilities Development Command, Army Research Laboratory（美国陆军作战能力发展司令部陆军研究实验室）
This paper presents Kimera-Multi, the first multi-robot system that (i) is robust and capable of identifying and rejecting incorrect inter and intra-robot loop closures resulting from perceptual aliasing, (ii) is fully distributed and only relies on local (peer-to-peer) communication to achieve distributed localization and mapping, and (iii) builds a globally consistent metric-semantic 3D mesh model of the environment in real-time, where faces of the mesh are annotated with semantic labels. Kimera-Multi is implemented by a team of robots equipped with visual-inertial sensors. Each robot builds…
We present the first fully distributed multi-robot system for dense metric-semantic Simultaneous Localization and Mapping (SLAM). Our system, dubbed Kimera-Multi, is implemented by a team of robots equipped with visual-inertial sensors, and builds a 3D mesh model of the environment in real-time, where each face of the mesh is annotated with a semantic label (e.g., building, road, objects). In Kimera-Multi, each robot builds a local trajectory estimate and a local mesh using Kimera. Then, when two robots are within communication range, they initiate a distributed place recognition and robust po…
The diversity of SLAM benchmarks affords extensive testing of SLAM algorithms to understand their performance, individually or in relative terms. The ad-hoc creation of these benchmarks does not necessarily illuminate the particular weak points of a SLAM algorithm when performance is evaluated. In this paper, we propose to use a decision tree to identify challenging benchmark properties for state-of-the-art SLAM algorithms and important components within the SLAM pipeline regarding their ability to handle these challenges. Establishing what factors of a particular sequence lead to track failur…
Simultaneous localisation and mapping (SLAM) is the problem of autonomous robots to construct or update a map of an undetermined unstructured environment while simultaneously estimate the pose in it. The current trend towards self-driving vehicles has influenced the development of robust SLAM techniques over the last 30 years. This problem is addressed by using a standard sensor or a sensor array (Ultrasonic sensor, LIDAR, Camera, Kinect RGB-D) with sensor fusion techniques to achieve the perception step. Sensing method is determined by considering the specifications of the environment to extr…
Northeastern University（东北大学）；The Chinese University of Hong Kong（香港中文大学）；Ulster University（阿尔斯特大学）
This paper presents the first active object mapping framework for complex robotic manipulation and autonomous perception tasks. The framework is built on an object SLAM system integrated with a simultaneous multi-object pose estimation process that is optimized for robotic grasping. Aiming to reduce the observation uncertainty on target objects and increase their pose estimation accuracy, we also design an object-driven exploration strategy to guide the object mapping process, enabling autonomous mapping and high-level perception. Combining the mapping module and the exploration strategy, an a…
RootSLAM: Square Root Bundle Adjustment for Visual-Inertial SLAM
Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；National University of Rosario（罗萨里奥国立大学）；CIFASIS (CONICET-UNR)（CIFASIS（CONICET-UNR））
LST-SLAM: A Stereo Thermal SLAM System for Kilometer-Scale Dynamic Environments
PEAK-Lab, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）峰值实验室）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）
ETH Zürich（苏黎世联邦理工学院）；Massachusetts Institute of Technology（麻省理工学院）；Universidad Sergio Arboleda（塞尔吉奥·阿博莱达大学）；Pontificia Universidad Javeriana（哈维里亚那天主教大学）；University of Adelaide（阿德莱德大学）；Australian Center for Robotic Vision（澳大利亚机器人视觉中心）
Simultaneous Localization and Mapping (SLAM)consists in the concurrent construction of a model of the environment (the map), and the estimation of the state of the robot moving within it. The SLAM community has made astonishing progress over the last 30 years, enabling large-scale real-world applications, and witnessing a steady transition of this technology to industry. We survey the current state of SLAM. We start by presenting what is now the de-facto standard formulation for SLAM. We then review related work, covering a broad set of topics including robustness and scalability in long-term …
Sensor Configuration Matters: A Systematic Evaluation of Multimodal SLAM on Quadruped Robots
Machine Intelligence and Robotics Lab, Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院智能机器人实验室）；Institute for Intelligent Systems, Esslingen University of Applied Sciences（埃森堡应用科学大学智能系统研究所）；Department of Computer Science, University of Freiburg（弗赖堡大学计算机科学系）
AgriGS-SLAM: Orchard Mapping Across Seasons via Multi-View Gaussian Splatting SLAM
Dipartimento di Bioingegneria, Elettronica e Informazione, Politecnico di Milano（生物工程、电子与信息系，米兰理工大学）；Agricultural Biosystems Engineering, Wageningen University & Research（农业生物系统工程，瓦赫宁根大学与研究）
Fudan University（复旦大学）；FAW (Nanjing) Technology Development Co.,Ltd（一汽（南京）科技开发有限公司）
Automatic underground parking has attracted considerable attention as the scope of autonomous driving expands. The auto-vehicle is supposed to obtain the environmental information, track its location, and build a reliable map of the scenario. Mainstream solutions consist of well-trained neural networks and simultaneous localization and mapping (SLAM) methods, which need numerous carefully labeled images and multiple sensor estimations. However, there is a lack of underground parking scenario datasets with multiple sensors and well-labeled images that support both SLAM tasks and perception task…
School of Integrated Circuits, Tsinghua University（清华大学集成电路学院）；Tsinghua University（清华大学）
Simultaneous localization and mapping (SLAM) is a critical technology that enables autonomous robots to be aware of their surrounding environment. With the development of deep learning, SLAM systems can achieve a higher level of perception of the environment, including the semantic and text levels. However, current works are limited in their ability to achieve a natural-language level of perception of the world. To address this limitation, we propose LP-SLAM, the first language-perceptive SLAM system that leverages large language models (LLMs). LP-SLAM has two major features: (a) it can detect…
Considering the scene's dynamics is the most effective solution to obtain an accurate perception of unknown environments for real vSLAM applications. Most existing methods attempt to address the non-rigid scene assumption by combining geometric and semantic approaches to determine dynamic elements that lack generalization and scene awareness. We propose a novel approach that overcomes these limitations by using scene-depth information to improve the accuracy of the localization from geometric and semantic modules. In addition, we use depth information to determine an area of influence of dynam…
PAS-SLAM: A Visual SLAM System for Planar Ambiguous Scenes
School of Electronic Information and Electrical Engineering, Dalian University of Technology（大连理工大学电子信息与电气工程学部）；Department of Automation, Tsinghua University（清华大学自动化系）；School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）
NRGS-SLAM: Monocular Non-Rigid SLAM for Endoscopy via Deformation-Aware 3D Gaussian Splatting
Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong（机械与自动化工程系和T Stone机器人研究所，香港中文大学）；School of Automation and Intelligent Sensing, the State Key Laboratory of Avionics Integration and Aviation System of-Systems Synthesis, and Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai Jiao Tong University（自动化与智能感知学院、航空系统集成与航空系统综合国家重点实验室、上海导航与定位服务重点实验室，上海交通大学）
Collaborative SLAM enables a group of agents to simultaneously co-localize and jointly map an environment, thus paving the way to wide-ranging applications of multi-robot perception and multi-user AR experiences by eliminating the need for external infrastructure or pre-built maps. This article presents COVINS, a novel collaborative SLAM system, that enables multi-agent, scalable SLAM in large environments and for large teams of more than 10 agents. The paradigm here is that each agent runs visual-inertial odomety independently onboard in order to ensure its autonomy, while sharing map informa…
University of Electronic Science and Technology of China（电子科技大学）
Object SLAM uses additional semantic information to detect and map objects in the scene, in order to improve the system's perception and map representation capabilities. Quadrics and cubes are often used to represent objects, but their single shape limits the accuracy of object map and thus affects the application of downstream tasks. In this paper, we introduce superquadrics (SQ) with shape parameters into SLAM for representing objects, and propose a separate parameter estimation method that can accurately estimate object pose and adapt to different shapes. Furthermore, we present a lightweig…
We introduce a new real-time pipeline for Simultaneous Localization and Mapping (SLAM) and Visual Inertial Odometry (VIO) in the context of planetary rovers. We leverage prior information of the location of the lander to propose an object-level SLAM approach that optimizes pose and shape of the lander together with camera trajectories of the rover. As a further refinement step, we propose to use techniques of interpolation between adjacent temporal samples; videlicet synthesizing non-existing images to improve the overall accuracy of the system. The experiments are conducted in the context of …
ONERA（法国航空航天实验室）；Université Paris-Saclay（巴黎萨克雷大学）；University of Toulouse（图卢兹大学）
This paper introduces an Online Localisation and Colored Mesh Reconstruction (OLCMR) ROS perception architecture for ground exploration robots aiming to perform robust Simultaneous Localisation And Mapping (SLAM) in challenging unknown environments and provide an associated colored 3D mesh representation in real time. It is intended to be used by a remote human operator to easily visualise the mapped environment during or after the mission or as a development base for further researches in the field of exploration robotics. The architecture is mainly composed of carefully-selected open-source …
Autonomous Systems Lab, ETH Zürich（苏黎世联邦理工学院自主系统实验室）；CVG Group, ETH Zürich（苏黎世联邦理工学院计算机视觉与几何组）
In autonomous racing, vehicles operate close to the limits of handling and a sensor failure can have critical consequences. To limit the impact of such failures, this paper presents the redundant perception and state estimation approaches developed for an autonomous race car. Redundancy in perception is achieved by estimating the color and position of the track delimiting objects using two sensor modalities independently. Specifically, learning-based approaches are used to generate color and pose estimates, from LiDAR and camera data respectively. The redundant perception inputs are fused by a…
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
Environment perception is a crucial ability for robot's interaction into an environment. One of the first steps in this direction is the combined problem of simultaneous localization and mapping (SLAM). A new method, called G-SLAM, is proposed, where the map is considered as a set of scattered points in the continuous space followed by a probability that states the existence of an obstacle in the subsequent point in space. A probabilistic approach with particle filters for the robot's pose estimation and an adaptive recursive algorithm for the map's probability distribution estimation is prese…
Polytechnique Montréal（蒙特利尔综合理工学院）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Massachusetts Institute of Technology（麻省理工学院）
To achieve collaborative tasks, robots in a team need to have a shared understanding of the environment and their location within it. Distributed Simultaneous Localization and Mapping (SLAM) offers a practical solution to localize the robots without relying on an external positioning system (e.g. GPS) and with minimal information exchange. Unfortunately, current distributed SLAM systems are vulnerable to perception outliers and therefore tend to use very conservative parameters for inter-robot place recognition. However, being too conservative comes at the cost of rejecting many valid loop clo…
Simultaneous Localization and Mapping (SLAM) achieves the purpose of simultaneous positioning and map construction based on self-perception. The paper makes an overview in SLAM including Lidar SLAM, visual SLAM, and their fusion. For Lidar or visual SLAM, the survey illustrates the basic type and product of sensors, open source system in sort and history, deep learning embedded, the challenge and future. Additionally, visual inertial odometry is supplemented. For Lidar and visual fused SLAM, the paper highlights the multi-sensors calibration, the fusion in hardware, data, task layer. The open …
An essential task for a multi-robot system is generating a common understanding of the environment and relative poses between robots. Cooperative tasks can be executed only when a vehicle has knowledge of its own state and the states of the team members. However, this has primarily been achieved with direct rendezvous between underwater robots, via inter-robot ranging. We propose a novel distributed multi-robot simultaneous localization and mapping (SLAM) framework for underwater robots using imaging sonar-based perception. By passing only scene descriptors between robots, we do not need to pa…
TAIL: A Terrain-Aware Multi-Modal SLAM Dataset for Robot Locomotion in Deformable Granular Environments
Southern University of Science and Technology (SUSTech)（南方科技大学）；Shenzhen Key Laboratory of Biomimetic Robotics and Intelligent Systems（深圳市仿生机器人与智能系统重点实验室）；Guangdong Provincial Key Laboratory of Human-Augmentation and Rehabilitation Robotics in Universities（广东省高校人机增强与康复机器人重点实验室）
CoPeD-Advancing Multi-Robot Collaborative Perception: A Comprehensive Dataset in Real-World Environments
New York University（纽约大学）；Tandon School of Engineering（坦登工程学院）；U.S. Army Combat Capabilities Development Command（美国陆军作战能力发展司令部）；Army Research Laboratory（陆军研究实验室）
A Semantic Communication Approach to Fiducial Marker Processing in 5G-Enabled Edge SLAM
Faculty of Technical Sciences, University of Novi Sad（诺威萨大学技术科学学院）；The Institute for Artificial Intelligence Research and Development of Serbia（塞尔维亚人工智能研究与发展研究所）
Prevalent solutions for Connected and Autonomous vehicle (CAV) mapping include high definition map (HD map) or real-time Simultaneous Localization and Mapping (SLAM). Both methods only rely on vehicle itself (onboard sensors or embedded maps) and can not adapt well to temporarily changed drivable areas such as work zones. Navigating CAVs in such areas heavily relies on how the vehicle defines drivable areas based on perception information. Difficulties in improving perception accuracy and ensuring the correct interpretation of perception results are challenging to the vehicle in these situatio…
Control-Barrier-Aided Teleoperation with Visual-Inertial SLAM for Safe MAV Navigation in Complex Environments
Learning Systems and Robotics Lab, School of Computation, Information and Technology, Technical University of Munich（学习系统与机器人实验室，计算、信息与技术学院，慕尼黑技术大学）；Smart Robotics Lab, School of Computation, Information and Technology, Technical University of Munich（智能机器人实验室，计算、信息与技术学院，慕尼黑技术大学）；Smart Robotics Lab, Department of Computing, Imperial College London（智能机器人实验室，计算系，伦敦帝国理工学院）；Munich Institute of Robotics and Machine Intellig（慕尼黑机器人与机器智能研究所）
We discuss and predict the evolution of Simultaneous Localisation and Mapping (SLAM) into a general geometric and semantic `Spatial AI' perception capability for intelligent embodied devices. A big gap remains between the visual perception performance that devices such as augmented reality eyewear or comsumer robots will require and what is possible within the constraints imposed by real products. Co-design of algorithms, processors and sensors will be needed. We explore the computational structure of current and future Spatial AI algorithms and consider this within the landscape of ongoing ha…
