Robustness is a crucial factor for the successful deployment of robots in unstructured environments, particularly in the domain of Simultaneous Localization and Mapping (SLAM). Simulation-based benchmarks have emerged as a highly scalable approach for robustness evaluation compared to real-world data collection. However, crafting a challenging and controllable noisy world with diverse perturbations remains relatively under-explored. To this end, we propose a novel, customizable pipeline for noisy data synthesis, aimed at assessing the resilience of multi-modal SLAM models against various perturbations. This pipeline incorporates customizable hardware setups, software components, and perturbed environments. In particular, we introduce comprehensive perturbation taxonomy along with a perturbation composition toolbox, allowing the transformation of clean simulations into challenging noisy environments. Utilizing the pipeline, we instantiate the Robust-SLAM benchmark, which includes diverse perturbation types, to evaluate the risk tolerance of existing advanced multi-modal SLAM models. Our extensive analysis uncovers the susceptibilities of existing SLAM models to real-world disturbance, despite their demonstrated accuracy in standard benchmarks. Our perturbation synthesis toolbox, SLAM robustness evaluation pipeline, and Robust-SLAM benchmark will be made publicly available at https://github.com/Xiaohao-Xu/SLAM-under-Perturbation/.
Benchmarking Simultaneous Localization and Mapping (SLAM) algorithms is important to scientists and users of robotic systems alike. But through their many configuration options in hardware and software, SLAM systems feature a vast parameter space that scientists up to now were not able to explore. The proposed SLAM Hive Benchmarking Suite is able to analyze SLAM algorithms in 1000's of mapping runs, through its utilization of container technology and deployment in a cluster. This paper presents the architecture and open source implementation of SLAM Hive and compares it to existing efforts on …
University of Edinburgh(爱丁堡大学);Imperial College London(帝国理工学院);University of Manchester(曼彻斯特大学);Stanford University(斯坦福大学)
SLAM is becoming a key component of robotics and augmented reality (AR) systems. While a large number of SLAM algorithms have been presented, there has been little effort to unify the interface of such algorithms, or to perform a holistic comparison of their capabilities. This is a problem since different SLAM applications can have different functional and non-functional requirements. For example, a mobile phonebased AR application has a tight energy budget, while a UAV navigation system usually requires high accuracy. SLAMBench2 is a benchmarking framework to evaluate existing and future SLAM…
Visual-inertial SLAM is essential for robot navigation in GPS-denied environments, e.g. indoor, underground. Conventionally, the performance of visual-inertial SLAM is evaluated with open-loop analysis, with a focus on the drift level of SLAM systems. In this paper, we raise the question on the importance of visual estimation latency in closed-loop navigation tasks, such as accurate trajectory tracking. To understand the impact of both drift and latency on visual-inertial SLAM systems, a closed-loop benchmarking simulation is conducted, where a robot is commanded to follow a desired trajectory…
Task-driven SLAM Benchmarking For Robot Navigation
School of Electrical and Computer Engineering, and Institute of Robotics and Intelligent Machines, Georgia Institute of Technology(电气与计算机工程学院,机器人与智能机器研究所,佐治亚理工学院);School of Mechanical Engineering and the School of Electrical and Computer Engineering, Georgia Institute of Technology(机械工程学院,电气与计算机工程学院,佐治亚理工学院)
This paper contains the performance analysis and benchmarking of two popular visual SLAM Algorithms: RGBD-SLAM and RTABMap. The dataset used for the analysis is the TUM RGBD Dataset from the Computer Vision Group at TUM. The dataset selected has a large set of image sequences from a Microsoft Kinect RGB-D sensor with highly accurate and time-synchronized ground truth poses from a motion capture system. The test sequences selected depict a variety of problems and camera motions faced by Simultaneous Localization and Mapping (SLAM) algorithms for the purpose of testing the robustness of the algo…
Imperial College(帝国理工学院);The Univ. of Edinburgh(爱丁堡大学);Univ. of Manchester(曼彻斯特大学)
Real-time dense computer vision and SLAM offer great potential for a new level of scene modelling, tracking and real environmental interaction for many types of robot, but their high computational requirements mean that use on mass market embedded platforms is challenging. Meanwhile, trends in low-cost, low-power processing are towards massive parallelism and heterogeneity, making it difficult for robotics and vision researchers to implement their algorithms in a performance-portable way. In this paper we introduce SLAMBench, a publicly-available software framework which represents a starting …
Spatiotemporal Calibration and Ground Truth Estimation for High-Precision SLAM Benchmarking in Extended Reality
Advanced Display and Sensing Research Center, Yongjiang Laboratory, Ningbo, China(先进显示与感知研究中心,永江实验室,宁波,中国);Shanghai Institute of Technical Physics, Chinese Academy of Sciences, Shanghai, China(上海技术物理研究所,中国科学院,上海,中国)
AquaticVision: Benchmarking Visual SLAM in Underwater Environment with Events and Frames
School of Data Science, The Chinese University of Hong Kong, Shenzhen, P. R. China(数据科学学院,香港中文大学(深圳));School of Life Sciences, The Chinese University of Hong Kong, Hongkong(生命科学学院,香港中文大学)
SMapper: A Multi-Modal Data Acquisition Platform for SLAM Benchmarking
Automation and Robotics Research Group (ARG), Interdisciplinary Centre for Security, Reliability, and Trust (SnT), University of Luxembourg(卢森堡大学自动化与机器人研究组(ARG)、安全、可靠与信任跨学科研究中心(SnT))
MoCap2GT: A High-Precision Ground Truth Estimator for SLAM Benchmarking Based on Motion Capture and IMU Fusion
Shanghai Institute of Technical Physics, Chinese Academy of Sciences(上海技术物理研究所,中国科学院);Advanced Display and Sensing Research Center, Yongjiang Laboratory(先进显示与传感研究中心,永江实验室)
PALoc: Advancing SLAM Benchmarking with Prior-Assisted 6-DoF Trajectory Generation and Uncertainty Estimation
Hong Kong University of Science and Technology(香港科技大学);Hong Kong University of Science and Technology, Guangzhou Campus(香港科技大学广州校区);South China Normal University(华南师范大学);University College London(伦敦大学学院)
SLAM has matured significantly over the past few years, and is beginning to appear in serious commercial products. While new SLAM systems are being proposed at every conference, evaluation is often restricted to qualitative visualizations or accuracy estimation against a ground truth. This is due to the lack of benchmarking methodologies which can holistically and quantitatively evaluate these systems. Further investigation at the level of individual kernels and parameter spaces of SLAM pipelines is non-existent, which is absolutely essential for systems research and integration. We extend the…
The diversity of SLAM benchmarks affords extensive testing of SLAM algorithms to understand their performance, individually or in relative terms. The ad-hoc creation of these benchmarks does not necessarily illuminate the particular weak points of a SLAM algorithm when performance is evaluated. In this paper, we propose to use a decision tree to identify challenging benchmark properties for state-of-the-art SLAM algorithms and important components within the SLAM pipeline regarding their ability to handle these challenges. Establishing what factors of a particular sequence lead to track failur…
HERO-SLAM: Hybrid Enhanced Robust Optimization of Neural SLAM
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Automation, Beijing Institute of Technology(北京理工大学自动化学院);Baidu Research(百度研究院)
Polytechnique Montréal(蒙特利尔综合理工学院);École Polytechnique Fédérale de Lausanne(洛桑联邦理工学院);Massachusetts Institute of Technology(麻省理工学院)
To achieve collaborative tasks, robots in a team need to have a shared understanding of the environment and their location within it. Distributed Simultaneous Localization and Mapping (SLAM) offers a practical solution to localize the robots without relying on an external positioning system (e.g. GPS) and with minimal information exchange. Unfortunately, current distributed SLAM systems are vulnerable to perception outliers and therefore tend to use very conservative parameters for inter-robot place recognition. However, being too conservative comes at the cost of rejecting many valid loop clo…
We present a novel dataset for training and benchmarking semantic SLAM methods. The dataset consists of 200 long sequences, each one containing 3000-5000 data frames. We generate the sequences using realistic home layouts. For that we sample trajectories that simulate motions of a simple home robot, and then render the frames along the trajectories. Each data frame contains a) RGB images generated using physically-based rendering, b) simulated depth measurements, c) simulated IMU readings and d) ground truth occupancy grid of a house. Our dataset serves a wider range of purposes compared to ex…
The S3LI Vulcano Dataset: A Dataset for Multi-Modal SLAM in Unstructured Planetary Environments
German Aerospace Center (DLR) Institute of Robotics and Mechatronics(德国航空航天中心(DLR)机器人与机电研究所);KTH Royal Institute of Technology(皇家理工学院);Deptartment of Intelligent Systems(智能系统部门)
Beijing Jiaotong University(北京交通大学);Intel Labs China(英特尔中国实验室);Beihang University(北京航空航天大学);Tsinghua University(清华大学);Gaussian Robotics(高斯机器人);Department of Electronic Engineering and BNRist(电子工程系和BNRist)
Service robots should be able to operate autonomously in dynamic and daily changing environments over an extended period of time. While Simultaneous Localization And Mapping (SLAM) is one of the most fundamental problems for robotic autonomy, most existing SLAM works are evaluated with data sequences that are recorded in a short period of time. In real-world deployment, there can be out-of-sight scene changes caused by both natural factors and human activities. For example, in home scenarios, most objects may be movable, replaceable or deformable, and the visual features of the same place may …
A Sensor-Aware Phenomenological Framework for Lidar Degradation Simulation and SLAM Robustness Evaluation
Turku Intelligent Embedded and Robotic Systems (TIERS) Lab , University of Turku(图尔库智能嵌入与机器人系统实验室,图尔库大学);School of Information Science and Technology, Fudan University(信息科学与技术学院,复旦大学)
Datasets have gained an enormous amount of popularity in the computer vision community, from training and evaluation of Deep Learning-based methods to benchmarking Simultaneous Localization and Mapping (SLAM). Without a doubt, synthetic imagery bears a vast potential due to scalability in terms of amounts of data obtainable without tedious manual ground truth annotations or measurements. Here, we present a dataset with the aim of providing a higher degree of photo-realism, larger scale, more variability as well as serving a wider range of purposes compared to existing datasets. Our dataset lev…
FunnyNodules: A Customizable Medical Dataset Tailored for Evaluating Explainable AI
Ulm University Medical Center(乌尔姆大学医学中心);XAIRAD - Cooperation for Artificial Intelligence in Experimental Radiology(XAIRAD——实验放射学人工智能合作);Department of Diagnostic and Interventional Radiology(诊断与介入放射学系)
École Polytechnique de Montréal(蒙特利尔理工学院);Massachusetts Institute of Technology(麻省理工学院)
Perceptual aliasing is one of the main causes of failure for Simultaneous Localization and Mapping (SLAM) systems operating in the wild. Perceptual aliasing is the phenomenon where different places generate a similar visual (or, in general, perceptual) footprint. This causes spurious measurements to be fed to the SLAM estimator, which typically results in incorrect localization and mapping results. The problem is exacerbated by the fact that those outliers are highly correlated, in the sense that perceptual aliasing creates a large number of mutually-consistent outliers. Another issue stems fr…
Instituto Federal de Educação, Ciência e Tecnologia de Minas Gerais (IFMG)(米纳斯吉拉斯联邦教育科学技术学院(IFMG));Federal University of Minas Gerais(米纳斯吉拉斯联邦大学);Graduate Program in Electrical Engineering - Federal University of Minas Gerais(米纳斯吉拉斯联邦大学电气工程研究生项目);Machine Intelligence and Data Science (MINDS) Laboratory, Federal University of Minas Gerais (UFMG)(米纳斯吉拉斯联邦大学(UFMG)机器智能与数据科学(MINDS)实验室);Institute of Polyners and Composites, University of Minho (Uminho)(米尼奥大学(Uminho)聚合物与复合材料研究所)
Solving many-objective problems (MaOPs) is still a significant challenge in the multi-objective optimization (MOO) field. One way to measure algorithm performance is through the use of benchmark functions (also called test functions or test suites), which are artificial problems with a well-defined mathematical formulation, known solutions and a variety of features and difficulties. In this paper we propose a parameterized generator of scalable and customizable benchmark problems for MaOPs. It is able to generate problems that reproduce features present in other benchmarks and also problems wi…
Laboratory for Information & Decision Systems, Massachusetts Institute of Technology(麻省理工学院信息与决策系统实验室);U.S. Army Combat Capabilities Development Command, Army Research Laboratory(美国陆军作战能力发展司令部陆军研究实验室)
This paper presents Kimera-Multi, the first multi-robot system that (i) is robust and capable of identifying and rejecting incorrect inter and intra-robot loop closures resulting from perceptual aliasing, (ii) is fully distributed and only relies on local (peer-to-peer) communication to achieve distributed localization and mapping, and (iii) builds a globally consistent metric-semantic 3D mesh model of the environment in real-time, where faces of the mesh are annotated with semantic labels. Kimera-Multi is implemented by a team of robots equipped with visual-inertial sensors. Each robot builds…
Fondazione Istituto Italiano di Tecnologia (IIT)(意大利技术研究院);Istituto Nanoscienze-CNR(意大利国家研究委员会纳米科学研究所);Scuola Normale Superiore(比萨高等师范学校);Istituto per le Applicazioni del Calcolo del Consiglio Nazionale delle Ricerche(意大利国家研究委员会计算应用研究所);Università degli Studi Roma tre(罗马第三大学);Università di Pisa(比萨大学)
Tracking droplets in microfluidics is a challenging task. The difficulty arises in choosing a tool to analyze general microfluidic videos to infer physical quantities. The state-of-the-art object detector algorithm You Only Look Once (YOLO) and the object tracking algorithm Simple Online and Realtime Tracking with a Deep Association Metric (DeepSORT) are customizable for droplet identification and tracking. The customization includes training YOLO and DeepSORT networks to identify and track the objects of interest. We trained several YOLOv5 and YOLOv7 models and the DeepSORT network for drople…
Lidar technology has evolved significantly over the last decade, with higher resolution, better accuracy, and lower cost devices available today. In addition, new scanning modalities and novel sensor technologies have emerged in recent years. Public datasets have enabled benchmarking of algorithms and have set standards for the cutting edge technology. However, existing datasets are not representative of the technological landscape, with only a reduced number of lidars available. This inherently limits the development and comparison of general-purpose algorithms in the evolving landscape. This…
Graph-SLAM is a well-established algorithm for constructing a topological map of the environment while simultaneously attempting the localisation of the robot. It relies on scan matching algorithms to align noisy observations along robot's movements to compute an estimate of the current robot's location. We propose a fundamentally different approach to scan matching tasks to improve the estimation of roto-translation displacements and therefore the performance of the full SLAM algorithm. A Monte-Carlo approach is used to generate weighted hypotheses of the geometrical displacement between two …
This paper provides the EVOPS dataset for plane segmentation from 3D data, both from RGBD images and LiDAR point clouds. We have designed two annotation methodologies (RGBD and LiDAR) running on well-known and widely-used datasets for SLAM evaluation and we have provided a complete set of benchmarking tools including point, planes and segmentation metrics. The data includes a total number of 10k RGBD and 7K LiDAR frames over different selected scenes which consist of high quality segmented planes. The experiments report quality of SOTA methods for RGBD plane segmentation on our annotated data.…
GRADE: Generating Realistic And Dynamic Environments for Robotics Research with Isaac Sim
Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of Stuttgart(斯图加特大学);Swiss Federal Institute of Technology Lausanne (EPFL)(洛桑联邦理工学院)
提出基于 NVIDIA Isaac Sim 的 GRADE 框架,通过生成逼真动态环境、收集合成数据集并训练模型以缩小仿真到现实差距,同时基准测试动态 V-SLAM 算法揭示其局限性。
Benchmarking Neural Radiance Fields for Autonomous Robots: An Overview
School of Computer Science, Hangzhou Dianzi University(杭州电子科技大学计算机学院);High-speed Aerodynamics Institute, CARDC(中国空气动力研究与发展中心高速空气动力研究所);Stevens Institute of Technology(史蒂文斯理工学院);Luddy School of Informatics, Computing, and Engineering, Indiana University(印第安纳大学勒迪信息、计算与工程学院);City College of New York(纽约城市学院);School of Computer Science, University of Bristol(布里斯托大学计算机学院);State Key Lab of CAD&CG, Zhejiang University(浙江大学CAD&CG国家重点实验室)
LanEvil: Benchmarking the Robustness of Lane Detection to Environmental Illusions
Beihang University(北京航空航天大学);Data Space Research Institute of Hefei Comprehensive National Science Centre(合肥综合性国家科学中心数据空间研究院);National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学)
Event-based SLAM Benchmark for High-Speed Maneuvers
Neuromorphic Automation and Intelligence Lab (NAIL) at the School of Artificial Intelligence and Robotics, Hunan University, Changsha, China(神经形态自动化与智能实验室(NAIL)位于湖南大学人工智能与机器人学院,长沙,中国);Department of Electrical Engineering and Computer Science, Technische Universität Berlin, the Science of Intelligence Excellence Cluster, the Robotics Institute Germany and the Einstein Center Digital Future, Berlin, Germany(柏林技术大学电气工程与计算机科学系,智能科学卓越中心,德国机器人研究所,数字未来爱因斯坦中心,柏林,德国);College of Intelligence Science and Technology, National University of Defense Technology, Changsha, China(国防科技大学智能科学与技术学院,长沙,中国);School of Artificial Intelligence and Robotics, Hunan University, Changsha China(湖南大学人工智能与机器人学院,长沙中国);Robotics and Perception Group, Department of Informatics, University of Zurich, Switzerland(苏黎世大学信息学院机器人与感知组,瑞士)
LIRIS, UMR CNRS 5205, Université de Lyon, INSA Lyon(里昂大学、法国国家科学研究中心、里昂国立应用科学学院);Naver Labs Europe(Naver Labs 欧洲)
Visual navigation by mobile robots is classically tackled through SLAM plus optimal planning, and more recently through end-to-end training of policies implemented as deep networks. While the former are often limited to waypoint planning, but have proven their efficiency even on real physical environments, the latter solutions are most frequently employed in simulation, but have been shown to be able learn more complex visual reasoning, involving complex semantical regularities. Navigation by real robots in physical environments is still an open problem. End-to-end training approaches have bee…