In robotic applications, the control, and actuation deal with a continuous description of the system and environment, while high-level planning usually works with a discrete description. This paper considers the problem of bridging the low-level control and high-level planning for robotic systems via sensor data. In particular, we propose a discretization algorithm that identifies free polytopes via lidar point cloud data. A transition graph is then constructed where each node corresponds to a free polytope and two nodes are connected with an edge if the two corresponding free polytopes intersect. Furthermore, a distance measure is associated with each edge, which allows for the assessment of quality (or cost) of the transition for high-level planning. For the low-level control, the free polytopes act as a convenient encoding of the environment and allow for the planning of collision-free trajectories that realizes the high-level plan. The results are demonstrated in high-fidelity ROS simulations and experiments with a drone and a Segway.
This paper presents a technique for navigation of mobile robot with Deep Q-Network (DQN) combined with Gated Recurrent Unit (GRU). The DQN integrated with the GRU allows action skipping for improved navigation performance. This technique aims at efficient navigation of mobile robot such as autonomous parking robot. Framework for reinforcement learning can be applied to the DQN combined with the GRU in a real environment, which can be modeled by the Partially Observable Markov Decision Process (POMDP). By allowing action skipping, the ability of the DQN combined with the GRU in learning key-act…
Person detection is a crucial task for mobile robots navigating in human-populated environments. LiDAR sensors are promising for this task, thanks to their accurate depth measurements and large field of view. Two types of LiDAR sensors exist: the 2D LiDAR sensors, which scan a single plane, and the 3D LiDAR sensors, which scan multiple planes, thus forming a volume. How do they compare for the task of person detection? To answer this, we conduct a series of experiments, using the public, large-scale JackRabbot dataset and the state-of-the-art 2D and 3D LiDAR-based person detectors (DR-SPAAM an…
In the past ten years, the use of 3D Time-of-Flight (ToF) LiDARs in mobile robotics has grown rapidly. Based on our accumulation of relevant research, this article systematically reviews and analyzes the use 3D ToF LiDARs in research and industrial applications. The former includes object detection, robot localization, long-term autonomy, LiDAR data processing under adverse weather conditions, and sensor fusion. The latter encompasses service robots, assisted and autonomous driving, and recent applications performed in response to public health crises. We hope that our efforts can effectively …
Singapore University of Technology and Design（新加坡科技设计大学）；Southwest University of Science and Technology（西南科技大学）；University of Turbat（图尔巴特大学）
Environment mapping is an essential prerequisite for mobile robots to perform different tasks such as navigation and mission planning. With the availability of low-cost 2D LiDARs, there are increasing applications of such 2D LiDARs in industrial environments. However, environment mapping in an unknown and feature-less environment with such low-cost 2D LiDARs remains a challenge. The challenge mainly originates from the short-range of LiDARs and complexities in performing scan matching in these environments. In order to resolve these shortcomings, we propose to fuse the ultra-wideband (UWB) wit…
Technical University of Munich（慕尼黑工业大学）；Sun Yat-sen University（中山大学）
Localization in the environment is an essential navigational capability for animals and mobile robots. In the indoor environment, the global localization problem remains challenging to be perfectly solved with probabilistic methods. However, animals are able to instinctively localize themselves with much less effort. Therefore, it is intriguing and promising to seek biological inspiration from animals. In this paper, we present a biologically-inspired global localization system using a LiDAR sensor that utilizes a hippocampal model and a landmark-based re-localization approach. The experiment …
LE2I-CNRS, University of Technology of Belfort-Montbéliard（法国国家科学研究中心LE2I实验室，贝尔福-蒙贝利亚尔技术大学）；University of Lincoln（林肯大学）
Human detection and tracking is an essential task for service robots, where the combined use of multiple sensors has potential advantages that are yet to be exploited. In this paper, we introduce a framework allowing a robot to learn a new 3D LiDAR-based human classifier from other sensors over time, taking advantage of a multisensor tracking system. The main innovation is the use of different detectors for existing sensors (i.e. RGB-D camera, 2D LiDAR) to train, online, a new 3D LiDAR-based human classifier, exploiting a so-called trajectory probability. Our framework uses this probability to…
ROS 2-Based LiDAR Perception Framework for Mobile Robots in Dynamic Production Environments, Utilizing Synthetic Data Generation, Transformation-Equivariant 3D Detection and Multi-Object Tracking
Chair of Intelligence in Quality Sensing (IQS), Laboratory for Machine Tools and Production Engineering (WZL), RWTH Aachen University（亚琛工业大学机床与生产工程实验室质量感知智能教席）；The Sirindhorn International Thai-German Graduate School of Engineering (TGGS), King Mongkut’s University of Technology North Bangkok (KMUTNB)（北曼谷先皇技术大学泰德国际工程研究生院）
Robust and accurate, map-based localization is crucial for autonomous mobile systems. In this paper, we exploit range images generated from 3D LiDAR scans to address the problem of localizing mobile robots or autonomous cars in a map of a large-scale outdoor environment represented by a triangular mesh. We use the Poisson surface reconstruction to generate the mesh-based map representation. Based on the range images generated from the current LiDAR scan and the synthetic rendered views from the mesh-based map, we propose a new observation model and integrate it into a Monte Carlo localization …
Real-Time Spatial Reasoning by Mobile Robots for Reconstruction and Navigation in Dynamic LiDAR Scenes
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Computer Science, University of Guelph（圭尔夫大学计算机科学学院）；School of Computing Science, Simon Fraser University（西蒙弗雷泽大学计算机科学学院）
Institute for Digital Technologies, Loughborough University（拉夫堡大学数字技术研究所）
Autonomous robots that assist humans in day to day living tasks are becoming increasingly popular. Autonomous mobile robots operate by sensing and perceiving their surrounding environment to make accurate driving decisions. A combination of several different sensors such as LiDAR, radar, ultrasound sensors and cameras are utilized to sense the surrounding environment of autonomous vehicles. These heterogeneous sensors simultaneously capture various physical attributes of the environment. Such multimodality and redundancy of sensing need to be positively utilized for reliable and consistent per…
Moving Object Localization based on the Fusion of Ultra-WideBand and LiDAR with a Mobile Robot
Singapore University of Technology and Design（新加坡科技设计大学）；Nanyang Technological University（南洋理工大学）；Southwest University of Science and Technology（西南科技大学）
Daimler AG（戴姆勒股份公司）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）
Lidar sensors are frequently used in environment perception for autonomous vehicles and mobile robotics to complement camera, radar, and ultrasonic sensors. Adverse weather conditions are significantly impacting the performance of lidar-based scene understanding by causing undesired measurement points that in turn effect missing detections and false positives. In heavy rain or dense fog, water drops could be misinterpreted as objects in front of the vehicle which brings a mobile robot to a full stop. In this paper, we present the first CNN-based approach to understand and filter out such adver…
Semantic2D: Enabling Semantic Scene Understanding with 2D Lidar Alone
School of Computing and Data Science, The University of Hong Kong（计算与数据科学学院，香港大学）；Department of Mechanical Engineering, Temple University（机械工程系， Temple大学）
Zen Aerologiks；BMS College of Engineering（BMS工程学院）
The navigation system is at the heart of any mobile robot it comprises of SLAM and path planning units, which is utilized by the robot to generate a map of the environment, localize itself within it and determine an optimal a path to the destination. This paper describes the conceptualization, development, simulation and hardware implementation of GenNav a generic indoor navigation system for any mobile aerial or ground robot. The generalization is brought about by modularizing and creating independence between the software computation and hardware actuation units by providing an alternate sou…
Radar and lidar, provided by two different range sensors, each has pros and cons of various perception tasks on mobile robots or autonomous driving. In this paper, a Monte Carlo system is used to localize the robot with a rotating radar sensor on 2D lidar maps. We first train a conditional generative adversarial network to transfer raw radar data to lidar data, and achieve reliable radar points from generator. Then an efficient radar odometry is included in the Monte Carlo system. Combining the initial guess from odometry, a measurement model is proposed to match the radar data and prior lidar…
FGGS-LiDAR: Ultra-Fast, GPU-Accelerated Simulation from General 3DGS Models to LiDAR
Tsinghua University, Shenzhen, China（清华大学深圳研究院）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；DISCOVER Robotics；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））
MINES ParisTech, PSL Research University（巴黎矿业学院，巴黎文理研究大学）；École des Ponts ParisTech（巴黎高科桥路学院）
In this paper, we address the problem of time-optimal coordination of mobile robots under kinodynamic constraints along specified paths. We propose a novel approach based on time discretization that leads to a mixed-integer linear programming (MILP) formulation. This problem can be solved using general-purpose MILP solvers in a reasonable time, resulting in a resolution-optimal solution. Moreover, unlike previous work found in the literature, our formulation allows an exact linear modeling (up to the discretization resolution) of second-order dynamic constraints. Extensive simulations are perf…
Knowing the position of the robot in the world is crucial for navigation. Nowadays, Bayesian filters, such as Kalman and particle-based, are standard approaches in mobile robotics. Recently, end-to-end learning has allowed for scaling-up to high-dimensional inputs and improved generalization. However, there are still limitations to providing reliable laser navigation. Here we show a proof-of-concept of the predictive processing-inspired approach to perception applied for localization and navigation using laser sensors, without the need for odometry. We learn the generative model of the laser t…
Localization is a crucial capability for mobile robots and autonomous cars. In this paper, we address learning an observation model for Monte-Carlo localization using 3D LiDAR data. We propose a novel, neural network-based observation model that computes the expected overlap of two 3D LiDAR scans. The model predicts the overlap and yaw angle offset between the current sensor reading and virtual frames generated from a pre-built map. We integrate this observation model into a Monte-Carlo localization framework and tested it on urban datasets collected with a car in different seasons. The experi…
The Hong Kong University of Science and Technology（香港科技大学）
Multiple LiDARs have progressively emerged on autonomous vehicles for rendering a wide field of view and dense measurements. However, the lack of precise calibration negatively affects their potential applications in localization and perception systems. In this paper, we propose a novel system that enables automatic multi-LiDAR calibration without any calibration target, prior environmental information, and initial values of the extrinsic parameters. Our approach starts with a hand-eye calibration for automatic initialization by aligning the estimated motions of each sensor. The resulting para…
Accurate LiDAR-camera extrinsic calibration is a precondition for many multi-sensor systems in mobile robots. Most calibration methods rely on laborious manual operations and calibration targets. While working online, the calibration methods should be able to extract information from the environment to construct the cross-modal data association. Convolutional neural networks (CNNs) have powerful feature extraction ability and have been used for calibration. However, most of the past methods solve the extrinsic as a regression task, without considering the geometric constraints involved. In thi…
3D laser scanning by LiDAR sensors plays an important role for mobile robots to understand their surroundings. Nevertheless, not all systems have high resolution and accuracy due to hardware limitations, weather conditions, and so on. Generative modeling of LiDAR data as scene priors is one of the promising solutions to compensate for unreliable or incomplete observations. In this paper, we propose a novel generative model for learning LiDAR data based on generative adversarial networks. As in the related studies, we process LiDAR data as a compact yet lossless representation, a cylindrical de…
Kyushu University（九州大学）；Jet Propulsion Laboratory, California Institute of Technology（加州理工学院喷气推进实验室）
3D LiDAR sensors are indispensable for the robust vision of autonomous mobile robots. However, deploying LiDAR-based perception algorithms often fails due to a domain gap from the training environment, such as inconsistent angular resolution and missing properties. Existing studies have tackled the issue by learning inter-domain mapping, while the transferability is constrained by the training configuration and the training is susceptible to peculiar lossy noises called ray-drop. To address the issue, this paper proposes a generative model of LiDAR range images applicable to the data-level dom…
ANSYS France（ANSYS法国公司）；ENSTA Paris, Institut Polytechnique de Paris（巴黎高等先进技术学院，巴黎理工学院）
LiDAR sensors provide rich 3D information about their surrounding{s} and are becoming increasingly important for autonomous vehicles tasks such as {localization}, semantic segmentation, object detection, and tracking. {Simulation} accelerates the testing, validation, and deployment of autonomous vehicles while {also} reducing cost and eliminating the risks of testing in real-world scenarios. We address the problem of high-fidelity LiDAR simulation and present a pipeline that leverages real-world point clouds acquired by mobile mapping systems. Point-based geometry representations, more specifi…
University of Oxford（牛津大学）；Imperial College London（帝国理工学院）
We present an efficient, elastic 3D LiDAR reconstruction framework which can reconstruct up to maximum LiDAR ranges (60 m) at multiple frames per second, thus enabling robot exploration in large-scale environments. Our approach only requires a CPU. We focus on three main challenges of large-scale reconstruction: integration of long-range LiDAR scans at high frequency, the capacity to deform the reconstruction after loop closures are detected, and scalability for long-duration exploration. Our system extends upon a state-of-the-art efficient RGB-D volumetric reconstruction technique, called sup…
Biomedical Research Institute of Malaga (IBIMA)（马拉加生物医学研究所（IBIMA））；University of Malaga（马拉加大学）
In order to fuse measurements from multiple sensors mounted on a mobile robot, it is needed to express them in a common reference system through their relative spatial transformations. In this paper, we present a method to estimate the full 6DoF extrinsic calibration parameters of multiple heterogeneous sensors (Lidars, Depth and RGB cameras) suitable for automatic execution on a mobile robot. Our method computes the 2D calibration parameters (x, y, yaw) through a motion-based approach, while for the remaining 3 parameters (z, pitch, roll) it requires the observation of the ground plane for a …
University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
Enabling mobile robots for solving challenging and diverse shape, texture, and motion related tasks with high fidelity vision requires the integration of novel multimodal imaging sensors and advanced fusion techniques. However, it is associated with high cost, power, hardware modification, and computing requirements which limit its scalability. In this paper, we propose a novel Simultaneously Learned Auto Encoder Domain Adaptation (SAEDA)-based transfer learning technique to empower noisy sensing with advanced sensor suite capabilities. In this regard, SAEDA trains both source and target auto-…
We were able to complete the full integration of the Robotic Technology Kernel (RTK) into an electric vehicle by-wire system using lidar and GPS sensors. The solution included a mobile application to interface with the RTK-enabled autonomous vehicle. Altogether the system was designed to be modular, using the concepts of message-based software design that is built into the Robot Operating System (ROS), which is at the foundation of RTK. The team worked incrementally to develop working software to demonstrate each milestone on the path to successfully completing the RTK integration for the deve…
University of Oxford（牛津大学）；KTH Royal Institute of Technology（KTH皇家理工学院）；Scania AB（斯堪尼亚公司）
We present an efficient multi-sensor odometry system for mobile platforms that jointly optimizes visual, lidar, and inertial information within a single integrated factor graph. This runs in real-time at full framerate using fixed lag smoothing. To perform such tight integration, a new method to extract 3D line and planar primitives from lidar point clouds is presented. This approach overcomes the suboptimality of typical frame-to-frame tracking methods by treating the primitives as landmarks and tracking them over multiple scans. True integration of lidar features with standard visual feature…
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）；CWB Institute of Autonomous Driving（CWB自动驾驶研究院）
The combination of LiDARs and cameras enables a mobile robot to perceive environments with multi-modal data, becoming a key factor in achieving robust perception. Traditional frame cameras are sensitive to changing illumination conditions, motivating us to introduce novel event cameras to make LiDAR-camera fusion more complete and robust. However, to jointly exploit these sensors, the challenging extrinsic calibration problem should be addressed. This paper proposes an automatic checkerboard-based approach to calibrate extrinsics between a LiDAR and a frame/event camera, where four contributio…
Accurate pose estimation is a fundamental ability that all mobile robots must posses in order to traverse robustly in a given environment. Much like a human, this ability is dependent on the robot's understanding of a given scene. For Autonomous Vehicles (AV's), detailed 3D maps created beforehand are widely used to augment the perceptive abilities and estimate pose based on current sensor measurements. This approach however is less suited for rural communities that are sparsely connected and cover large areas. To deal with the challenge of localizing a vehicle in a rural setting, this paper p…
PERAL: Perception-Aware Motion Control for Passive LiDAR Excitation in Spherical Robots
Centre for Advanced Robotics Technology Innovation (CARTIN), School of Electrical and Electronic Engineering, Nanyang Technological University（先进机器人技术创新中心（CARTIN）、电气与电子工程学院、南洋理工大学）
Due to their ubiquity and long-term stability, pole-like objects are well suited to serve as landmarks for vehicle localization in urban environments. In this work, we present a complete mapping and long-term localization system based on pole landmarks extracted from 3-D lidar data. Our approach features a novel pole detector, a mapping module, and an online localization module, each of which are described in detail, and for which we provide an open-source implementation at www.github.com/acschaefer/polex. In extensive experiments, we demonstrate that our method improves on the state of the ar…
This paper proposes a framework to measure the important metrics (throughput, delay, packet retransmits, signal strength, etc.) to determine Wi-Fi network performance of mobile robots supported by the Robot Operating Systems (ROS) middleware. We analyze the bidirectional network performance of mobile robots through an experimental setup in an indoor environment, where a mobile robot is communicating vital sensor data such as video streaming from the camera(s) and LiDAR scan values to a command station while it navigates an indoor environment through teleoperated velocity commands received from…
