Visual-Inertial Telepresence for Aerial Manipulation
German Aerospace Center (DLR)（德国航空航天中心（DLR））；Skolkovo Institute of Science and Technology (Skoltech)（斯科尔科沃科学技术研究院（Skoltech））；Technical University of Munich（慕尼黑工业大学）
Jongseok Lee, Ribin Balachandran, Yuri S. Sarkisov, Marco De Stefano, Andre Coelho, Kashmira Shinde, Min Jun Kim, Rudolph Triebel, Konstantin Kondak
备注：
Accepted to International Conference on Robotics and Automation (ICRA) 2020, IEEE copyright, 8 pages, 10 figures
This paper presents a novel telepresence system for enhancing aerial manipulation capabilities. It involves not only a haptic device, but also a virtual reality that provides a 3D visual feedback to a remotely-located teleoperator in real-time. We achieve this by utilizing onboard visual and inertial sensors, an object tracking algorithm and a pre-generated object database. As the virtual reality has to closely match the real remote scene, we propose an extension of a marker tracking algorithm with visual-inertial odometry. Both indoor and outdoor experiments show benefits of our proposed system in achieving advanced aerial manipulation tasks, namely grasping, placing, force exertion and peg-in-hole insertion.
Feelium: A Touchable Blimp Body for Aerial Telepresence
Stony Brook University（石溪大学）；New York University（纽约大学）；Massachusetts Institute of Technology（麻省理工学院）；Middlebury Institute of International Studies at Monterey（明德蒙特雷国际研究院）
German Aerospace Center (DLR)（德国航空航天中心（DLR））；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院（KIT））；University of Twente (UT)（特文特大学（UT））；Technical University of Munich (TUM)（慕尼黑工业大学（TUM））
This article presents a novel telepresence system for advancing aerial manipulation in dynamic and unstructured environments. The proposed system not only features a haptic device, but also a virtual reality (VR) interface that provides real-time 3D displays of the robot's workspace as well as a haptic guidance to its remotely located operator. To realize this, multiple sensors namely a LiDAR, cameras and IMUs are utilized. For processing of the acquired sensory data, pose estimation pipelines are devised for industrial objects of both known and unknown geometries. We further propose an active…
This paper presents the design of a small aerial robot for inhabited microgravity environments, such as orbiting space stations (e.g., ISS). In particular, we target a fleet of robots, called Space CoBots, for collaborative tasks with humans, such as telepresence and cooperative mobile manipulation. The design is modular, comprising an hexrotor based propulsion system, and a stack of modules including batteries, cameras for navigation, a screen for telepresence, a robotic arm, space for extension modules, and a pair of docking ports. These ports can be used for docking and for mechanically att…
Visual-Inertial odometry (VIO) is the process of estimating the state (pose and velocity) of an agent (e.g., an aerial robot) by using only the input of one or more cameras plus one or more Inertial Measurement Units (IMUs) attached to it. VIO is the only viable alternative to GPS and lidar-based odometry to achieve accurate state estimation. Since both cameras and IMUs are very cheap, these sensor types are ubiquitous in all today's aerial robots.
Autonomous Robots Lab, University of Nevada, Reno（内华达大学雷诺分校自主机器人实验室）
For robotic inspection tasks in known environments fiducial markers provide a reliable and low-cost solution for robot localization. However, detection of such markers relies on the quality of RGB camera data, which degrades significantly in the presence of visual obscurants such as fog and smoke. The ability to navigate known environments in the presence of obscurants can be critical for inspection tasks especially, in the aftermath of a disaster. Addressing such a scenario, this work proposes a method for the design of fiducial markers to be used with thermal cameras for the pose estimation …
University of Maryland Institute for Advanced Computer Studies（马里兰大学高级计算机研究所）
Odometry on aerial robots has to be of low latency and high robustness whilst also respecting the Size, Weight, Area and Power (SWAP) constraints as demanded by the size of the robot. A combination of visual sensors coupled with Inertial Measurement Units (IMUs) has proven to be the best combination to obtain robust and low latency odometry on resource-constrained aerial robots. Recently, deep learning approaches for Visual Inertial fusion have gained momentum due to their high accuracy and robustness. However, the remarkable advantages of these techniques are their inherent scalability (adapt…
Hong Kong University of Science and Technology（香港科技大学）
Decentralized state estimation is one of the most fundamental components of autonomous aerial swarm systems in GPS-denied areas yet it still remains a highly challenging research topic. Omni-swarm, a decentralized omnidirectional visual-inertial-UWB state estimation system for aerial swarms, is proposed in this paper to address this research niche. To solve the issues of observability, complicated initialization, insufficient accuracy, and lack of global consistency, we introduce an omnidirectional perception front-end in Omni-swarm. It consists of stereo wide-FoV cameras and ultra-wideband se…
This paper presents a visual-inertial odometry-enhanced geometrically stable Iterative Closest Point (ICP) algorithm for accurate mapping using aerial robots. The proposed method employs a visual-inertial odometry framework in order to provide robust priors to the ICP step and calculate the overlap among point clouds derived from an onboard time-of-flight depth sensor. Within the overlapping parts of the point clouds, the method samples points such that the distribution of normals among them is as large as possible. As different geometries and sensor trajectories will influence the performance…
Hong Kong University of Science and Technology（香港科技大学）
The collaboration of unmanned aerial vehicles (UAVs) has become a popular research topic for its practicability in multiple scenarios. The collaboration of multiple UAVs, which is also known as aerial swarm is a highly complex system, which still lacks a state-of-art decentralized relative state estimation method. In this paper, we present a novel fully decentralized visual-inertial-UWB fusion framework for relative state estimation and demonstrate the practicability by performing extensive aerial swarm flight experiments. The comparison result with ground truth data from the motion capture sy…
This paper addresses the problem of cooperative transport of a point mass hoisted by two aerial robots. Treating the robots as a leader and a follower, the follower stabilizes the system with respect to the leader using only feedback from its Inertial Measurement Units (IMU). This is accomplished by neglecting the acceleration of the leader, analyzing the system through the generalized coordinates or the cables' angles, and employing an observation model based on the IMU measurements. A lightweight estimator based on an Extended Kalman Filter (EKF) and a controller are derived to stabilize the…
Huazhong University of Science and Technology（华中科技大学）
This paper presents an indoor pose estimation system for micro aerial vehicles (MAVs) with a single WiFi access point. Conventional approaches based on computer vision are limited by illumination conditions and environmental texture. Our system is free of visual limitations and instantly deployable, working upon existing WiFi infrastructure without any deployment cost. Our system consists of two coupled modules. First, we propose an angle-of-arrival (AoA) estimation algorithm to estimate MAV attitudes and disentangle the AoA for positioning. Second, we formulate a WiFi-inertial sensor fusion m…
Autonomous Systems Lab, ETH Zurich（苏黎世联邦理工学院自主系统实验室）；Google Inc.（谷歌公司）
Industrial facilities often require periodic visual inspections of key installations. Examining these points of interest is time consuming, potentially hazardous or require special equipment to reach. MAVs are ideal platforms to automate this expensive and tedious task. In this work we present a novel system that enables a human operator to teach a visual inspection task to an autonomous aerial vehicle by simply demonstrating the task using a handheld device. To enable robust operation in confined, GPS-denied environments, the system employs the Google Tango visual-inertial mapping framework a…
Korea Advanced Institute of Science and Technology（韩国科学技术院）；Korea Electronics Technology Institute (KETI)（韩国电子技术研究院）；KI-AI；KIR at KAIST（韩国科学技术院KIR）
This paper presents benchmark tests of various visual(-inertial) odometry algorithms on NVIDIA Jetson platforms. The compared algorithms include mono and stereo, covering Visual Odometry (VO) and Visual-Inertial Odometry (VIO): VINS-Mono, VINS-Fusion, Kimera, ALVIO, Stereo-MSCKF, ORB-SLAM2 stereo, and ROVIO. As these methods are mainly used for unmanned aerial vehicles (UAVs), they must perform well in situations where the size of the processing board and weight is limited. Jetson boards released by NVIDIA satisfy these constraints as they have a sufficiently powerful central processing unit (…
Autonomous Systems Lab, ETH Zurich（苏黎世联邦理工学院自主系统实验室）
This paper proposes a computationally efficient method to estimate the time-varying relative pose between two visual-inertial sensor rigs mounted on the flexible wings of a fixed-wing unmanned aerial vehicle (UAV). The estimated relative poses are used to generate highly accurate depth maps in real-time and can be employed for obstacle avoidance in low-altitude flights or landing maneuvers. The approach is structured as follows: Initially, a wing model is identified by fitting a probability density function to measured deviations from the nominal relative baseline transformation. At run-time, …
Range-Visual-Inertial Sensor Fusion for Micro Aerial Vehicle Localization and Navigation
Technical University of Munich（慕尼黑工业大学）；University of Toronto Institute for Aerospace Studies（多伦多大学航空航天研究所）；University of Toronto Robotics Institute（多伦多大学机器人研究所）；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与机器智能研究所）；Vector Institute for Artificial Intelligence（向量人工智能研究所）
In recent years, autonomous robots have become ubiquitous in research and daily life. Among many factors, public datasets play an important role in the progress of this field, as they waive the tall order of initial investment in hardware and manpower. However, for research on autonomous aerial systems, there appears to be a relative lack of public datasets on par with those used for autonomous driving and ground robots. Thus, to fill in this gap, we conduct a data collection exercise on an aerial platform equipped with an extensive and unique set of sensors: two 3D lidars, two hardware-synchr…
Czech Technical University in Prague（捷克布拉格技术大学）；Federal University of São Carlos（圣卡洛斯联邦大学）
Integration of Visual Inertial Odometry (VIO) methods into a modular control system designed for deployment of Unmanned Aerial Vehicles (UAVs) and teams of cooperating UAVs in real-world conditions are presented in this paper. Reliability analysis and fair performance comparison of several methods integrated into a control pipeline for achieving full autonomy in real conditions is provided. Although most VIO algorithms achieve excellent localization precision and negligible drift on artificially created datasets, the aspects of reliability in non-ideal situations, robustness to degraded sensor…
The main contribution of this paper is a high frequency, low-complexity, on-board visual-inertial odometry system for quadrotor micro air vehicles. The system consists of an extended Kalman filter (EKF) based state estimation algorithm that fuses information from a low cost MEMS inertial measurement unit acquired at 200Hz and VGA resolution images from a monocular camera at 50Hz. The dynamic model describing the quadrotor motion is employed in the estimation algorithm as a third source of information. Visual information is incorporated into the EKF by enforcing the epipolar constraint on featu…
Nanyang Technological University（南洋理工大学）；Institute for Infocomm Research, A-STAR（新加坡科技研究局资讯通信研究院）
Localization can be achieved by different sensors and techniques such as a global positioning system (GPS), wifi, ultrasonic sensors, and cameras. In this paper, we focus on the laser-based localization method for unmanned aerial vehicle (UAV) applications in a GPS denied environment such as a deep tunnel system. Other than a low-cost 2D LiDAR for the planar axes, a single axis Lidar for the vertical axis as well as an inertial measurement unit (IMU) device is used to increase the reliability and accuracy of the localization performance. We present a comparative analysis of the three selected …
With an ever-widening domain of aerial robotic applications, including many mission critical tasks such as disaster response operations, search and rescue missions and infrastructure inspections taking place in GPS-denied environments, the need for reliable autonomous operation of aerial robots has become crucial. Operating in GPS-denied areas aerial robots rely on a multitude of sensors to localize and navigate. Visible spectrum cameras are the most commonly used sensors due to their low cost and weight. However, in environments that are visually-degraded such as in conditions of poor illumin…
Parameter Sensitivity Analysis for Aerial LiDAR-Inertial Odometries in low-altitude flights
CoE MARBLE - Centre of Excellence in Maritime Robotics and Technologies for Sustainable Blue Economy（MARBLE卓越中心 - 可持续蓝色经济海事机器人与技术卓越中心）；Universidad de Sevilla（塞维利亚大学）；University of Zagreb（萨格勒布大学）
Control and State Estimation of Vehicle-Mounted Aerial Systems in GPS-Denied, Non-Inertial Environments
Robotics, Intelligent Systems, and Control Lab, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia（机器人、智能系统与控制实验室，国王阿卜杜勒-阿齐兹大学科学与技术学院（KAUST），沙特阿拉伯图瓦尔）
Fusion of Visual-Inertial Odometry with LiDAR Relative Localization for Cooperative Guidance of a Micro-Scale Aerial Vehicle
Multi-robot Systems Group, Department of Cybernetics, Faculty of Electrical Engineering, Czech Technical University in Prague, Technick\'a 2, Prague, 166 27, Czech Republic
The VINS-Mono is a monocular visual-inertial 6 DOF state estimator proposed by Aerial Robotics Group of HKUST in 2017. It can be performed on MAVs, smartphones and many other intelligent platforms. Because of the excellent robustness, accuracy and scalability, it has gained extensive attention worldwide. In this manuscript, the main equations including IMU pre-integration, visual-inertial co-initialization and tightly-coupled nonlinear optimization are derived and analyzed.
Autonomous Robots Lab, University of Nevada, Reno（内华达大学雷诺分校自主机器人实验室）
This paper proposes a method for tight fusion of visual, depth and inertial data in order to extend robotic capabilities for navigation in GPS-denied, poorly illuminated, and texture-less environments. Visual and depth information are fused at the feature detection and descriptor extraction levels to augment one sensing modality with the other. These multimodal features are then further integrated with inertial sensor cues using an extended Kalman filter to estimate the robot pose, sensor bias terms, and landmark positions simultaneously as part of the filter state. As demonstrated through a s…
Precise geolocalization is crucial for unmanned aerial vehicles (UAVs). However, most current deployed UAVs rely on the global navigation satellite systems (GNSS) or high precision inertial navigation systems (INS) for geolocalization. In this paper, we propose to use a lightweight visual-inertial system with a 2D georeference map to obtain accurate and consecutive geodetic positions for UAVs. The proposed system firstly integrates a micro inertial measurement unit (MIMU) and a monocular camera as odometry to consecutively estimate the navigation states and reconstruct the 3D position of the o…
The flight characteristics of bats remarkably have been overlooked in aerial drone designs. Unlike other animals, bats leverage the manipulation of inertial dynamics to exhibit aerial flip turns when they perch. Inspired by this unique maneuver, this work develops and uses a tiny robot called \textit{Harpoon} to demonstrate that the preparation for upside-down landing is possible through: 1) reorientation towards the landing surface through zero-angular-momentum turns and 2) reaching to the surface through shooting a detachable landing gear. The closed-loop manipulations of inertial dynamics t…
SCAR: Satellite Imagery-Based Calibration for Aerial Recordings
Department of Sensor Data and Information Fusion, Fraunhofer FKIE（传感器数据与信息融合系，弗劳恩霍夫FKIE研究所）；Chair of Astronautics, University of the Bundeswehr Munich（航天学系，联邦国防军 Munich 大学）
As inertial and visual sensors are becoming ubiquitous, visual-inertial navigation systems (VINS) have prevailed in a wide range of applications from mobile augmented reality to aerial navigation to autonomous driving, in part because of the complementary sensing capabilities and the decreasing costs and size of the sensors. In this paper, we survey thoroughly the research efforts taken in this field and strive to provide a concise but complete review of the related work -- which is unfortunately missing in the literature while being greatly demanded by researchers and engineers -- in the hope…
FlyAware: Inertia-Aware Aerial Manipulation via Vision-Based Estimation and Post-Grasp Adaptation
School of Intelligent Systems Engineering, Sun Yat-sen University（中山大学智能系统工程学院）；Visual Intelligence Lab, Cheng Kar-Shun Robotics Institute, The Hong Kong University of Science and Technology（香港科技大学成卡顺机器人研究所视觉智能实验室）；Differential Robotics Technology Company, Ltd.（差分机器人技术有限公司）
Instituto de Investigación en Ingeniería de Aragón (I3A)（阿拉贡工程研究所 (I3A)）；Universidad de Zaragoza（萨拉戈萨大学）
In recent years there have been excellent results in Visual-Inertial Odometry techniques, which aim to compute the incremental motion of the sensor with high accuracy and robustness. However these approaches lack the capability to close loops, and trajectory estimation accumulates drift even if the sensor is continually revisiting the same place. In this work we present a novel tightly-coupled Visual-Inertial Simultaneous Localization and Mapping system that is able to close loops and reuse its map to achieve zero-drift localization in already mapped areas. While our approach can be applied to…
Visual-inertial odometry (VIO) is the pose estimation backbone for most AR/VR and autonomous robotic systems today, in both academia and industry. However, these systems are highly sensitive to the initialization of key parameters such as sensor biases, gravity direction, and metric scale. In practical scenarios where high-parallax or variable acceleration assumptions are rarely met (e.g. hovering aerial robot, smartphone AR user not gesticulating with phone), classical visual-inertial initialization formulations often become ill-conditioned and/or fail to meaningfully converge. In this paper …
The Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
State-of-the-art forward facing monocular visual-inertial odometry algorithms are often brittle in practice, especially whilst dealing with initialisation and motion in directions that render the state unobservable. In such cases having a reliable complementary odometry algorithm enables robust and resilient flight. Using the common local planarity assumption, we present a fast, dense, and direct frame-to-frame visual-inertial odometry algorithm for downward facing cameras that minimises a joint cost function involving a homography based photometric cost and an IMU regularisation term. Via ext…
The University of Tokyo（东京大学）；NASA Jet Propulsion Laboratory, California Institute of Technology（美国国家航空航天局喷气推进实验室、加州理工学院）
Autonomous exploration of unknown environments with aerial vehicles remains a challenge, especially in perceptually degraded conditions. Dust, fog, or a lack of visual or LiDAR-based features results in severe difficulties for state estimation algorithms, which failure can be catastrophic. In this work, we show that it is indeed possible to navigate in such conditions without any exteroceptive sensing by exploiting collisions instead of treating them as constraints. To this end, we present a novel contact-based inertial odometry (CIO) algorithm: it uses estimated external forces with the envir…
In this work, we propose a new learning approach for autonomous navigation and landing of an Unmanned-Aerial-Vehicle (UAV). We develop a multimodal fusion of deep neural architectures for visual-inertial odometry. We train the model in an end-to-end fashion to estimate the current vehicle pose from streams of visual and inertial measurements. We first evaluate the accuracy of our estimation by comparing the prediction of the model to traditional algorithms on the publicly available EuRoC MAV dataset. The results illustrate a $25 \%$ improvement in estimation accuracy over the baseline. Finally…
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
Simultaneous Localization and Mapping (SLAM) is a fundamental task to mobile and aerial robotics. LiDAR based systems have proven to be superior compared to vision based systems due to its accuracy and robustness. In spite of its superiority, pure LiDAR based systems fail in certain degenerate cases like traveling through a tunnel. We propose Stereo Visual Inertial LiDAR (VIL) SLAM that performs better on these degenerate cases and has comparable performance on all other cases. VIL-SLAM accomplishes this by incorporating tightly-coupled stereo visual inertial odometry (VIO) with LiDAR mapping …
Learn Structure, Adapt on the Fly: Multi-Scale Residual Learning and Online Adaptation for Aerial Manipulators
Robotics Research Center, IIIT Hyderabad, India（IIIT Hyderabad机器人研究中心，印度）；Department of Computer Science, University of Manchester, UK（曼彻斯特大学计算机科学系，英国）；Newcastle University, UK（新castle大学，英国）
Modular Adaptive Aerial Manipulation under Unknown Dynamic Coupling Forces
The University of Manchester（曼彻斯特大学）；Czech Technical University in Prague（布拉格捷克技术大学）；International Institute of Information Technology Hyderabad（海德拉巴国际信息技术学院）；Southeast University（东南大学）
This letter proposes a reactive navigation strategy for recovering the altitude, translational velocity and orientation of Micro Aerial Vehicles. The main contribution lies in the direct and tight fusion of Inertial Measurement Unit (IMU) measurements with monocular feedback under an assumption of a single planar scene. An Iterated Extended Kalman Filter (IEKF) scheme is employed. The state prediction makes use of IMU readings while the state update relies directly on photometric feedback as measurements. Unlike feature-based methods, the photometric difference for the innovation term renders …
Zhejiang University（浙江大学）；Huzhou Institute, Zhejiang University（浙江大学湖州研究院）；Alibaba DAMO Academy（阿里巴巴达摩院）
Recently, the community has witnessed numerous datasets built for developing and testing state estimators. However, for some applications such as aerial transportation or search-and-rescue, the contact force or other disturbance must be perceived for robust planning and control, which is beyond the capacity of these datasets. This paper introduces a Visual-Inertial-Dynamical (VID) dataset, not only focusing on traditional six degrees of freedom (6-DOF) pose estimation but also providing dynamical characteristics of the flight platform for external force perception or dynamics-aided estimation.…
Sensor Configuration Matters: A Systematic Evaluation of Multimodal SLAM on Quadruped Robots
Machine Intelligence and Robotics Lab, Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院智能机器人实验室）；Institute for Intelligent Systems, Esslingen University of Applied Sciences（埃森堡应用科学大学智能系统研究所）；Department of Computer Science, University of Freiburg（弗赖堡大学计算机科学系）
