The miniaturisation of sensors and processors, the advancements in connected edge intelligence, and the exponential interest in Artificial Intelligence are boosting the affirmation of autonomous nano-size drones in the Internet of Robotic Things ecosystem. However, achieving safe autonomous navigation and high-level tasks such as exploration and surveillance with these tiny platforms is extremely challenging due to their limited resources. This work focuses on enabling the safe and autonomous flight of a pocket-size, 30-gram platform called Crazyflie 2.1 in a partially known environment. We propose a novel AI-aided, vision-based reactive planning method for obstacle avoidance under the ambit of Integrated Sensing, Computing and Communication paradigm. We deal with the constraints of the nano-drone by splitting the navigation task into two parts: a deep learning-based object detector runs on the edge (external hardware) while the planning algorithm is executed onboard. The results show the ability to command the drone at $\sim8$ frames-per-second and a model performance reaching a COCO mean-average-precision of $60.8$. Field experiments demonstrate the feasibility of the solution with the drone flying at a top speed of $1$ m/s while steering away from an obstacle placed in an unknown position and reaching the target destination. The outcome highlights the compatibility of the communication delay and the model performance with the requirements of the real-time navigation task. We provide a feasible alternative to a fully onboard implementation that can be extended to autonomous exploration with nano-drones.
The Perspective-n-Point problem aims to estimate the relative pose between a calibrated monocular camera and a known 3D model, by aligning pairs of 2D captured image points to their corresponding 3D points in the model. We suggest an algorithm that runs on weak IoT devices in real-time but still provides provable theoretical guarantees for both running time and correctness. Existing solvers provide only one of these requirements. Our main motivation was to turn the popular DJI's Tello Drone (<90gr, <\$100) into an autonomous drone that navigates in an indoor environment with no external human/…
Delft University of Technology（代尔夫特理工大学）；Micro Air Vehicle lab, Control and Simulation, Delft University of Technology（代尔夫特理工大学控制与仿真系微型飞行器实验室）
Drone racing is becoming a popular sport where human pilots have to control their drones to fly at high speed through complex environments and pass a number of gates in a pre-defined sequence. In this paper, we develop an autonomous system for drones to race fully autonomously using only onboard resources. Instead of commonly used visual navigation methods, such as simultaneous localization and mapping and visual inertial odometry, which are computationally expensive for micro aerial vehicles (MAVs), we developed the highly efficient snake gate detection algorithm for visual navigation, which …
ETH Zürich（苏黎世联邦理工学院）；University of Zürich（苏黎世大学）；University of Bologna（博洛尼亚大学）；GreenWaves Technologies
Fully-autonomous miniaturized robots (e.g., drones), with artificial intelligence (AI) based visual navigation capabilities are extremely challenging drivers of Internet-of-Things edge intelligence capabilities. Visual navigation based on AI approaches, such as deep neural networks (DNNs) are becoming pervasive for standard-size drones, but are considered out of reach for nanodrones with size of a few cm${}^\mathrm{2}$. In this work, we present the first (to the best of our knowledge) demonstration of a navigation engine for autonomous nano-drones capable of closed-loop end-to-end DNN-based vi…
Universidade Federal de Minas Gerais（米纳斯吉拉斯联邦大学）；Instituto Tecnológico Vale (ITV)（淡水河谷技术研究所）
The use of delivery services is an increasing trend worldwide, further enhanced by the COVID pandemic. In this context, drone delivery systems are of great interest as they may allow for faster and cheaper deliveries. This paper presents a navigation system that makes feasible the delivery of parcels with autonomous drones. The system generates a path between a start and a final point and controls the drone to follow this path based on its localization obtained through GPS, 9DoF IMU, and barometer. In the landing phase, information of poses estimated by a marker (ArUco) detection technique usi…
Technische Universität München（慕尼黑工业大学）；Oculus Research（奥库鲁斯研究院）；RWTH Aachen University（亚琛工业大学）
Micro aerial vehicles (MAVs) are strongly limited in their payload and power capacity. In order to implement autonomous navigation, algorithms are therefore desirable that use sensory equipment that is as small, low-weight, and low-power consuming as possible. In this paper, we propose a method for autonomous MAV navigation and exploration using a low-cost consumer-grade quadrocopter equipped with a monocular camera. Our vision-based navigation system builds on LSD-SLAM which estimates the MAV trajectory and a semi-dense reconstruction of the environment in real-time. Since LSD-SLAM only deter…
Aerial navigation in GPS-denied, indoor environments, is still an open challenge. Drones can perceive the environment from a richer set of viewpoints, while having more stringent compute and energy constraints than other autonomous platforms. To tackle that problem, this research displays a biologically inspired deep-learning algorithm for simultaneous localization and mapping (SLAM) and its application in a drone navigation system. We propose an unsupervised representation learning method that yields low-dimensional latent state descriptors, that mitigates the sensitivity to perceptual aliasi…
Zewail City of Science & Technology（扎耶德科学与技术城）；INRIA Sophia Antipolis - Méditerranée（法国国家信息与自动化研究所索菲亚·安蒂波利斯-地中海分院）；Institute of Aviation Engineering and Technology（航空工程与技术学院）
Mobile robotics is a research area that has witnessed incredible advances for the last decades. Robot navigation is an essential task for mobile robots. Many methods are proposed for allowing robots to navigate within different environments. This thesis studies different deep learning-based approaches, highlighting the advantages and disadvantages of each scheme. In fact, these approaches are promising that some of them can navigate the robot in unknown and dynamic environments. In this thesis, one of the deep learning methods based on convolutional neural network (CNN) is realized by software…
This paper presents Navion, an energy-efficient accelerator for visual-inertial odometry (VIO) that enables autonomous navigation of miniaturized robots (e.g., nano drones), and virtual/augmented reality on portable devices. The chip uses inertial measurements and mono/stereo images to estimate the drone's trajectory and a 3D map of the environment. This estimate is obtained by running a state-of-the-art VIO algorithm based on non-linear factor graph optimization, which requires large irregularly structured memories and heterogeneous computation flow. To reduce the energy consumption and footp…
NASA-Jet Propulsion Laboratory, California Institute of Technology（美国国家航空航天局喷气推进实验室，加州理工学院）
Robots and particularly drones are especially useful in exploring extreme environments that pose hazards to humans. To ensure safe operations in these situations, usually perceptually degraded and without good GNSS, it is critical to have a reliable and robust state estimation solution. The main body of literature in robot state estimation focuses on developing complex algorithms favoring accuracy. Typically, these approaches rely on a strong underlying assumption: the main estimation engine will not fail during operation. In contrast, we propose an architecture that pursues robustness in stat…
A Map-free Deep Learning-based Framework for Gate-to-Gate Monocular Visual Navigation aboard Miniaturized Aerial Vehicles
Dalle Molle Institute for Artificial Intelligence (IDSIA)（达勒·莫勒人工智能研究所）；USI-SUPSI（瑞士意大利语区大学与瑞士应用科学大学联盟）；Integrated Systems Laboratory (IIS)（集成系统实验室）；ETHZ（苏黎世联邦理工学院）
Simultaneous Localization and 3D-Semi Dense Mapping for Micro Drones Using Monocular Camera and Inertial Sensors
The Hatter Department of Marine Technologies, University of Haifa, 3498838, Israel（海洋技术系，海法大学）；Computer Science, University of Haifa, 3498838, Israel（计算机科学，海法大学）
Toshiba Europe Limited（东芝欧洲有限公司）；University of Cambridge（剑桥大学）
We present NavACL, a method of automatic curriculum learning tailored to the navigation task. NavACL is simple to train and efficiently selects relevant tasks using geometric features. In our experiments, deep reinforcement learning agents trained using NavACL significantly outperform state-of-the-art agents trained with uniform sampling -- the current standard. Furthermore, our agents can navigate through unknown cluttered indoor environments to semantically-specified targets using only RGB images. Obstacle-avoiding policies and frozen feature networks support transfer to unseen real-world en…
Survey Copter / Airbus Defense and Space（空客防务与航天公司Survey Copter部门）；Hubert Curien Laboratory（于贝尔·居里安实验室）；Center for Research in Computer Vision, UCF（中佛罗里达大学计算机视觉研究中心）
In recent years, consumer Unmanned Aerial Vehicles have become very popular, everyone can buy and fly a drone without previous experience, which raises concern in regards to regulations and public safety. In this paper, we present a novel approach towards enabling safe operation of such vehicles in urban areas. Our method uses geodetically accurate dataset images with Geographical Information System (GIS) data of road networks and buildings provided by Google Maps, to compute a weighted A* shortest path from start to end locations of a mission. Weights represent the potential risk of injuries …
Scalable Outdoors Autonomous Drone Flight with Visual-Inertial SLAM and Dense Submaps Built without LiDAR
Technical University of Munich（技术大学慕尼黑）；School of Computation, Information and Technology（计算、信息与技术学院）；Mobile Robotics Lab（移动机器人实验室）；Department of Computing（计算学院）；Imperial College London（伦敦帝国学院）；Department of Mechanical and Process Engineering（机械与过程工程学院）；ETH Zurich（苏黎世联邦理工学院）；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与机器智能研究所）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
Korea Advanced Institute of Science and Technology（韩国科学技术院）
This paper presents a vision-based modularized drone racing navigation system that uses a customized convolutional neural network (CNN) for the perception module to produce high-level navigation commands and then leverages a state-of-the-art planner and controller to generate low-level control commands, thus exploiting the advantages of both data-based and model-based approaches. Unlike the state-of-the-art method which only takes the current camera image as the CNN input, we further add the latest three drone states as part of the inputs. Our method outperforms the state-of-the-art method in …
Fully Onboard SLAM for Distributed Mapping with a Swarm of Nano-Drones
Department of Information Technology and Electrical Engineering (D-ITET), ETH Zürich（信息科技与电气工程系（D-ITET），苏黎世联邦理工学院）；Department of Electrical, Electronic and Information Engineering (DEI), University of Bologna（电气、电子与信息工程系（DEI），博洛尼亚大学）
This paper reports on the development, execution, and open-sourcing of a new robotics course at MIT. The course is a modern take on "Visual Navigation for Autonomous Vehicles" (VNAV) and targets first-year graduate students and senior undergraduates with prior exposure to robotics. VNAV has the goal of preparing the students to perform research in robotics and vision-based navigation, with emphasis on drones and self-driving cars. The course spans the entire autonomous navigation pipeline; as such, it covers a broad set of topics, including geometric control and trajectory optimization, 2D and…
ETH Zürich（苏黎世联邦理工学院）；University of Bologna（博洛尼亚大学）
Nano-size unmanned aerial vehicles (UAVs), with few centimeters of diameter and sub-10 Watts of total power budget, have so far been considered incapable of running sophisticated visual-based autonomous navigation software without external aid from base-stations, ad-hoc local positioning infrastructure, and powerful external computation servers. In this work, we present what is, to the best of our knowledge, the first 27g nano-UAV system able to run aboard an end-to-end, closed-loop visual pipeline for autonomous navigation based on a state-of-the-art deep-learning algorithm, built upon the op…
In this work we showcase the design and assessment of the performance of a multi-camera UAV, when coupled with state-of-the-art planning and mapping algorithms for autonomous navigation. The system leverages state-of-the-art receding horizon exploration techniques for Next-Best-View (NBV) planning with 3D and semantic information, provided by a reconfigurable multi stereo camera system. We employ our approaches in an autonomous drone-based inspection task and evaluate them in an autonomous exploration and mapping scenario. We discuss the advantages and limitations of using multi stereo camera …
One of the biggest challenges is to determine the deployment and navigation of the drones to benefit the most for different applications. Many research questions have been raised about this topic. For example, drone-enabled wildlife monitoring has received much attention in recent years. Unfortunately, this approach results in significant disturbance to different species of wild animals. Moreover, with the capability of rapidly moving communication supply towards demand when required, the drone equipped with a base station, i.e., drone-cell, is becoming a promising solution for providing cellu…
In this paper, we present a control system that allows a drone to fly autonomously through a series of gates marked with ArUco tags. A simple and low-cost DJI Tello EDU quad-rotor platform was used. Based on the API provided by the manufacturer, we have created a Python application that enables the communication with the drone over WiFi, realises drone positioning based on visual feedback, and generates control. Two control strategies were proposed, compared, and critically analysed. In addition, the accuracy of the positioning method used was measured. The application was evaluated on a lapto…
Indian Institute of Technology, Gandhinagar（印度理工学院甘地纳格尔分校）
This paper proposes a novel framework for autonomous drone navigation through a cluttered environment. Control policies are learnt in a low-level environment during training and are applied to a complex environment during inference. The controller learnt in the training environment is tricked into believing that the robot is still in the training environment when it is actually navigating in a more complex environment. The framework presented in this paper can be adapted to reuse simple policies in more complex tasks. We also show that the framework can be used as an interpretation tool for re…
Enabling the capability of assessing risk and making risk-aware decisions is essential to applying reinforcement learning to safety-critical robots like drones. In this paper, we investigate a specific case where a nano quadcopter robot learns to navigate an apriori-unknown cluttered environment under partial observability. We present a distributional reinforcement learning framework to generate adaptive risk-tendency policies. Specifically, we propose to use lower tail conditional variance of the learnt return distribution as intrinsic uncertainty estimation, and use exponentially weighted av…
AirDreamer: Generalist Drone Navigation with World Models
Institute for AI Industry Research, Tsinghua University, Beijing, China（人工智能产业研究院，清华大学，北京，中国）；Department of Electronic Engineering, Tsinghua University, Beijing, China（电子工程系，清华大学，北京，中国）；School of Remote Sensing and Information Engineering, Wuhan University, Wuhan, China（遥感与信息工程学院，武汉大学，武汉，中国）
Event-based vision has already revolutionized the perception task for robots by promising faster response, lower energy consumption, and lower bandwidth without introducing motion blur. In this work, a novel deep learning method based on gated recurrent units utilizing sparse convolutions for detecting gates in a race track is proposed using event-based vision for the autonomous drone racing problem. We demonstrate the efficiency and efficacy of the perception pipeline on a real robot platform that can safely navigate a typical autonomous drone racing track in real-time. Throughout the experim…
Autonomous navigation of drones using computer vision has achieved promising performance. Nano-sized drones based on edge computing platforms are lightweight, flexible, and cheap, thus suitable for exploring narrow spaces. However, due to their extremely limited computing power and storage, vision algorithms designed for high-performance GPU platforms cannot be used for nano drones. To address this issue this paper presents a lightweight CNN depth estimation network deployed on nano drones for obstacle avoidance. Inspired by Knowledge Distillation (KD), a Channel-Aware Distillation Transformer…
Tiny-PULP-Dronets: Squeezing Neural Networks for Faster and Lighter Inference on Multi-Tasking Autonomous Nano-Drones
University of Bologna（博洛尼亚大学）；ETH Zürich（苏黎世联邦理工学院）；Technology Innovation Institute（技术创新研究院）；Dalle Molle Institute for Artificial Intelligence（达勒·莫勒人工智能研究所）
Building upon the theory of Kalman Filtering on Lie Groups, this paper describes an Extended Kalman Filter and Smoother for Loosely Coupled Integration of GNSS/INS tailored for post-processing applications. The approach employs a dynamic model on a matrix Lie Group that aggregates position, velocity, attitude, and the IMU biases as a single element of a Lie group. The development was motivated by a drone-borne Differential Interferometric SAR (DinSAR) application, which requires high-precision navigation information for short-flight missions using low-cost MEMS sensors. The filter and the Rauc…
High-throughput Visual Nano-drone to Nano-drone Relative Localization using Onboard Fully Convolutional Networks
Dalle Molle Institute for Artificial Intelligence (IDSIA)（戴尔·莫勒人工智能研究所）；USI-SUPSI（瑞士意大利语区大学与瑞士应用科学大学联盟）；Integrated Systems Laboratory (IIS), ETH Zürich（苏黎世联邦理工学院集成系统实验室）
