Nano-sized drones, with palm-sized form factor, are gaining relevance in the Internet-of-Things ecosystem. Achieving a high degree of autonomy for complex multi-objective missions (e.g., safe flight, exploration, object detection) is extremely challenging for the onboard chip-set due to tight size, payload (<10g), and power envelope constraints, which strictly limit both memory and computation. Our work addresses this complex problem by combining bio-inspired navigation policies, which rely on time-of-flight distance sensor data, with a vision-based convolutional neural network (CNN) for object detection. Our field-proven nano-drone is equipped with two microcontroller units (MCUs), a single-core ARM Cortex-M4 (STM32) for safe navigation and exploration policies, and a parallel ultra-low power octa-core RISC-V (GAP8) for onboard CNN inference, with a power envelope of just 134mW, including image sensors and external memories. The object detection task achieves a mean average precision of 50% (at 1.6 frame/s) on an in-field collected dataset. We compare four bio-inspired exploration policies and identify a pseudo-random policy to achieve the highest coverage area of 83% in a ~36m^2 unknown room in a 3 minutes flight. By combining the detection CNN and the exploration policy, we show an average detection rate of 90% on six target objects in a never-seen-before environment.
California Institute of Technology（加州理工学院）；Northeastern University（东北大学）
This work presents an actuation framework for a bioinspired flapping drone called Aerobat. This drone, capable of producing dynamically versatile wing conformations, possesses 14 body joints and is tail-less. Therefore, in our robot, unlike mainstream flapping wing designs that are open-loop stable and have no pronounced morphing characteristics, the actuation, and closed-loop feedback design can pose significant challenges. We propose a framework based on integrating mechanical intelligence and control. In this design framework, small adjustments led by several tiny low-power actuators called…
Dalle Molle Institute for Artificial Intelligence, USI-SUPSI（达摩院人工智能研究所，USI-SUPSI）；Integrated Systems Laboratory, ETH Zürich（集成系统实验室，ETH Zurich）；Department of Electrical, Electronic and Information Engineering, University of Bologna（电子、电气与信息工程系，博洛尼亚大学）
Land & Localize: An Infrastructure-free and Scalable Nano-Drones Swarm with UWB-based Localization
University of Bologna（博洛尼亚大学）；Technology Innovation Institute（技术创新研究所）；KU Leuven（荷语鲁汶大学）；ETH Zürich（苏黎世联邦理工学院）；Dalle Molle Institute for Artificial Intelligence (IDSIA)（达勒莫勒人工智能研究所（IDSIA））；USI-SUPSI（瑞士意大利语区大学-瑞士南部应用科学与艺术大学）
Autonomous navigation of drones using computer vision has achieved promising performance. Nano-sized drones based on edge computing platforms are lightweight, flexible, and cheap, thus suitable for exploring narrow spaces. However, due to their extremely limited computing power and storage, vision algorithms designed for high-performance GPU platforms cannot be used for nano drones. To address this issue this paper presents a lightweight CNN depth estimation network deployed on nano drones for obstacle avoidance. Inspired by Knowledge Distillation (KD), a Channel-Aware Distillation Transformer…
High-throughput Visual Nano-drone to Nano-drone Relative Localization using Onboard Fully Convolutional Networks
Dalle Molle Institute for Artificial Intelligence (IDSIA)（戴尔·莫勒人工智能研究所）；USI-SUPSI（瑞士意大利语区大学与瑞士应用科学大学联盟）；Integrated Systems Laboratory (IIS), ETH Zürich（苏黎世联邦理工学院集成系统实验室）
Biologically-inspired robots are a very interesting and difficult branch of robotics dues to its very rich dynamical and morphological complexities. Among them, flying animals, such as bats, have been among the most difficult to take inspiration from as they exhibit complex wing articulation. We attempt to capture several of the key degrees-of-freedom that are present in the natural flapping gait of a bat. In this work, we present the mechanical design and analysis of our flapping wing robot, the Aerobat, where we capture the plunging and flexion-extension in the bat's flapping modes. This rob…
Multi-sensory Anti-collision Design for Autonomous Nano-swarm Exploration
University of Bologna（博洛尼亚大学）；KU Leuven（鲁汶大学）；ETH Zürich（苏黎世联邦理工学院）；Dalle Molle Institute for Artificial Intelligence (IDSIA), USI-SUPSI（USI-SUPSI 达尔·莫勒人工智能研究所（IDSIA））
Flocking phase transition and threat responses in bio-inspired autonomous drone swarms
École Nationale de l’Aviation Civile, Université de Toulouse, France（法国国家民用航空学院，图卢兹大学）；Centre de Recherches sur la Cognition Animale, Centre de Biologie Intégrative (CBI), Centre National de la Recherche Scientifique (CNRS) & Université de Toulouse – Paul Sabatier, France（动物认知研究中心，整合生物学研究所（CBI），法国国家科学研究中心（CNRS）及图卢兹大学–保罗·萨巴蒂埃大学，法国）；Laboratoire de Physique Théorique, CNRS & Université de Toulouse – Paul Sabatier, France（理论物理实验室，法国国家科学研究中心（CNRS）及图卢兹大学–保罗·萨巴蒂埃大学，法国）
Tiny-PULP-Dronets: Squeezing Neural Networks for Faster and Lighter Inference on Multi-Tasking Autonomous Nano-Drones
University of Bologna（博洛尼亚大学）；ETH Zürich（苏黎世联邦理工学院）；Technology Innovation Institute（技术创新研究院）；Dalle Molle Institute for Artificial Intelligence（达勒·莫勒人工智能研究所）
Exploring Unstructured Environments using Minimal Sensing on Cooperative Nano-Drones
Computer Vision and Aerial Robotics Group at Universidad Politécnica de Madrid（马德里理工大学计算机视觉与空中机器人小组）；Centre for Automation and Robotics C.A.R. (UPM-CSIC)（自动化与机器人中心（UPM-CSIC））；Department of Mechanical Engineering, Texas A&M University（机械工程系，德克萨斯A&M大学）
Fully Onboard AI-powered Human-Drone Pose Estimation on Ultra-low Power Autonomous Flying Nano-UAVs
Dalle Molle Institute for Artificial Intelligence (IDSIA), University of Lugano & SUPSI（达勒·莫勒人工智能研究所（IDSIA），卢加诺大学与瑞士南部应用科学大学）；University of Bologna（博洛尼亚大学）；ETH Zürich（苏黎世联邦理工学院）
Self-supervised Domain Adaptation for Visual 3D Pose Estimation of Nano-drone Racing Gates by Enforcing Geometric Consistency
Dalle Molle Institute for Artificial Intelligence (IDSIA)（达勒莫利人工智能研究所）；USI-SUPSI；University of Milan（米兰大学）；Integrated Systems Laboratory (IIS)（集成系统实验室）；ETHZ（苏黎世联邦理工学院）
Soft Haptics Lab., School of Materials Science, Japan Advanced Institute of Science and Technology（日本先进科学技术学院材料科学学院软触觉实验室）；Tandon School of Engineering, New York University（纽约大学坦登工程学院）；Japan Science and Technology Agency, PRESTO（日本科学技术厅）
There is a growing need for vertical take-off and landing vehicles, including drones, which are safe to use and can adapt to collisions. The risks of damage by collision, to humans, obstacles in the environment, and drones themselves, are significant. This has prompted a search into nature for a highly resilient structure that can inform a design of propellers to reduce those risks and enhance safety. Inspired by the flexibility and resilience of dragonfly wings, we propose a novel design for a biomimetic drone propeller called Tombo propeller. Here, we report on the design and fabrication pro…
Enabling the capability of assessing risk and making risk-aware decisions is essential to applying reinforcement learning to safety-critical robots like drones. In this paper, we investigate a specific case where a nano quadcopter robot learns to navigate an apriori-unknown cluttered environment under partial observability. We present a distributional reinforcement learning framework to generate adaptive risk-tendency policies. Specifically, we propose to use lower tail conditional variance of the learnt return distribution as intrinsic uncertainty estimation, and use exponentially weighted av…
This paper demonstrates for the first time that a biologically-plausible spiking neural network (SNN) equipped with Spike-Timing-Dependent Plasticity (STDP) can continuously learn to detect walking people on the fly using retina-inspired, event-based cameras. Our pipeline works as follows. First, a short sequence of event data ($<2$ minutes), capturing a walking human by a flying drone, is forwarded to a convolutional SNNSTDP system which also receives teacher spiking signals from a readout (forming a semi-supervised system). Then, STDP adaptation is stopped and the learned system is assessed …
Harvard University（哈佛大学）；Delft University of Technology（代尔夫特理工大学）；Robotics at Google（谷歌机器人部门）
We present fully autonomous source seeking onboard a highly constrained nano quadcopter, by contributing application-specific system and observation feature design to enable inference of a deep-RL policy onboard a nano quadcopter. Our deep-RL algorithm finds a high-performance solution to a challenging problem, even in presence of high noise levels and generalizes across real and simulation environments with different obstacle configurations. We verify our approach with simulation and in-field testing on a Bitcraze CrazyFlie using only the cheap and ubiquitous Cortex-M4 microcontroller unit. T…
Vision-based indoor localization of nano drones in controlled environment with its applications
Electrical Engineering Department, Indian Institute of Technology Bombay（印度理工学院班加罗尔分校电子工程系）；Embedded Real-Time Systems / e-Yantra Lab, Indian Institute of Technology Bombay（印度理工学院班加罗尔分校嵌入式实时系统/ e-Yantra实验室）；Computer Science and Engineering Department, Indian Institute of Technology Bombay（印度理工学院班加罗尔分校计算机科学与工程系）
Safety in autonomous systems has been mostly studied from a human-centered perspective. Besides the loads they may carry, autonomous systems are also valuable property, and self-preservation mechanisms are needed to protect them in the presence of external threats, including malicious robots and antagonistic humans. We present a biologically inspired risk-based triggering mechanism to initiate self-preservation strategies. This mechanism considers environmental and internal system factors to measure the overall risk at any moment in time, to decide whether behaviours such as fleeing or hiding …
Ecole Polytechnique Federale de Lausanne（洛桑联邦理工学院）；The University of Electro-Communications（电气通信大学）；Wageningen University and Research（瓦赫宁根大学及研究中心）
Drones have shown to be useful aerial vehicles for unmanned transport missions such as food and medical supply delivery. This can be leveraged to deliver life-saving nutrition and medicine for people in emergency situations. However, commercial drones can generally only carry 10 % - 30 % of their own mass as payload, which limits the amount of food delivery in a single flight. One novel solution to noticeably increase the food-carrying ratio of a drone, is recreating some structures of a drone, such as the wings, with edible materials. We thus propose a drone, which is no longer only a food tr…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
Reinforcement learning (RL) methods have been actively applied in the field of robotics, allowing the system itself to find a solution for a task otherwise requiring a complex decision-making algorithm. In this paper, we present a novel RL-based Tic-tac-toe scenario, i.e. SwarmPlay, where each playing component is presented by an individual drone that has its own mobility and swarm intelligence to win against a human player. Thus, the combination of challenging swarm strategy and human-drone collaboration aims to make the games with machines tangible and interactive. Although some research on …
Rigid grippers used in existing aerial manipulators require precise positioning to achieve successful grasps and transmit large contact forces that may destabilize the drone. This limits the speed during grasping and prevents "dynamic grasping", where the drone attempts to grasp an object while moving. On the other hand, biological systems (e.g., birds) rely on compliant and soft parts to dampen contact forces and compensate for grasping inaccuracy, enabling impressive feats. This paper presents the first prototype of a soft drone -- a quadrotor where traditional (i.e., rigid) landing gears ar…
The objective of this work is to design and develop a bio-inspired soft and articulated armwing structure which will be an integral component of a morphing aerial co-bot, Aerobat. In our design, we draw inspiration from bats. Bat membranous wings possess unique functions that make them a good example to take inspiration from and transform current aerial drones. In contrast with other flying vertebrates, bats have an extremely articulated musculoskeletal system, key to their body impact-survivability and deliver an impressively adaptive and multimodal locomotion behavior. Bats exclusively use t…
We propose a novel system SwarmCloak for landing of a fleet of four flying robots on the human arms using light-sensitive landing pads with vibrotactile feedback. We developed two types of wearable tactile displays with vibromotors which are activated by the light emitted from the LED array at the bottom of quadcopters. In a user study, participants were asked to adjust the position of the arms to land up to two drones, having only visual feedback, only tactile feedback or visual-tactile feedback. The experiment revealed that when the number of drones increases, tactile feedback plays a more i…
Ultra-low Power Deep Learning-based Monocular Relative Localization Onboard Nano-quadrotors
Dalle Molle Institute for Artificial Intelligence, USI and SUPSI（达勒·莫勒人工智能研究所（瑞士意大利语区大学与瑞士应用科学大学））；Integrated Systems Laboratory, ETH Zürich（苏黎世联邦理工学院集成系统实验室）
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Laboratory of Intelligent Systems (LIS)（智能系统实验室）
Among the available solutions for drone swarm simulations, we identified a gap in simulation frameworks that allow easy algorithms prototyping, tuning, debugging and performance analysis, and do not require the user to interface with multiple programming languages. We present SwarmLab, a software entirely written in Matlab, that aims at the creation of standardized processes and metrics to quantify the performance and robustness of swarm algorithms, and in particular, it focuses on drones. We showcase the functionalities of SwarmLab by comparing two state-of-the-art algorithms for the navigati…
Anthropomorphic robot avatars present a conceptually novel approach to remote affective communication, allowing people across the world a wider specter of emotional and social exchanges over traditional 2D and 3D image data. However, there are several limitations of current telepresence robots, such as the high weight, complexity of the system that prevents its fast deployment, and the limited workspace of the avatars mounted on either static or wheeled mobile platforms. In this paper, we present a novel concept of telecommunication through a robot avatar based on an anthropomorphic swarm of d…
Using a game engine, we have developed a virtual environment which models important aspects of critical incident scenarios. We focused on modelling phenomena relating to the identification and gathering of key forensic evidence, in order to develop and test a system which can handle chemical, biological, radiological/nuclear or explosive (CBRNe) events autonomously. This allows us to build and validate AI-based technologies, which can be trained and tested in our custom virtual environment before being deployed in real-world scenarios. We have used our virtual scenario to rapidly prototype a s…
Biological sensing and processing is asynchronous and sparse, leading to low-latency and energy-efficient perception and action. In robotics, neuromorphic hardware for event-based vision and spiking neural networks promises to exhibit similar characteristics. However, robotic implementations have been limited to basic tasks with low-dimensional sensory inputs and motor actions due to the restricted network size in current embedded neuromorphic processors and the difficulties of training spiking neural networks. Here, we present the first fully neuromorphic vision-to-control pipeline for contro…
Learning on the Fly: Replay-Based Continual Object Perception for Indoor Drones
National University of Science and Technology Politehnica Bucharest, Romania（罗马尼亚布加勒斯特多项技术大学）；Institute of Mathematics ``Simion Stoilow'' of the Romanian Academy, Romania（罗马尼亚科学院模拟昂斯数学研究所）；NORCE Norwegian Research Center, Norway（挪威NORCE研究中心）
A Sim-to-Real Deep Learning-based Framework for Autonomous Nano-drone Racing
University of Bologna（博洛尼亚大学）；Dalle Molle Institute for Artificial Intelligence, USI-SUPSI（USI-SUPSI 达勒·莫勒人工智能研究所）；Autonomous Robotics Research Center, Technology Innovation Institute（技术创新研究院自主机器人研究中心）；ETH Zürich（苏黎世联邦理工学院）
ETH Zürich（苏黎世联邦理工学院）；University of Zürich（苏黎世大学）；University of Bologna（博洛尼亚大学）；GreenWaves Technologies
Fully-autonomous miniaturized robots (e.g., drones), with artificial intelligence (AI) based visual navigation capabilities are extremely challenging drivers of Internet-of-Things edge intelligence capabilities. Visual navigation based on AI approaches, such as deep neural networks (DNNs) are becoming pervasive for standard-size drones, but are considered out of reach for nanodrones with size of a few cm${}^\mathrm{2}$. In this work, we present the first (to the best of our knowledge) demonstration of a navigation engine for autonomous nano-drones capable of closed-loop end-to-end DNN-based vi…
This paper presents Navion, an energy-efficient accelerator for visual-inertial odometry (VIO) that enables autonomous navigation of miniaturized robots (e.g., nano drones), and virtual/augmented reality on portable devices. The chip uses inertial measurements and mono/stereo images to estimate the drone's trajectory and a 3D map of the environment. This estimate is obtained by running a state-of-the-art VIO algorithm based on non-linear factor graph optimization, which requires large irregularly structured memories and heterogeneous computation flow. To reduce the energy consumption and footp…
Fully Onboard SLAM for Distributed Mapping with a Swarm of Nano-Drones
Department of Information Technology and Electrical Engineering (D-ITET), ETH Zürich（信息科技与电气工程系（D-ITET），苏黎世联邦理工学院）；Department of Electrical, Electronic and Information Engineering (DEI), University of Bologna（电气、电子与信息工程系（DEI），博洛尼亚大学）
A Map-free Deep Learning-based Framework for Gate-to-Gate Monocular Visual Navigation aboard Miniaturized Aerial Vehicles
Dalle Molle Institute for Artificial Intelligence (IDSIA)（达勒·莫勒人工智能研究所）；USI-SUPSI（瑞士意大利语区大学与瑞士应用科学大学联盟）；Integrated Systems Laboratory (IIS)（集成系统实验室）；ETHZ（苏黎世联邦理工学院）
