This paper demonstrates for the first time that a biologically-plausible spiking neural network (SNN) equipped with Spike-Timing-Dependent Plasticity (STDP) can continuously learn to detect walking people on the fly using retina-inspired, event-based cameras. Our pipeline works as follows. First, a short sequence of event data ($<2$ minutes), capturing a walking human by a flying drone, is forwarded to a convolutional SNNSTDP system which also receives teacher spiking signals from a readout (forming a semi-supervised system). Then, STDP adaptation is stopped and the learned system is assessed on testing sequences. We conduct several experiments to study the effect of key parameters in our system and to compare it against conventionally-trained CNNs. We show that our system reaches a higher peak $F_1$ score (+19%) compared to CNNs with event-based camera frames, while enabling on-line adaptation.
Zhejiang University（浙江大学）；University of Pennsylvania（宾夕法尼亚大学）
Current motion capture (MoCap) systems generally require markers and multiple calibrated cameras, which can be used only in constrained environments. In this work we introduce a drone-based system for 3D human MoCap. The system only needs an autonomously flying drone with an on-board RGB camera and is usable in various indoor and outdoor environments. A reconstruction algorithm is developed to recover full-body motion from the video recorded by the drone. We argue that, besides the capability of tracking a moving subject, a flying drone also provides fast varying viewpoints, which is beneficia…
We demonstrate formation flying for drone swarm services. A set of drones fly in four different swarm formations. A dataset is collected to study the effect of formation flying on energy consumption. We conduct a set of experiments to study the effect of wind on formation flying. We examine the forces the drones exert on each other when flying in a formation. We finally identify and classify the formations that conserve most energy under varying wind conditions. The collected dataset aims at providing researchers data to conduct further research in swarm-based drone service delivery. Demo: htt…
Indian Institute of Science（印度科学学院）；Robert Bosch Center for Cyber Physical Systems（罗伯特·博世网络物理系统中心）
In this paper, we present a novel passive single Degree-of-Freedom (DoF) manipulator design and its integration on an autonomous drone to capture a moving target. The end-effector is designed to be passive, to disengage the moving target from a flying UAV and capture it efficiently in the presence of disturbances, with minimal energy usage. It is also designed to handle target sway and the effect of downwash. The passive manipulator is integrated with the drone through a single Degree of Freedom (DoF) arm, and experiments are carried out in an outdoor environment. The rack-and-pinion mechanism…
Learning on the Fly: Replay-Based Continual Object Perception for Indoor Drones
National University of Science and Technology Politehnica Bucharest, Romania（罗马尼亚布加勒斯特多项技术大学）；Institute of Mathematics ``Simion Stoilow'' of the Romanian Academy, Romania（罗马尼亚科学院模拟昂斯数学研究所）；NORCE Norwegian Research Center, Norway（挪威NORCE研究中心）
Lacking the ability to sense ambient environments effectively, blind and visually impaired people (BVIP) face difficulty in walking outdoors, especially in urban areas. Therefore, tools for assisting BVIP are of great importance. In this paper, we propose a novel "flying guide dog" prototype for BVIP assistance using drone and street view semantic segmentation. Based on the walkable areas extracted from the segmentation prediction, the drone can adjust its movement automatically and thus lead the user to walk along the walkable path. By recognizing the color of pedestrian traffic lights, our p…
KAIST（韩国科学技术院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Helsinki（赫尔辛基大学）；Fudan University（复旦大学）；The Hong Kong University of Science and Technology（香港科技大学）
Research attention on natural user interfaces (NUIs) for drone flights are rising. Nevertheless, NUIs are highly diversified, and primarily evaluated by different physical environments leading to hard-to-compare performance between such solutions. We propose a virtual environment, namely VRFlightSim, enabling comparative evaluations with enriched drone flight details to address this issue. We first replicated a state-of-the-art (SOTA) interface and designed two tasks (crossing and pointing) in our virtual environment. Then, two user studies with 13 participants demonstrate the necessity of VRF…
A highly maneuverable flying squirrel drone with agility-improving foldable wings
Department of Convergence IT Engineering, Pohang University of Science and Technology (POSTECH)（融合IT工程系，浦项科学技术大学）；Defence AI Center, Agency for Defense Development(ADD)（国防人工智能中心，国防开发机构）
We present NAVREN-RL, an approach to NAVigate an unmanned aerial vehicle in an indoor Real ENvironment via end-to-end reinforcement learning RL. A suitable reward function is designed keeping in mind the cost and weight constraints for micro drone with minimum number of sensing modalities. Collection of small number of expert data and knowledge based data aggregation is integrated into the RL process to aid convergence. Experimentation is carried out on a Parrot AR drone in different indoor arenas and the results are compared with other baseline technologies. We demonstrate how the drone succe…
Soft Haptics Lab., School of Materials Science, Japan Advanced Institute of Science and Technology（日本先进科学技术学院材料科学学院软触觉实验室）；Tandon School of Engineering, New York University（纽约大学坦登工程学院）；Japan Science and Technology Agency, PRESTO（日本科学技术厅）
There is a growing need for vertical take-off and landing vehicles, including drones, which are safe to use and can adapt to collisions. The risks of damage by collision, to humans, obstacles in the environment, and drones themselves, are significant. This has prompted a search into nature for a highly resilient structure that can inform a design of propellers to reduce those risks and enhance safety. Inspired by the flexibility and resilience of dragonfly wings, we propose a novel design for a biomimetic drone propeller called Tombo propeller. Here, we report on the design and fabrication pro…
Fly360: Omnidirectional Obstacle Avoidance within Drone View
Institute of Automation Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所）；Insta360 Research, Shenzhen, China（Insta360研究院）；Wuhan University,Wuhan, China（武汉大学）
Flying Robotics Art: ROS-based Drone Draws the Record-Breaking Mural
Lomonosov Moscow State University（莫斯科国立罗蒙诺索夫大学）；Copter Express Technologies Ltd.（Copter Express 技术有限公司）；Sverk Ltd.（Sverk 有限公司）；Cognitive Pilot Ltd.（Cognitive Pilot 有限公司）
Current drone delivery systems mostly focus on point-to-point package delivery. We present a multi-stop drone service system to deliver packages anywhere anytime within a specified geographic area. We define a skyway network which takes into account flying regulations, including restricted areas and no-fly zones. The skyway nodes typically represent building rooftops which may act as both recharging stations and delivery destinations. A heuristic-based A* algorithm is used to compute an optimal path from source to destination taking into account a number of constraints, including delivery time…
FlyCo: Foundation Model-Empowered Drones for Autonomous 3D Structure Scanning in Open-World Environments
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（香港科技大学电子与计算机工程系）；School of Artificial Intelligence, Sun Yat-sen University（中山大学人工智能学院）；Department of Mechanical and Energy Engineering, Southern University of Science and Technology（南方科技大学机械与能源工程系）；Differential Robotics, Hangzhou, China（杭州差分机器人）
Zhejiang University（浙江大学）；Zhejiang University of Technology（浙江工业大学）；Northwestern Polytechnical University（西北工业大学）
Autonomous drone racing is becoming an excellent platform to challenge quadrotors' autonomy techniques including planning, navigation and control technologies. However, most research on this topic mainly focuses on single drone scenarios. In this paper, we describe a novel time-optimal trajectory generation method for generating time-optimal trajectories for a swarm of quadrotors to fly through pre-defined waypoints with their maximum maneuverability without collision. We verify the method in the Gazebo simulations where a swarm of 5 quadrotors can fly through a complex 6-waypoint racing track…
In this work we showcase the design and assessment of the performance of a multi-camera UAV, when coupled with state-of-the-art planning and mapping algorithms for autonomous navigation. The system leverages state-of-the-art receding horizon exploration techniques for Next-Best-View (NBV) planning with 3D and semantic information, provided by a reconfigurable multi stereo camera system. We employ our approaches in an autonomous drone-based inspection task and evaluate them in an autonomous exploration and mapping scenario. We discuss the advantages and limitations of using multi stereo camera …
This work investigates an efficient trajectory generation for chasing a dynamic target, which incorporates the detectability objective. The proposed method actively guides the motion of a cinematographer drone so that the color of a target is well-distinguished against the colors of the background in the view of the drone. For the objective, we define a measure of color detectability given a chasing path. After computing a discrete path optimized for the metric, we generate a dynamically feasible trajectory. The whole pipeline can be updated on-the-fly to respond to the motion of the target. F…
University of Zurich（苏黎世大学）；ETH Zurich（苏黎世联邦理工学院）；Intel（英特尔）
Performing acrobatic maneuvers with quadrotors is extremely challenging. Acrobatic flight requires high thrust and extreme angular accelerations that push the platform to its physical limits. Professional drone pilots often measure their level of mastery by flying such maneuvers in competitions. In this paper, we propose to learn a sensorimotor policy that enables an autonomous quadrotor to fly extreme acrobatic maneuvers with only onboard sensing and computation. We train the policy entirely in simulation by leveraging demonstrations from an optimal controller that has access to privileged in…
Volkswagen Group, Machine Learning Research Lab（大众集团机器学习研究实验室）；Technical University of Darmstadt（达姆施塔特工业大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Learning to control robots without requiring engineered models has been a long-term goal, promising diverse and novel applications. Yet, reinforcement learning has only achieved limited impact on real-time robot control due to its high demand of real-world interactions. In this work, by leveraging a learnt probabilistic model of drone dynamics, we learn a thrust-attitude controller for a quadrotor through model-based reinforcement learning. No prior knowledge of the flight dynamics is assumed; instead, a sequential latent variable model, used generatively and as an online filter, is learnt fro…
A concept of drone launched short range rockets (DLSRR) is presented. A drone or an aircraft rises DLSRR to a release altitude of up to 20 $km$. At the release altitude, the drone or an aircraft is moving at a velocity of up to 700 $m/s$ and a steep angle of up to 68$^o$ to the horizontal. After DLSRRs are released, their motors start firing. DLSRRs use slow burning motors to gain altitude and velocity. At the apogee of their flight DLSRRs release projectiles which fly to the target and strike it at high impact velocity. The projectiles reach a target at ranges of up to 442 $km$ and impact vel…
How do you learn to navigate an Unmanned Aerial Vehicle (UAV) and avoid obstacles? One approach is to use a small dataset collected by human experts: however, high capacity learning algorithms tend to overfit when trained with little data. An alternative is to use simulation. But the gap between simulation and real world remains large especially for perception problems. The reason most research avoids using large-scale real data is the fear of crashes! In this paper, we propose to bite the bullet and collect a dataset of crashes itself! We build a drone whose sole purpose is to crash into obje…
Drone swarms are required for the simultaneous delivery of multiple packages. We demonstrate a multi-stop drone swarm-based delivery in a smart city. We leverage formation flying to conserve energy and increase the flight range of a drone swarm. An adaptive formation is presented in which a swarm adjusts to extrinsic constraints and changes the formation pattern in-flight. We utilize the existing building rooftops in a city and build a line-of-sight skyway network to safely operate the swarms. We use a heuristic-based A* algorithm to route a drone swarm in a skyway network.
An autonomous drone flying near obstacles needs to be able to detect and avoid the obstacles or it will collide with them. In prior work, drones can detect and avoid walls using data from camera, ultrasonic or laser sensors mounted either on the drone or in the environment. It is not always possible to instrument the environment, and sensors added to the drone consume payload and power - both of which are constrained for drones. This paper studies how data mining classification techniques can be used to predict where an obstacle is in relation to the drone based only on monitoring air-disturba…
Drone racing is a recreational sport in which the goal is to pass through a sequence of gates in a minimum amount of time while avoiding collisions. In autonomous drone racing, one must accomplish this task by flying fully autonomously in an unknown environment by relying only on computer vision methods for detecting the target gates. Due to the challenges such as background objects and varying lighting conditions, traditional object detection algorithms based on colour or geometry tend to fail. Convolutional neural networks offer impressive advances in computer vision but require an immense a…
Indian Institute of Science（印度科学学院）；National Institute of Technology, Srinagar（斯利那加国立理工学院）
In the modern era of automation and robotics, autonomous vehicles are currently the focus of academic and industrial research. With the ever increasing number of unmanned aerial vehicles getting involved in activities in the civilian and commercial domain, there is an increased need for autonomy in these systems too. Due to guidelines set by the governments regarding the operation ceiling of civil drones, road-tracking based navigation is garnering interest . In an attempt to achieve the above mentioned tasks, we propose an imitation learning based, data-driven solution to UAV autonomy for nav…
In this paper, we present a control system that allows a drone to fly autonomously through a series of gates marked with ArUco tags. A simple and low-cost DJI Tello EDU quad-rotor platform was used. Based on the API provided by the manufacturer, we have created a Python application that enables the communication with the drone over WiFi, realises drone positioning based on visual feedback, and generates control. Two control strategies were proposed, compared, and critically analysed. In addition, the accuracy of the positioning method used was measured. The application was evaluated on a lapto…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
We propose a concept of a novel interaction strategy for providing rich haptic feedback in Virtual Reality (VR), when each user's finger is connected to micro-quadrotor with a wire. Described technology represents the first flying wearable haptic interface. The solution potentially is able to deliver high resolution force feedback to each finger during fine motor interaction in VR. The tips of tethers are connected to the centers of quadcopters under their bottom. Therefore, flight stability is increasing and the interaction forces are becoming stronger which allows to use smaller drones.
University of Zurich（苏黎世大学）；ETH Zurich（苏黎世联邦理工学院）；Intel（英特尔）
Autonomous micro aerial vehicles still struggle with fast and agile maneuvers, dynamic environments, imperfect sensing, and state estimation drift. Autonomous drone racing brings these challenges to the fore. Human pilots can fly a previously unseen track after a handful of practice runs. In contrast, state-of-the-art autonomous navigation algorithms require either a precise metric map of the environment or a large amount of training data collected in the track of interest. To bridge this gap, we propose an approach that can fly a new track in a previously unseen environment without a precise …
California Institute of Technology（加州理工学院）；Northeastern University（东北大学）
This work presents an actuation framework for a bioinspired flapping drone called Aerobat. This drone, capable of producing dynamically versatile wing conformations, possesses 14 body joints and is tail-less. Therefore, in our robot, unlike mainstream flapping wing designs that are open-loop stable and have no pronounced morphing characteristics, the actuation, and closed-loop feedback design can pose significant challenges. We propose a framework based on integrating mechanical intelligence and control. In this design framework, small adjustments led by several tiny low-power actuators called…
University of Perugia（佩鲁贾大学）；Jadavpur University（贾达普大学）；University of Florence（佛罗伦萨大学）
Drones have become increasingly popular in a variety of fields, including agriculture, emergency response, and package delivery. However, most drone operations are currently limited to within Visual Line of Sight (vlos) due to safety concerns. Flying drones Beyond Visual Line of Sight (bvlos) presents new challenges and opportunities, but also requires new technologies and regulatory frameworks, not yet implemented, to ensure that the drone is constantly under the control of a remote operator. In this preliminary study, we assume to remotely control the drone using the available ground cellula…
In literature, several approaches are trying to make the UAVs fly autonomously i.e., by extracting perspective cues such as straight lines. However, it is only available in well-defined human made environments, in addition to many other cues which require enough texture information. Our main target is to detect and avoid frontal obstacles from a monocular camera using a quad rotor Ar.Drone 2 by exploiting optical flow as a motion parallax, the drone is permitted to fly at a speed of 1 m/s and an altitude ranging from 1 to 4 meters above the ground level. In general, detecting and avoiding fron…
Drones are enabling new forms of cinematography. However, quadrotor cinematography requires accurate comprehension of the scene, technical skill of flying, artistic skill of composition and simultaneous realization of all the requirements in real time. These requirements could pose real challenge to drone amateurs because unsuitable camera viewpoint and motion could result in unpleasing visual composition and affect the target's visibility. In this paper, we propose a novel autonomous drone camera system which captures action scenes using proper camera viewpoint and motion. The key novelty is …
Universidad Politécnica de Madrid（马德里理工大学）；Texas A&M University（德州农工大学）；Technische Universiteit Delft（代尔夫特理工大学）
Obstacle avoidance is a key feature for safe Unmanned Aerial Vehicle (UAV) navigation. While solutions have been proposed for static obstacle avoidance, systems enabling avoidance of dynamic objects, such as drones, are hard to implement due to the detection range and field-of-view (FOV) requirements, as well as the constraints for integrating such systems on-board small UAVs. In this work, a dataset of 6k synthetic depth maps of drones has been generated and used to train a state-of-the-art deep learning-based drone detection model. While many sensing technologies can only provide relative al…
Fully Onboard AI-powered Human-Drone Pose Estimation on Ultra-low Power Autonomous Flying Nano-UAVs
Dalle Molle Institute for Artificial Intelligence (IDSIA), University of Lugano & SUPSI（达勒·莫勒人工智能研究所（IDSIA），卢加诺大学与瑞士南部应用科学大学）；University of Bologna（博洛尼亚大学）；ETH Zürich（苏黎世联邦理工学院）
Delft University of Technology（代尔夫特理工大学）；University of Liverpool（利物浦大学）；Radboud University of Nijmegen（奈梅亨拉德堡德大学）
Miniature Micro Aerial Vehicles (MAV) are very suitable for flying in indoor environments, but autonomous navigation is challenging due to their strict hardware limitations. This paper presents a highly efficient computer vision algorithm called Edge-FS for the determination of velocity and depth. It runs at 20 Hz on a 4 g stereo camera with an embedded STM32F4 microprocessor (168 MHz, 192 kB) and uses feature histograms to calculate optical flow and stereo disparity. The stereo-based distance estimates are used to scale the optical flow in order to retrieve the drone's velocity. The velocity …
