Free LSD: Prior-Free Visual Landing Site Detection for Autonomous Planes
Autonomous Systems Lab, ETH Zurich（苏黎世联邦理工学院自主系统实验室）；CSAIL, MIT（麻省理工学院计算机科学与人工智能实验室）
Timo Hinzmann, Thomas Stastny, Cesar Cadena, Roland Siegwart, Igor Gilitschenski
备注：
Accepted for publication in IEEE International Conference on Robotics and Automation (ICRA), 2018, Brisbane and IEEE Robotics and Automation Letters (RA-L), 2018
Full autonomy for fixed-wing unmanned aerial vehicles (UAVs) requires the capability to autonomously detect potential landing sites in unknown and unstructured terrain, allowing for self-governed mission completion or handling of emergency situations. In this work, we propose a perception system addressing this challenge by detecting landing sites based on their texture and geometric shape without using any prior knowledge about the environment. The proposed method considers hazards within the landing region such as terrain roughness and slope, surrounding obstacles that obscure the landing approach path, and the local wind field that is estimated by the on-board EKF. The latter enables applicability of the proposed method on small-scale autonomous planes without landing gear. A safe approach path is computed based on the UAV dynamics, expected state estimation and actuator uncertainty, and the on-board computed elevation map. The proposed framework has been successfully tested on photo-realistic synthetic datasets and in challenging real-world environments.
Landing is the most challenging and risky aspect of multirotor drone flight, and only simple landing methods exist for autonomous drones. We explore methods for autonomous drone landing in two scenarios. In the first scenario, we examine methods for landing on known landing pads using fiducial markers and a gimbal-mounted monocular camera. This method has potential in drone applications where a drone must land more accurately than GPS can provide (e.g.~package delivery in an urban canyon). We expand on previous methods by actuating the drone's camera to track the marker over time, and we addre…
The paper discusses an intelligent vision-based control solution for autonomous tracking and landing of Vertical Take-Off and Landing (VTOL) capable Unmanned Aerial Vehicles (UAVs) on ships without utilizing GPS signal. The central idea involves automating the Navy helicopter ship landing procedure where the pilot utilizes the ship as the visual reference for long-range tracking; however, refers to a standardized visual cue installed on most Navy ships called the "horizon bar" for the final approach and landing phases. This idea is implemented using a uniquely designed nonlinear controller int…
Unmanned Aerial Vehicles (UAVs) equipped with bioradars are a life-saving technology that can enable identification of survivors under collapsed buildings in the aftermath of natural disasters such as earthquakes or gas explosions. However, these UAVs have to be able to autonomously land on debris piles in order to accurately locate the survivors. This problem is extremely challenging as the structure of these debris piles is often unknown and no prior knowledge can be leveraged. In this work, we propose a computationally efficient system that is able to reliably identify safe landing sites an…
Southern University of Science and Technology（南方科技大学）；Shandong University（山东大学）；The Chinese University of Hong Kong（香港中文大学）；Shenzhen Research Institute of the Chinese University of Hong Kong（香港中文大学深圳研究院）
This paper introduces a quadrotor's autonomous take-off and landing system on a moving platform. The designed system addresses three challenging problems: fast pose estimation, restricted external localization, and effective obstacle avoidance. Specifically, first, we design a landing recognition and positioning system based on the AruCo marker to help the quadrotor quickly calculate the relative pose; second, we leverage a gradient-based local motion planner to generate collision-free reference trajectories rapidly for the quadrotor; third, we build an autonomous state machine that enables th…
Landing is a challenging part of autonomous drone flight and a great research opportunity. This PhD proposes to improve on fiducial autonomous landing algorithms by making them more flexible. Further, it leverages its location, Iceland, to develop a method for landing on lava flows in cooperation with analog Mars exploration missions taking place in Iceland now - and potentially for future Mars landings.
Soloviev Rybinsk State Aviation Technical University（索洛维约夫雷宾斯克国立航空技术大学）；NPP SATEK plus（NPP SATEK plus 公司）；Program System Institute of Russian Academy of Sciences（俄罗斯科学院程序系统研究所）；Institute For Systems Analysis of Russian Academy of Sciences（俄罗斯科学院系统分析研究所）
Autonomous fixed-wing UAV landing based on differential GPS is now a mainstream providing reliable and precise landing. But the task still remains challenging when GPS availability is limited like for military UAVs. We discuss a solution of this problem based on computer vision and dot markings along stationary or makeshift runway. We focus our attempts on using infrared beacons along with narrow-band filter as promising way to mark any makeshift runway and utilize particle filtering to fuse both IMU and visual data. We believe that unlike many other vision-based methods this solution is capab…
The paper focuses on a heterogeneous swarm of drones to achieve a dynamic landing of formation on a moving robot. This challenging task was not yet achieved by scientists. The key technology is that instead of facilitating each agent of the swarm of drones with computer vision that considerably increases the payload and shortens the flight time, we propose to install only one camera on the leader drone. The follower drones receive the commands from the leader UAV and maintain a collision-free trajectory with the artificial potential field. The experimental results revealed a high accuracy of t…
Precision landing is a remaining challenge in autonomous drone flight. Fiducial markers provide a computationally cheap way for a drone to locate a landing pad and autonomously execute precision landings. However, most work in this field depends on either rigidly-mounted or downward-facing cameras which restrict the drone's ability to detect the marker. We present a method of autonomous landing that uses a gimbal-mounted camera to quickly search for the landing pad by simply spinning in place while tilting the camera up and down, and to continually aim the camera at the landing pad during appr…
Harnessing LSTM for Nonlinear Ship Deck Motion Prediction in UAV Autonomous Landing amidst High Sea States
Zhejiang Provincial Engineering Research Centre for Special Aircrafts（浙江省特种飞行器工程研究中心）；Ningbo Institute of Materials Technology & Engineering, CAS（中国科学院宁波材料技术与工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Faculty of Electrical Engineering and Computer Science, Ningbo University（宁波大学电气工程与计算机科学学院）
Jet Propulsion Laboratory, California Institute of Technology（加州理工学院喷气推进实验室）
The next generation of Mars rotorcrafts requires on-board autonomous hazard avoidance landing. To this end, this work proposes a system that performs continuous multi-resolution height map reconstruction and safe landing spot detection. Structure-from-Motion measurements are aggregated in a pyramid structure using a novel Optimal Mixture of Gaussians formulation that provides a comprehensive uncertainty model. Our multiresolution pyramid is built more efficiently and accurately than past work by decoupling pyramid filling from the measurement updates of different resolutions. To detect the saf…
Mila - Quebec Institute of Artificial Intelligence, Université de Montréal（米拉-魁北克人工智能研究所，蒙特利尔大学）；Alternova Tech SAS（Alternova科技公司）；Universidad Autónoma de Occidente（西部自治大学）
Autonomous landing is a capability that is essential to achieve the full potential of multi-rotor drones in many social and industrial applications. The implementation and testing of this capability on physical platforms is risky and resource-intensive; hence, in order to ensure both a sound design process and a safe deployment, simulations are required before implementing a physical prototype. This paper presents the development of a monocular visual system, using a software-in-the-loop methodology, that autonomously and efficiently lands a quadcopter drone on a predefined landing pad, thus r…
Visual-based Safe Landing for UAVs in Populated Areas: Real-time Validation in Virtual Environments
Universidad Autónoma de Ciudad Juárez UACJ（华雷斯自治大学）；Center for Research in Mathematics CIMAT AC（数学研究中心CIMAT AC）；University of Texas at El Paso（德克萨斯大学埃尔帕索分校）
Evidence-Based Landing Site Selection and Vison-Based Landing for UAVs in Unstructured Environments
Aerospace Research Centre, National Research Council Canada（加拿大国家研究委员会航空航天研究中心）；Department of Mechanical and Industrial Engineering, Toronto Metropolitan University（多伦多 Metropolitan 大学机械与工业工程系）
Risk Assessment for Autonomous Landing in Urban Environments using Semantic Segmentation
Center for Research in Mathematics, CIMAT AC（墨西哥数学研究中心（CIMAT AC））；Centro de Investigación en Matemáticas, A.C.（墨西哥数学研究中心）；Centro de Investigaciones en Óptica, A.C.（墨西哥光学研究中心）；CONAHCYT（墨西哥国家科学技术委员会）
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
Selecting safe landing sites in non-cooperative environments is a key step towards the full autonomy of UAVs. However, the existing methods have the common problems of poor generalization ability and robustness. Their performance in unknown environments is significantly degraded and the error cannot be self-detected and corrected. In this paper, we construct a UAV system equipped with low-cost LiDAR and binocular cameras to realize autonomous landing in non-cooperative environments by detecting the flat and safe ground area. Taking advantage of the non-repetitive scanning and high FOV coverage…
Autonomous and safe landing is important for unmanned aerial vehicles. We present a monocular and stereo image based method for fast and accurate landing zone evaluation for UAVs in various scenarios. Many existing methods rely on Lidar or depth sensor to provide accurate and dense surface reconstruction. We utilize stereo images to evaluate the slope and monocular images to compute homography error. By combining them together, our approach works for both rigid and non-rigid dynamic surfaces. Experiments on many outdoor scenes such as water, grass and roofs, demonstrate the robustness and effe…
KIOS Research and Innovation Center of Excellence（KIOS卓越研究与创新中心）；University of Cyprus（塞浦路斯大学）
While Unmanned Aerial Vehicles (UAVs) are increasingly deployed in several missions, their inability of reliable and consistent autonomous landing poses a major setback for deploying such systems truly autonomously. In this paper we present an autonomous UAV landing system for landing on a moving platform. In contrast to existing attempts, the proposed system relies only on the camera sensor, and has been designed as lightweight as possible. The proposed system can be deployed on a low power platform as part of the drone payload, whilst being indifferent to any external communication or any ot…
With the development of industry, drones are appearing in various field. In recent years, deep reinforcement learning has made impressive gains in games, and we are committed to applying deep reinforcement learning algorithms to the field of robotics, moving reinforcement learning algorithms from game scenarios to real-world application scenarios. We are inspired by the LunarLander of OpenAI Gym, we decided to make a bold attempt in the field of reinforcement learning to control drones. At present, there is still a lack of work applying reinforcement learning algorithms to robot control, the p…
The paper discusses a novel vision-based estimation and control approach to enable fully autonomous tracking and landing of vertical take-off and landing (VTOL) capable unmanned aerial vehicles (UAVs) on moving platforms without relying on a GPS signal. A unique feature of the present method is that it accomplishes this task without tracking the landing pad itself; however, by utilizing a standardized visual cue installed normal to the landing pad and parallel to the pilot's/vehicle's line of sight. A computer vision system using a single monocular camera is developed to detect the visual cue …
This paper presents a system for hardware-in-the-loop (HiL) simulation of unmanned aerial vehicle (UAV) control algorithms implemented on a heterogeneous SoC FPGA computing platforms. The AirSim simulator running on a PC and an Arty Z7 development board with a Zynq SoC chip from AMD Xilinx were used. Communication was carried out via a serial USB link. An application for autonomous landing on a specially marked landing strip was selected as a case study. A landing site detection algorithm was implemented on the Zynq SoC platform. This allowed processing a 1280 x 720 @ 60 fps video stream in re…
Safe autonomous landing in urban cities is a necessity for the growing Unmanned Aircraft Systems (UAS) industry. In urgent situations, building rooftops, particularly flat rooftops, can provide local safe landing zones for small UAS. This paper investigates the real-time identification and selection of safe landing zones on rooftops based on LiDAR and camera sensor feedback. A visual high fidelity simulated city is constructed in the Unreal game engine, with particular attention paid to accurately generating rooftops and the common obstructions found thereon, e.g., ac units, water towers, air …
Amazon（亚马逊公司）；West Virginia University（西弗吉尼亚大学）；Microsoft（微软公司）；Gather AI（Gather AI公司）；Mowito（Mowito公司）；Carnegie Mellon University（卡内基梅隆大学）
Many aerial robotic applications require the ability to land on moving platforms, such as delivery trucks and marine research boats. We present a method to autonomously land an Unmanned Aerial Vehicle on a moving vehicle. A visual servoing controller approaches the ground vehicle using velocity commands calculated directly in image space. The control laws generate velocity commands in all three dimensions, eliminating the need for a separate height controller. The method has shown the ability to approach and land on the moving deck in simulation, indoor and outdoor environments, and compared t…
This paper addresses the problem of developing an algorithm for autonomous ship landing of vertical take-off and landing (VTOL) capable unmanned aerial vehicles (UAVs), using only a monocular camera in the UAV for tracking and localization. Ship landing is a challenging task due to the small landing space, six degrees of freedom ship deck motion, limited visual references for localization, and adversarial environmental conditions such as wind gusts. We first develop a computer vision algorithm which estimates the relative position of the UAV with respect to a horizon reference bar on the landi…
Autonomous Marine System Research Group, School of Engineering, Plymouth University, UK（普利茅斯大学工程学院自主海洋系统研究小组）；Centre for Robotics and Neural Systems, School of Computing, Electronics and Mathematics, Plymouth University, UK（普利茅斯大学计算、电子与数学学院机器人与神经系统中心）
Landing an unmanned aerial vehicle (UAV) on a ground marker is an open problem despite the effort of the research community. Previous attempts mostly focused on the analysis of hand-crafted geometric features and the use of external sensors in order to allow the vehicle to approach the land-pad. In this article, we propose a method based on deep reinforcement learning that only requires low-resolution images taken from a down-looking camera in order to identify the position of the marker and land the UAV on it. The proposed approach is based on a hierarchy of Deep Q-Networks (DQNs) used as hig…
Delft University of Technology（代尔夫特理工大学）；Czech Technical University in Prague（布拉格捷克理工大学）
Landing a quadrotor on an inclined surface is a challenging maneuver. The final state of any inclined landing trajectory is not an equilibrium, which precludes the use of most conventional control methods. We propose a deep reinforcement learning approach to design an autonomous landing controller for inclined surfaces. Using the proximal policy optimization (PPO) algorithm with sparse rewards and a tailored curriculum learning approach, an inclined landing policy can be trained in simulation in less than 90 minutes on a standard laptop. The policy then directly runs on a real Crazyflie 2.1 qu…
An Experimental Study of Trojan Vulnerabilities in UAV Autonomous Landing
Department of Electrical and Computer Engineering at North Carolina A&T State University（北卡罗来纳A&T州立大学电气与计算机工程系）；University of Alabama（阿拉巴马大学）；Department of Electrical and Computer Engineering, Windsor University（温莎大学电气与计算机工程系）
Center for Research in Mathematics CIMAT AC（数学研究中心（CIMAT AC））
Autonomous landing of Unmanned Aerial Vehicles (UAVs) in crowded scenarios is crucial for successful deployment of UAVs in populated areas, particularly in emergency landing situations where the highest priority is to avoid hurting people. In this work, a new visual-based algorithm for identifying Safe Landing Zones (SLZ) in crowded scenarios is proposed, considering a camera mounted on an UAV, where the people in the scene move with unknown dynamics. To do so, a density map is generated for each image frame using a Deep Neural Network, from where a binary occupancy map is obtained aiming to o…
Automatic optimization of robotic behavior has been the long-standing goal of Evolutionary Robotics. Allowing the problem at hand to be solved by automation often leads to novel approaches and new insights. A common problem encountered with this approach is that when this optimization occurs in a simulated environment, the optimized policies are subject to the reality gap when implemented in the real world. This often results in sub-optimal behavior, if it works at all. This paper investigates the automatic optimization of neurocontrollers to perform quick but safe landing maneuvers for a quad…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
Heterogeneous teams of mobile robots and UAVs are offering a substantial benefit in an autonomous exploration of the environment. Nevertheless, although joint exploration scenarios for such systems are widely discussed, they are still suffering from low adaptability to changes in external conditions and faults of swarm agents during the UAV docking. We propose a novel vision-based drone swarm docking system for robust landing on a moving platform when one of the agents lost its position signal. The proposed SwarmHawk system relies on vision-based detection for the mobile platform tracking and …
LLM-Land: Large Language Models for Context-Aware Drone Landing
Department of Electrical and Computer Engineering（电气与计算机工程系）；Drexel University（德雷塞尔大学）；Department of Electrical and Systems Engineering（电气与系统工程系）；University of Pennsylvania（宾夕法尼亚大学）
As UAV technology improves, more uses have been found for these versatile autonomous vehicles, from surveillance to aerial photography, to package delivery, and each of these applications poses unique challenges. This paper implements a solution for one such challenge: To land on a moving target. This problem has been addressed before with varying degrees of success, however, most implementations focus on indoor applications. Outdoor poses greater challenges in the form of variables such as wind and lighting, and outdoor drones are heavier and more susceptible to inertial effects. Our approach…
