We focus on the problem of reliably training Reinforcement Learning (RL) models (agents) for stable low-level control in embedded systems and test our methods on a high-performance, custom-built quadrotor platform. A common but often under-studied problem in developing RL agents for continuous control is that the control policies developed are not always smooth. This lack of smoothness can be a major problem when learning controllers %intended for deployment on real hardware as it can result in control instability and hardware failure. Issues of noisy control are further accentuated when training RL agents in simulation due to simulators ultimately being imperfect representations of reality - what is known as the reality gap. To combat issues of instability in RL agents, we propose a systematic framework, `REinforcement-based transferable Agents through Learning' (RE+AL), for designing simulated training environments which preserve the quality of trained agents when transferred to real platforms. RE+AL is an evolution of the Neuroflight infrastructure detailed in technical reports prepared by members of our research group. Neuroflight is a state-of-the-art framework for training RL agents for low-level attitude control. RE+AL improves and completes Neuroflight by solving a number of important limitations that hindered the deployment of Neuroflight to real hardware. We benchmark RE+AL on the NF1 racing quadrotor developed as part of Neuroflight. We demonstrate that RE+AL significantly mitigates the previously observed issues of smoothness in RL agents. Additionally, RE+AL is shown to consistently train agents that are flight-capable and with minimal degradation in controller quality upon transfer. RE+AL agents also learn to perform better than a tuned PID controller, with better tracking errors, smoother control and reduced power consumption.
East China University of Science and Technology（华东理工大学）；Western Sydney University（西悉尼大学）
The ability to perform aggressive movements, which are called aggressive flights, is important for quadrotors during navigation. However, aggressive quadrotor flights are still a great challenge to practical applications. The existing solutions to aggressive flights heavily rely on a predefined trajectory, which is a time-consuming preprocessing step. To avoid such path planning, we propose a curiosity-driven reinforcement learning method for aggressive flight missions and a similarity-based curiosity module is introduced to speed up the training procedure. A branch structure exploration (BSE)…
cc-DRL: a Convex Combined Deep Reinforcement Learning Flight Control Design for a Morphing Quadrotor
Beihang University（北京航空航天大学）；University of Science and Technology Beijing（北京科技大学）；Hangzhou International Innovation Institute of Beihang University（北京航空航天大学杭州国际创新研究院）
This technical report provides detailed theoretical analysis of the algorithm used in \textit{Alternating Minimization Based Trajectory Generation for Quadrotor Aggressive Flight}. An assumption is provided to ensure that settings for the objective function are meaningful. What's more, we explore the structure of the optimization problem and analyze the global/local convergence rate of the employed algorithm.
Temasek Laboratories, NUS（新加坡国立大学淡马锡实验室）；Department of Electrical & Computer Engineering, NUS（新加坡国立大学电气与计算机工程系）
Majority of today's fixed-pitch, electric-power quadrotors have short flight endurance ($<$ 1 hour) which greatly limits their applications. This paper presents a design methodology for the construction of a long-endurance quadrotor using variable-pitch rotors and a gasoline-engine. The methodology consists of three aspects. Firstly, the rotor blades and gasoline engine are selected as a pair, so that sufficient lift can be comfortably provided by the engine. Secondly, drivetrain and airframe are designed. Major challenges include airframe vibration minimization and power transmission from one…
Time-Optimal Planning for Long-Range Quadrotor Flights: An Automatic Optimal Synthesis Approach
University of Toronto Institute for Aerospace Studies（多伦多大学航空航天研究所）；University of Toronto Robotics Institute（多伦多大学机器人研究所）；University of Toronto Division of Engineering Science（多伦多大学工程科学系）；Technical University of Munich（慕尼黑工业大学）；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与机器智能研究所）
Quadrotors are among the most agile flying robots. However, planning time-optimal trajectories at the actuation limit through multiple waypoints remains an open problem. This is crucial for applications such as inspection, delivery, search and rescue, and drone racing. Early works used polynomial trajectory formulations, which do not exploit the full actuator potential because of their inherent smoothness. Recent works resorted to numerical optimization but require waypoints to be allocated as costs or constraints at specific discrete times. However, this time allocation is a priori unknown an…
Robotics and Perception Group（机器人与感知小组）；Depts. Informatics and Neuroinformatics（信息学与神经信息学系）；University of Zurich and ETH Zurich（苏黎世大学和苏黎世联邦理工学院）
State-of-the-art quadrotor simulators have a rigid and highly-specialized structure: either are they really fast, physically accurate, or photo-realistic. In this work, we propose a novel quadrotor simulator: Flightmare. Flightmare is composed of two main components: a configurable rendering engine built on Unity and a flexible physics engine for dynamics simulation. Those two components are totally decoupled and can run independently of each other. This makes our simulator extremely fast: rendering achieves speeds of up to 230 Hz, while physics simulation of up to 200,000 Hz on a laptop. In a…
This paper considers onboard control of a small-sized quadrotor using a strapdown embedded optical flow sensor which is conventionally used for desktop mice. The vehicle considered in this paper can carry only few dozen grams of payload, therefore conventional camera-based optical flow methods are not applicable. We present hovering control of the small-sized quadrotor using a single-chip optical flow sensor, implemented on an 8-bit microprocessor without external sensors or communication with a ground control station. Detailed description of all the system components is provided along with ev…
We address one of the main challenges towards autonomous quadrotor flight in complex environments, which is flight through narrow gaps. While previous works relied on off-board localization systems or on accurate prior knowledge of the gap position and orientation, we rely solely on onboard sensing and computing and estimate the full state by fusing gap detection from a single onboard camera with an IMU. This problem is challenging for two reasons: (i) the quadrotor pose uncertainty with respect to the gap increases quadratically with the distance from the gap; (ii) the quadrotor has to active…
We tackle the problem of planning a minimum-time trajectory for a quadrotor over a sequence of specified waypoints in the presence of obstacles while exploiting the full quadrotor dynamics. This problem is crucial for autonomous search and rescue and drone racing scenarios but was, so far, unaddressed by the robotics community \emph{in its entirety} due to the challenges of minimizing time in the presence of the non-convex constraints posed by collision avoidance. Early works relied on simplified dynamics or polynomial trajectory representations that did not exploit the full actuator potential…
Collision detection and recovery for aerial robots remain a challenge because of the limited space for sensors and local stability of the flight controller. We introduce a novel collision-resilient quadrotor that features a compliant arm design to enable free flight while allowing for one passive degree of freedom to absorb shocks. We further propose a novel collision detection and characterization method based on Hall sensors, as well as a new recovery control method to generate and track a smooth trajectory after a collision occurs. Experimental results demonstrate that the robot can detect …
Rotor-Failure-Aware Quadrotors Flight in Unknown Environments
School of Robotics and Automation, Nanjing University（南京大学机器人与自动化学院）；Institute of Cyber-Systems and Control, College of Control Science and Engineering, Zhejiang University（浙江大学控制系统研究所）；Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University（香港理工大学航空工程系）；School of Aeronautic Science and Engineering, Beihang University（北航航空科学与工程学院）
University of Michigan（密歇根大学）；University of Arizona（亚利桑那大学）
This paper develops and experimentally evaluates a navigation function for quadrotor formation flight that is resilient to abrupt quadrotor failures and other obstacles. The navigation function is based on modeling healthy quadrotors as particles in an ideal fluid flow. We provide three key contributions: (i) A Containment Exclusion Mode (CEM) safety theorem and proof which guarantees safety and formally specifies a minimum safe distance between quadrotors in formation, (ii) A real-time, computationally efficient CEM navigation algorithm, (iii) Simulation and experimental algorithm validation.…
In this paper, we tackle the problem of flying a quadrotor using time-optimal control policies that can be replanned online when the environment changes or when encountering unknown disturbances. This problem is challenging as the time-optimal trajectories that consider the full quadrotor dynamics are computationally expensive to generate (order of minutes or even hours). We introduce a sampling-based method for efficient generation of time-optimal paths of a point-mass model. These paths are then tracked using a Model Predictive Contouring Control approach that considers the full quadrotor dy…
University of Zurich（苏黎世大学）；Delft University of Technology（代尔夫特理工大学）
Fault-tolerant control is crucial for safety-critical systems, such as quadrotors. State-of-art flight controllers can stabilize and control a quadrotor even when subjected to the complete loss of a rotor. However, these methods rely on external sensors, such as GPS or motion capture systems, for state estimation. To the best of our knowledge, this has not yet been achieved with only onboard sensors. In this paper, we propose the first algorithm that combines fault-tolerant control and onboard vision-based state estimation to achieve position control of a quadrotor subjected to complete failur…
Hong Kong University of Science and Technology（香港科技大学）；Zhejiang University（浙江大学）
Recent advances in trajectory replanning have enabled quadrotor to navigate autonomously in unknown environments. However, high-speed navigation still remains a significant challenge. Given very limited time, existing methods have no strong guarantee on the feasibility or quality of the solutions. Moreover, most methods do not consider environment perception, which is the key bottleneck to fast flight. In this paper, we present RAPTOR, a robust and perception-aware replanning framework to support fast and safe flight. A path-guided optimization (PGO) approach that incorporates multiple topolog…
High speed navigation through unknown environments is a challenging problem in robotics. It requires fast computation and tight integration of all the subsystems on the robot such that the latency in the perception-action loop is as small as possible. Aerial robots add a limitation of payload capacity, which restricts the amount of computation that can be carried onboard. This requires efficient algorithms for each component in the navigation system. In this paper, we describe our quadrotor system which is able to smoothly navigate through mixed indoor and outdoor environments and is able to f…
Hong Kong University of Science and Technology（香港科技大学）
In this paper, we propose a robust and efficient quadrotor motion planning system for fast flight in 3-D complex environments. We adopt a kinodynamic path searching method to find a safe, kinodynamic feasible and minimum-time initial trajectory in the discretized control space. We improve the smoothness and clearance of the trajectory by a B-spline optimization, which incorporates gradient information from a Euclidean distance field (EDF) and dynamic constraints efficiently utilizing the convex hull property of B-spline. Finally, by representing the final trajectory as a non-uniform B-spline, …
We present a new quadrotor geometric control scheme that is capable of tracking highly aggressive trajectories. Our geometric controller uses the logarithmic map of SO(3) to express rotational error in the Lie algebra, and we show that it is globally attractive without requiring a complicated hybrid switching scheme. We show the performance of our controller against highly aggressive trajectories in simulation experiments. Additionally, we present an adaptation of this controller that allows us to interface effectively with the angular rate controllers on an onboard flight control unit and sho…
We consider the problem of generating a time-optimal quadrotor trajectory that attains a set of prescribed waypoints. This problem is challenging since the optimal trajectory is located on the boundary of the set of dynamically feasible trajectories. This boundary is hard to model as it involves limitations of the entire system, including hardware and software, in agile high-speed flight. In this work, we propose a multi-fidelity Bayesian optimization framework that models the feasibility constraints based on analytical approximation, numerical simulation, and real-world flight experiments. By…
Perception-aware Planning for Quadrotor Flight in Unknown and Feature-limited Environments
School of Intelligence Science and Engineering, Harbin Institute of Technology（哈尔滨工业大学智能科学与工程学院）；Department of Mechanical and Energy Engineering, Southern University of Science and Technology（南方科技大学机械与能源工程系）
APACE: Agile and Perception-Aware Trajectory Generation for Quadrotor Flights
School of Artificial Intelligence, Sun Yat-Sen University（中山大学人工智能学院）；Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology（香港科技大学电子与计算机工程学系）
Newton-Raphson Flow for Aggressive Quadrotor Tracking Control
Georgia Institute of Technology（佐治亚理工学院）；School of Electrical and Computer Engineering（电气与计算机工程学院）；School of Civil and Environmental Engineering（土木与环境工程学院）
Motion-Specific Battery Health Assessment for Quadrotors Using High-Fidelity Battery Models
Department of Convergence IT Engineering, Pohang University of Science and Technology（convergence IT工程系，釜山科学技术大学）；Department of Electrical Engineering, Pohang University of Science and Technology（电气工程系，釜山科学技术大学）
With much research has been conducted into trajectory planning for quadrotors, planning with spatial and temporal optimal trajectories in real-time is still challenging. In this paper, we propose a framework for generating large-scale piecewise polynomial trajectories for aggressive autonomous flights, with highlights on its superior computational efficiency and simultaneous spatial-temporal optimality. Exploiting the implicitly decoupled structure of the planning problem, we conduct alternating minimization between boundary conditions and time durations of trajectory pieces. In each minimizat…
University of Maryland Institute for Advanced Computer Studies（马里兰大学高级计算机研究所）
Although quadrotors, and aerial robots in general, are inherently active agents, their perceptual capabilities in literature so far have been mostly passive in nature. Researchers and practitioners today use traditional computer vision algorithms with the aim of building a representation of general applicability: a 3D reconstruction of the scene. Using this representation, planning tasks are constructed and accomplished to allow the quadrotor to demonstrate autonomous behavior. These methods are inefficient as they are not task driven and such methodologies are not utilized by flying insects a…
MAD: Mapping-Aware World Models for Agile Quadrotor Flight
State Key Lab of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology, Beijing 100081, China（自主智能无人系统国家重点实验室，北京理工大学，北京100081，中国）；Zhongguancun Academy, Beijing 100094, China（中关村学院，北京100094，中国）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；Department of Mechanical and Energy Engineering, Southern University of Science and Technology（南方科技大学机械与能源工程系）；Harbin Institute of Technology（哈尔滨工业大学）
This study has developed a fault-tolerant controller that is able to recover a quadrotor from arbitrary initial orientations and angular velocities, despite the complete failure of a rotor. This cascaded control method includes a position/altitude controller, an almost-global convergence attitude controller, and a control allocation method based on quadratic programming. As a major novelty, a constraint of undesirable angular velocity is derived and fused into the control allocator, which significantly improves the recovery performance. For validation, we have conducted a set of Monte-Carlo si…
We tackle the problem of flying time-optimal trajectories through multiple waypoints with quadrotors. State-of-the-art solutions split the problem into a planning task - where a global, time-optimal trajectory is generated - and a control task - where this trajectory is accurately tracked. However, at the current state, generating a time-optimal trajectory that considers the full quadrotor model requires solving a difficult time allocation problem via optimization, which is computationally demanding (in the order of minutes or even hours). This is detrimental for replanning in presence of dist…
