A Domain-Knowledge-Aided Deep Reinforcement Learning Approach for Flight Control Design
School of Aerospace, Transport and Manufacturing, Cranfield University（克兰菲尔德大学航空、运输与制造学院）；School of Aerospace Engineering, Beijing Institute of Technology（北京理工大学航空航天学院）
Hyo-Sang Shin, Shaoming He, Antonios Tsourdos
备注：
This work has been submitted to the IEEE for possible publication
This paper aims to examine the potential of using the emerging deep reinforcement learning techniques in flight control. Instead of learning from scratch, we suggest to leverage domain knowledge available in learning to improve learning efficiency and generalisability. More specifically, the proposed approach fixes the autopilot structure as typical three-loop autopilot and deep reinforcement learning is utilised to learn the autopilot gains. To solve the flight control problem, we then formulate a Markovian decision process with a proper reward function that enable the application of reinforcement learning theory. Another type of domain knowledge is exploited for defining the reward function, by shaping reference inputs in consideration of important control objectives and using the shaped reference inputs in the reward function. The state-of-the-art deep deterministic policy gradient algorithm is utilised to learn an action policy that maps the observed states to the autopilot gains. Extensive empirical numerical simulations are performed to validate the proposed computational control algorithm.
cc-DRL: a Convex Combined Deep Reinforcement Learning Flight Control Design for a Morphing Quadrotor
Beihang University（北京航空航天大学）；University of Science and Technology Beijing（北京科技大学）；Hangzhou International Innovation Institute of Beihang University（北京航空航天大学杭州国际创新研究院）
An Overview of Machine Learning-Enabled Optimization for Reconfigurable Intelligent Surfaces-Aided 6G Networks: From Reinforcement Learning to Large Language Models
School of Computer Science, McGill University（麦吉尔大学计算机科学学院）
Quadcopters can suffer from loss of propellers in mid-flight, thus requiring a need to have a system that detects single and multiple propeller failures and an adaptive controller that stabilizes the propeller-deficient quadcopter. This paper presents reinforcement learning based controllers for quadcopters with 4, 3, and 2 (opposing) functional propellers. The paper also proposes a neural network based propeller fault detection system to detect propeller loss and switch to the appropriate controller. The simulation results demonstrate a stable quadcopter with efficient waypoint tracking for a…
Boston University Graduate School of Arts and Sciences（波士顿大学文理研究生院）；University of Rhode Island（罗德岛大学）；Stevens Institute of Technology（史蒂文斯理工学院）
Traditional control methods are inadequate in many deployment settings involving control of Cyber-Physical Systems (CPS). In such settings, CPS controllers must operate and respond to unpredictable interactions, conditions, or failure modes. Dealing with such unpredictability requires the use of executive and cognitive control functions that allow for planning and reasoning. Motivated by the sport of drone racing, this dissertation addresses these concerns for state-of-the-art flight control by investigating the use of deep neural networks to bring essential elements of higher-level cognition …
Fault-tolerant flight control faces challenges, as developing a model-based controller for each unexpected failure is unrealistic, and online learning methods can handle limited system complexity due to their low sample efficiency. In this research, a model-free coupled-dynamics flight controller for a jet aircraft able to withstand multiple failure types is proposed. An offline trained cascaded Soft Actor-Critic Deep Reinforcement Learning controller is successful on highly coupled maneuvers, including a coordinated 40 degree bank climbing turn with a normalized Mean Absolute Error of 2.64%. …
We consider an Intelligent Reflecting Surface (IRS)-aided multiple-input single-output (MISO) system for downlink transmission. We compare the performance of Deep Reinforcement Learning (DRL) and conventional optimization methods in finding optimal phase shifts of the IRS elements to maximize the user signal-to-noise (SNR) ratio. Furthermore, we evaluate the robustness of these methods to channel impairments and changes in the system. We demonstrate numerically that DRL solutions show more robustness to noisy channels and user mobility.
University of California, Berkeley（加州大学伯克利分校）；Facebook AI Research（脸书人工智能研究院）
Transporting suspended payloads is challenging for autonomous aerial vehicles because the payload can cause significant and unpredictable changes to the robot's dynamics. These changes can lead to suboptimal flight performance or even catastrophic failure. Although adaptive control and learning-based methods can in principle adapt to changes in these hybrid robot-payload systems, rapid mid-flight adaptation to payloads that have a priori unknown physical properties remains an open problem. We propose a meta-learning approach that "learns how to learn" models of altered dynamics within seconds …
Nanyang Technological University（南洋理工大学）；University of North Texas（北德克萨斯大学）
Metaverse is expected to create a virtual world closely connected with reality to provide users with immersive experience with the support of 5G high data rate communication technique. A huge amount of data in physical world needs to be synchronized to the virtual world to provide immersive experience for users, and there will be higher requirements on coverage to include more users into Metaverse. However, 5G signal suffers severe attenuation, which makes it more expensive to maintain the same coverage. Unmanned aerial vehicle (UAV) is a promising candidate technique for future implementation…
University of Saskatchewan（萨斯喀彻温大学）；Cisco Systems Inc.（思科系统公司）
Employing unmanned aerial vehicles (UAVs) has attracted growing interests and emerged as the state-of-the-art technology for data collection in Internet-of-Things (IoT) networks. In this paper, with the objective of minimizing the total energy consumption of the UAV-IoT system, we formulate the problem of jointly designing the UAV's trajectory and selecting cluster heads in the IoT network as a constrained combinatorial optimization problem which is classified as NP-hard and challenging to solve. We propose a novel deep reinforcement learning (DRL) with a sequential model strategy that can eff…
The impact of Vehicle-to-Everything (V2X) communications on platoon control performance is investigated. Platoon control is essentially a sequential stochastic decision problem (SSDP), which can be solved by Deep Reinforcement Learning (DRL) to deal with both the control constraints and uncertainty in the platoon leading vehicle's behavior. In this context, the value of V2X communications for DRL-based platoon controllers is studied with an emphasis on the tradeoff between the gain of including exogenous information in the system state for reducing uncertainty and the performance erosion due t…
Flocking control is a challenging problem, where multiple agents, such as drones or vehicles, need to reach a target position while maintaining the flock and avoiding collisions with obstacles and collisions among agents in the environment. Multi-agent reinforcement learning has achieved promising performance in flocking control. However, methods based on traditional reinforcement learning require a considerable number of interactions between agents and the environment. This paper proposes a sub-optimal policy aided multi-agent reinforcement learning algorithm (SPA-MARL) to boost sample effici…
Técnico Lisboa（里斯本理工大学）；MBDA Deutschland GmbH（德国MBDA有限公司）；Technical University of Munich（慕尼黑工业大学）
Designing missiles' autopilot controllers has been a complex task, given the extensive flight envelope and the nonlinear flight dynamics. A solution that can excel both in nominal performance and in robustness to uncertainties is still to be found. While Control Theory often debouches into parameters' scheduling procedures, Reinforcement Learning has presented interesting results in ever more complex tasks, going from videogames to robotic tasks with continuous action domains. However, it still lacks clearer insights on how to find adequate reward functions and exploration strategies. To the b…
University of Southampton（南安普顿大学）；Bournemouth University（伯恩茅斯大学）；Beihang University（北京航空航天大学）
Data packet routing in aeronautical ad-hoc networks (AANETs) is challenging due to their high-dynamic topology. In this paper, we invoke deep reinforcement learning for routing in AANETs aiming at minimizing the end-to-end (E2E) delay. Specifically, a deep Q-network (DQN) is conceived for capturing the relationship between the optimal routing decision and the local geographic information observed by the forwarding node. The DQN is trained in an offline manner based on historical flight data and then stored by each airplane for assisting their routing decisions during flight. To boost the learn…
Baidu Inc.（百度公司）；City University of HongKong（香港城市大学）
Combining deep neural networks with reinforcement learning has shown great potential in the next-generation intelligent control. However, there are challenges in terms of safety and cost in practical applications. In this paper, we propose the Intervention Aided Reinforcement Learning (IARL) framework, which utilizes human intervened robot-environment interaction to improve the policy. We used the Unmanned Aerial Vehicle (UAV) as the test platform. We built neural networks as our policy to map sensor readings to control signals on the UAV. Our experiment scenarios cover both simulation and rea…
Contact-Aided Navigation of Flexible Robotic Endoscope Using Deep Reinforcement Learning in Dynamic Stomach
Department of Electronic Engineering, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong, China（电子工程系，香港中文大学（深圳）沙田分校，香港，中国）；Department of Mechanical Engineering, Stanford University（机械工程系，斯坦福大学）
Dense and complex air traffic scenarios require higher levels of automation than those exhibited by tactical conflict detection and resolution (CD\&R) tools that air traffic controllers (ATCO) use today. However, the air traffic control (ATC) domain, being safety critical, requires AI systems to which operators are comfortable to relinquishing control, guaranteeing operational integrity and automation adoption. Two major factors towards this goal are quality of solutions, and transparency in decision making. This paper proposes using a graph convolutional reinforcement learning method operatin…
Nanjing University of Science and Technology（南京理工大学）
Vehicle tracking has become one of the key applications of wireless sensor networks (WSNs) in the fields of rescue, surveillance, traffic monitoring, etc. However, the increased tracking accuracy requires more energy consumption. In this letter, a decentralized vehicle tracking strategy is conceived for improving both tracking accuracy and energy saving, which is based on adjusting the intersection area between the fixed sensing area and the dynamic activation area. Then, two deep reinforcement learning (DRL) aided solutions are proposed relying on the dynamic selection of the activation area …
University of Electronic Science and Technology of China（电子科技大学）；Yibin Institute of UESTC（电子科技大学宜宾研究院）；Zhejiang University（浙江大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Reconfigurable intelligent surface (RIS) has recently gained popularity as a promising solution for improving the signal transmission quality of wireless communications with less hardware cost and energy consumption. This letter offers a novel deep reinforcement learning (DRL) algorithm based on a location-aware imitation environment for the joint beamforming design in an RIS-aided mmWave multiple-input multiple-output system. Specifically, we design a neural network to imitate the transmission environment based on the geometric relationship between the user's location and the mmWave channel. …
Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）
This paper proposes a novel multiple-input multiple-output (MIMO) symbol detector that incorporates a deep reinforcement learning (DRL) agent into the Monte Carlo tree search (MCTS) detection algorithm. We first describe how the MCTS algorithm, used in many decision-making problems, is applied to the MIMO detection problem. Then, we introduce a self-designed deep reinforcement learning agent, consisting of a policy value network and a state value network, which is trained to detect MIMO symbols. The outputs of the trained networks are adopted into a modified MCTS detection algorithm to provide…
Politecnico di Bari（巴里理工大学）；Università degli Studi di Siena（锡耶纳大学）；Università degli Studi di Pisa（比萨大学）；CNIT, Consorzio Nazionale Interuniversitario per le Telecomunicazioni（意大利国家电信大学间联盟（CNIT））
The combination of cloud computing capabilities at the network edge and artificial intelligence promise to turn future mobile networks into service- and radio-aware entities, able to address the requirements of upcoming latency-sensitive applications. In this context, a challenging research goal is to exploit edge intelligence to dynamically and optimally manage the Radio Access Network Slicing (that is a less mature and more complex technology than fifth-generation Network Slicing) and Radio Resource Management, which is a very complex task due to the mostly unpredictably nature of the wirele…
University of Bologna（博洛尼亚大学）；King’s College London（伦敦国王学院）
Unmanned aerial base stations (UABSs) can be deployed in vehicular wireless networks to support applications such as extended sensing via vehicle-to-everything (V2X) services. A key problem in such systems is designing algorithms that can efficiently optimize the trajectory of the UABS in order to maximize coverage. In existing solutions, such optimization is carried out from scratch for any new traffic configuration, often by means of conventional reinforcement learning (RL). In this paper, we propose the use of continual meta-RL as a means to transfer information from previously experienced …
Guidance Design for Escape Flight Vehicle Using Evolution Strategy Enhanced Deep Reinforcement Learning
School of Aerospace Engineering, Tsinghua University（清华大学航天航空学院）；China Academy of Launch Vehicle Technology（中国运载火箭技术研究院）；School of Information and Communication Engineering, Beijing University of Posts and Telecommunications（北京邮电大学信息与通信工程学院）
IRIDA Research Centre for Communication Technologies（IRIDA通信技术研究中心）；University of Cyprus（塞浦路斯大学）；Ionian University（爱奥尼亚大学）；Aalto University（阿尔托大学）；Rutgers University（罗格斯大学）
Mobile networks are experiencing tremendous increase in data volume and user density. An efficient technique to alleviate this issue is to bring the data closer to the users by exploiting the caches of edge network nodes, such as fixed or mobile access points and even user devices. Meanwhile, the fusion of machine learning and wireless networks offers a viable way for network optimization as opposed to traditional optimization approaches which incur high complexity, or fail to provide optimal solutions. Among the various machine learning categories, reinforcement learning operates in an online…
Nanyang Technological University（南洋理工大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Harvard University（哈佛大学）
Incentive mechanisms for crowdsourcing are designed to incentivize financially self-interested workers to generate and report high-quality labels. Existing mechanisms are often developed as one-shot static solutions, assuming a certain level of knowledge about worker models (expertise levels, costs of exerting efforts, etc.). In this paper, we propose a novel inference aided reinforcement mechanism that learns to incentivize high-quality data sequentially and requires no such prior assumptions. Specifically, we first design a Gibbs sampling augmented Bayesian inference algorithm to estimate wo…
Bilkent University（比尔肯特大学）；KTH Royal Institute of Technology（KTH皇家理工学院）
We introduce a novel deep reinforcement learning (DRL) approach to jointly optimize transmit beamforming and reconfigurable intelligent surface (RIS) phase shifts in a multiuser multiple input single output (MU-MISO) system to maximize the sum downlink rate under the phase-dependent reflection amplitude model. Our approach addresses the challenge of imperfect channel state information (CSI) and hardware impairments by considering a practical RIS amplitude model. We compare the performance of our approach against a vanilla DRL agent in two scenarios: perfect CSI and phase-dependent RIS amplitud…
The ubiquitous growth of mobility-on-demand services for passenger and goods delivery has brought various challenges and opportunities within the realm of transportation systems. As a result, intelligent transportation systems are being developed to maximize operational profitability, user convenience, and environmental sustainability. The growth of last mile deliveries alongside ridesharing calls for an efficient and cohesive system that transports both passengers and goods. Existing methods address this using static routing methods considering neither the demands of requests nor the transfer…
Deep Reinforcement Learning-Based Precoding for Multi-RIS-Aided Multiuser Downlink Systems with Practical Phase Shift
Research Center for Information Technology Innovation (CITI), Academia Sinica（信息科技创新研究中心（CITI），学术院）；Institute of Communications Engineering, National Sun Yat-sen University（通讯工程研究所，国立中山大学）；Bradley Department of Electrical and Computer Engineering, Virginia Tech（电气与计算机工程系，弗吉尼亚理工大学）
Nanyang Technological University（南洋理工大学）；Energy Research Institute, Nanyang Technological University（南洋理工大学能源研究院）
Decision-making is critical for lane change in autonomous driving. Reinforcement learning (RL) algorithms aim to identify the values of behaviors in various situations and thus they become a promising pathway to address the decision-making problem. However, poor runtime safety hinders RL-based decision-making strategies from complex driving tasks in practice. To address this problem, human demonstrations are incorporated into the RL-based decision-making strategy in this paper. Decisions made by human subjects in a driving simulator are treated as safe demonstrations, which are stored into the…
Deep Reinforcement Learning (DRL) is gaining attention as a potential approach to design trajectories for autonomous unmanned aerial vehicles (UAV) used as flying access points in the context of cellular or Internet of Things (IoT) connectivity. DRL solutions offer the advantage of on-the-go learning hence relying on very little prior contextual information. A corresponding drawback however lies in the need for many learning episodes which severely restricts the applicability of such approach in real-world time- and energy-constrained missions. Here, we propose a model-aided deep Q-learning ap…
In this paper, we present a novel developmental reinforcement learning-based controller for a quadcopter with thrust vectoring capabilities. This multirotor UAV design has tilt-enabled rotors. It utilizes the rotor force magnitude and direction to achieve the desired state during flight. The control policy of this robot is learned using the policy transfer from the learned controller of the quadcopter (comparatively simple UAV design without thrust vectoring). This approach allows learning a control policy for systems with multiple inputs and multiple outputs. The performance of the learned po…
We propose a joint simulation and real-world learning framework for mapping navigation instructions and raw first-person observations to continuous control. Our model estimates the need for environment exploration, predicts the likelihood of visiting environment positions during execution, and controls the agent to both explore and visit high-likelihood positions. We introduce Supervised Reinforcement Asynchronous Learning (SuReAL). Learning uses both simulation and real environments without requiring autonomous flight in the physical environment during training, and combines supervised learni…
Multi-User Beamforming with Deep Reinforcement Learning in Sensing-Aided Communication
Department of Electronic Systems, Aalborg University（电子系统系，奥胡斯大学）；Department of Electrical and Computer Engineering, Aarhus University（电气与计算机工程系，奥胡斯大学）
Berkeley AI Research (BAIR), University of California, Berkeley（伯克利人工智能研究中心（BAIR），加州大学伯克利分校）
Deep reinforcement learning provides a promising approach for vision-based control of real-world robots. However, the generalization of such models depends critically on the quantity and variety of data available for training. This data can be difficult to obtain for some types of robotic systems, such as fragile, small-scale quadrotors. Simulated rendering and physics can provide for much larger datasets, but such data is inherently of lower quality: many of the phenomena that make the real-world autonomous flight problem challenging, such as complex physics and air currents, are modeled poor…
Active RIS-aided EH-NOMA Networks: A Deep Reinforcement Learning Approach
Anqing Normal University（安庆师范大学）；Dalian University of Technology（大连理工大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）；Xiamen University（厦门大学）；Zhejiang University（浙江大学）；Concordia University（康考迪亚大学）；Princeton University（普林斯顿大学）；University of Manchester（曼彻斯特大学）
The significant components of any successful autonomous flight system are task completion and collision avoidance. Most deep learning algorithms successfully execute these aspects under the environment and conditions they are trained. However, they fail when subjected to novel environments. This paper presents an autonomous multi-rotor flight algorithm, using Deep Reinforcement Learning augmented with Self-Attention Models, that can effectively reason when subjected to varying inputs. In addition to their reasoning ability, they are also interpretable, enabling it to be used under real-world c…
Edith Cowan University（伊迪丝·考恩大学）；Cybersecurity Cooperative Research Centre（网络安全合作研究中心）；CSIRO’s Data61（澳大利亚联邦科学与工业研究组织Data61研究所）；Charles Sturt University（查尔斯史都华大学）；The University of Sydney（悉尼大学）；The Alan Turing Institute（艾伦·图灵研究所）
Spear Phishing is a harmful cyber-attack facing business and individuals worldwide. Considerable research has been conducted recently into the use of Machine Learning (ML) techniques to detect spear-phishing emails. ML-based solutions may suffer from zero-day attacks; unseen attacks unaccounted for in the training data. As new attacks emerge, classifiers trained on older data are unable to detect these new varieties of attacks resulting in increasingly inaccurate predictions. Spear Phishing detection also faces scalability challenges due to the growth of the required features which is proporti…
Collision avoidance is a crucial task in vision-guided autonomous navigation. Solutions based on deep reinforcement learning (DRL) has become increasingly popular. In this work, we proposed several novel agent state and reward function designs to tackle two critical issues in DRL-based navigation solutions: 1) smoothness of the trained flight trajectories; and 2) model generalization to handle unseen environments. Formulated under a DRL framework, our model relies on margin reward and smoothness constraints to ensure UAVs fly smoothly while greatly reducing the chance of collision. The propose…
In this paper, the downlink packet scheduling problem for cellular networks is modeled, which jointly optimizes throughput, fairness and packet drop rate. Two genie-aided heuristic search methods are employed to explore the solution space. A deep reinforcement learning (DRL) framework with A2C algorithm is proposed for the optimization problem. Several methods have been utilized in the framework to improve the sampling and training efficiency and to adapt the algorithm to a specific scheduling problem. Numerical results show that DRL outperforms the baseline algorithm and achieves similar perf…
Deep Reinforcement Learning for Interference Suppression in RIS-Aided Space-Air-Ground Integrated Networks
Department of Artificial Intelligence, Indian Institute of Technology Hyderabad, Telangana, India（印度海得拉巴印度理工学院人工智能系）；School of Informatics, Kami, Kochi University of Technology, Kochi, Japan（日本科技大学信息学院）；Graduate School of Information Sciences, Tohoku University, Sendai, Miyagi, Japan（东北大学信息科学研究生院）
Deep Reinforcement Learning for Real-Time Ground Delay Program Revision and Corresponding Flight Delay Assignments
University of California at Berkeley（加州大学伯克利分校）；Northwestern University（西北大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；University of British Columbia（不列颠哥伦比亚大学）；University of Toronto（多伦多大学）；Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）
Graph Neural Network Aided Deep Reinforcement Learning for Resource Allocation in Dynamic Terahertz UAV Networks
Terahertz Wireless Communications (TWC) Laboratory, Shanghai Jiao Tong University（太赫兹无线通信实验室，上海交通大学）；Cooperative Medianet Innovation Center (CMIC), School of Information Science and Electronic Engineering, Shanghai Jiao Tong University（协同中继网络创新中心，信息科学与电子工程学院，上海交通大学）
Technology Innovation Institute（技术创新研究院）；Trinity College Dublin（都柏林圣三一学院）
Network slicing is a well assessed approach enabling virtualization of the mobile core and radio access network (RAN) in the emerging 5th Generation New Radio. Slicing is of paramount importance when dealing with the emerging and diverse vertical applications entailing heterogeneous sets of requirements. 5G is also envisioning Unmanned Aerial Vehicles (UAVs) to be a key element in the cellular network standard, aiming at their use as aerial base stations and exploiting their flexible and quick deployment to enhance the wireless network performance. This work presents a UAV-assisted 5G network,…
University of Bristol（布里斯托大学）；Vilicom UK Ltd.（英国Vilicom有限公司）；InterDigital Communications, Inc.（InterDigital通信公司）
Radio access network (RAN) technologies continue to evolve, with Open RAN gaining the most recent momentum. In the O-RAN specifications, the RAN intelligent controllers (RICs) are software-defined orchestration and automation functions for the intelligent management of RAN. This article introduces principles for machine learning (ML), in particular, reinforcement learning (RL) applications in the O-RAN stack. Furthermore, we review the state-of-the-art research in wireless networks and cast it onto the RAN framework and the hierarchy of the O-RAN architecture. We provide a taxonomy for the cha…
