Safe interaction with the environment is one of the most challenging aspects of Reinforcement Learning (RL) when applied to real-world problems. This is particularly important when unsafe actions have a high or irreversible negative impact on the environment. In the context of network management operations, Remote Electrical Tilt (RET) optimisation is a safety-critical application in which exploratory modifications of antenna tilt angles of base stations can cause significant performance degradation in the network. In this paper, we propose a modular Safe Reinforcement Learning (SRL) architecture which is then used to address the RET optimisation in cellular networks. In this approach, a safety shield continuously benchmarks the performance of RL agents against safe baselines, and determines safe antenna tilt updates to be performed on the network. Our results demonstrate improved performance of the SRL agent over the baseline while ensuring the safety of the performed actions.
KTH Royal Institute of Technology（KTH皇家理工学院）；Ericsson Research（爱立信研究院）；LM Ericsson（爱立信公司）
Remote Electrical Tilt (RET) optimization is an efficient method for adjusting the vertical tilt angle of Base Stations (BSs) antennas in order to optimize Key Performance Indicators (KPIs) of the network. Reinforcement Learning (RL) provides a powerful framework for RET optimization because of its self-learning capabilities and adaptivity to environmental changes. However, an RL agent may execute unsafe actions during the course of its interaction, i.e., actions resulting in undesired network performance degradation. Since the reliability of services is critical for Mobile Network Operators (…
Mirror Descent Policy Optimisation for Robust Constrained Markov Decision Processes
Centre for Frontier AI Research, Agency for Science, Technology and Research (A*STAR)（前沿人工智能研究中心，科技研究局（A*STAR））；Singapore Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)（新加坡高性能计算研究所，科技研究局（A*STAR））；College of Computing and Data Science, Nanyang Technological University, Singapore（计算与数据科学学院，南洋理工大学，新加坡）
Five AI Ltd.（Five AI有限公司）；University of Oxford（牛津大学）；University of Edinburgh（爱丁堡大学）
Achieving a proper balance between planning quality, safety and efficiency is a major challenge for autonomous driving. Optimisation-based motion planners are capable of producing safe, smooth and comfortable plans, but often at the cost of runtime efficiency. On the other hand, naively deploying trajectories produced by efficient-to-run deep imitation learning approaches might risk compromising safety. In this paper, we present PILOT -- a planning framework that comprises an imitation neural network followed by an efficient optimiser that actively rectifies the network's plan, guaranteeing fu…
Reinforcement Learning with a Bilevel World-Model Architecture for Scan-Order Optimisation in Laser Directed Energy Deposition
College of Engineering, Design and Physical Sciences, Brunel University London（布鲁内尔大学伦敦工程、设计与物理科学学院）；Pattern Recognition Laboratory, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别实验室）；ISIS Neutron and Muon Source, Science and Technology Facilities Council, Rutherford Appleton Laboratory（Rutherford Appleton实验室，科学与技术设施委员会ISIS中子与μ子源）
In reinforcement learning (RL), an agent must explore an initially unknown environment in order to learn a desired behaviour. When RL agents are deployed in real world environments, safety is of primary concern. Constrained Markov decision processes (CMDPs) can provide long-term safety constraints; however, the agent may violate the constraints in an effort to explore its environment. This paper proposes a model-based RL algorithm called Explicit Explore, Exploit, or Escape ($E^{4}$), which extends the Explicit Explore or Exploit ($E^{3}$) algorithm to a robust CMDP setting. $E^4$ explicitly s…
Likelihood Hacking in Probabilistic Program Synthesis
University of Oxford, Department of Computer Science（牛津大学计算机科学系）；University of Edinburgh, School of Informatics（爱丁堡大学信息学院）；CIFAR Fellow, Learning in Machines and Brains（CIFAR Fellow, 机器与大脑学习）
In outdoor environments, mobile robots are required to navigate through terrain with varying characteristics, some of which might significantly affect the integrity of the platform. Ideally, the robot should be able to identify areas that are safe for navigation based on its own percepts about the environment while avoiding damage to itself. Bayesian optimisation (BO) has been successfully applied to the task of learning a model of terrain traversability while guiding the robot through more traversable areas. An issue, however, is that localisation uncertainty can end up guiding the robot to u…
MC-CPO: Mastery-Conditioned Constrained Policy Optimization for Pedagogically Safe Intelligent Tutoring Systems
School of Computing Sciences（计算科学学院）；Computer Engineering, University of Southern Mississippi, Hattiesburg, MS 39406, USA（计算机工程，密西西比大学，哈特斯伯格，MS 39406，USA）
We demonstrate the first application of deep reinforcement learning to autonomous driving. From randomly initialised parameters, our model is able to learn a policy for lane following in a handful of training episodes using a single monocular image as input. We provide a general and easy to obtain reward: the distance travelled by the vehicle without the safety driver taking control. We use a continuous, model-free deep reinforcement learning algorithm, with all exploration and optimisation performed on-vehicle. This demonstrates a new framework for autonomous driving which moves away from rel…
Alan Turing Institute（艾伦·图灵研究所）；Cranfield University（克兰菲尔德大学）
As the 5th Generation (5G) mobile networks are bringing about global societal benefits, the design phase for the 6th Generation (6G) has started. 6G will need to enable greater levels of autonomy, improve human machine interfacing, and achieve deep connectivity in more diverse environments. The need for increased explainability to enable trust is critical for 6G as it manages a wide range of mission critical services (e.g. autonomous driving) to safety critical tasks (e.g. remote surgery). As we migrate from traditional model-based optimisation to deep learning, the trust we have in our optimi…
