In the context of reinforcement learning we introduce the concept of criticality of a state, which indicates the extent to which the choice of action in that particular state influences the expected return. That is, a state in which the choice of action is more likely to influence the final outcome is considered as more critical than a state in which it is less likely to influence the final outcome.
We formulate a criticality-based varying step number algorithm (CVS) - a flexible step number algorithm that utilizes the criticality function provided by a human, or learned directly from the environment. We test it in three different domains including the Atari Pong environment, Road-Tree environment, and Shooter environment. We demonstrate that CVS is able to outperform popular learning algorithms such as Deep Q-Learning and Monte Carlo.
Reinforcement learning methods carry a well known bias-variance trade-off in n-step algorithms for optimal control. Unfortunately, this has rarely been addressed in current research. This trade-off principle holds independent of the choice of the algorithm, such as n-step SARSA, n-step Expected SARSA or n-step Tree backup. A small n results in a large bias, while a large n leads to large variance. The literature offers no straightforward recipe for the best choice of this value. While currently all n-step algorithms use a fixed value of n over the state space we extend the framework of n-step …
Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles
College of Intelligence Science and Technology, National University of Defense Technology（国防科技大学智能科学学院）；School of Automation, Central South University（中南大学自动化学院）；Department of Cognitive Robotics, Delft University of Technology（代尔夫特理工大学认知机器人学系）
Reinforcement learning (RL) algorithms find applications in inventory control, recommender systems, vehicular traffic management, cloud computing and robotics. The real-world complications of many tasks arising in these domains makes them difficult to solve with the basic assumptions underlying classical RL algorithms. RL agents in these applications often need to react and adapt to changing operating conditions. A significant part of research on single-agent RL techniques focuses on developing algorithms when the underlying assumption of stationary environment model is relaxed. This paper pro…
Successfully navigating a complex environment to obtain a desired outcome is a difficult task, that up to recently was believed to be capable only by humans. This perception has been broken down over time, especially with the introduction of deep reinforcement learning, which has greatly increased the difficulty of tasks that can be automated. However, for traditional reinforcement learning agents this requires an environment to be able to provide frequent extrinsic rewards, which are not known or accessible for many real-world environments. This project aims to explore and contrast existing r…
DeepMind（深度思维）；Harvard University（哈佛大学）；University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所（Amii））
The ability to learn continually is essential in a complex and changing world. In this paper, we characterize the behavior of canonical value-based deep reinforcement learning (RL) approaches under varying degrees of non-stationarity. In particular, we demonstrate that deep RL agents lose their ability to learn good policies when they cycle through a sequence of Atari 2600 games. This phenomenon is alluded to in prior work under various guises -- e.g., loss of plasticity, implicit under-parameterization, primacy bias, and capacity loss. We investigate this phenomenon closely at scale and analy…
University of Michigan（密歇根大学）；Ford Motor Company（福特汽车公司）
It is essential for an automated vehicle in the field to perform discretionary lane changes with appropriate roadmanship - driving safely and efficiently without annoying or endangering other road users - under a wide range of traffic cultures and driving conditions. While deep reinforcement learning methods have excelled in recent years and been applied to automated vehicle driving policy, there are concerns about their capability to quickly adapt to unseen traffic with new environment dynamics. We formulate this challenge as a multi-Markov Decision Processes (MDPs) adaptation problem and dev…
Massachusetts Institute of Technology（麻省理工学院）；Brown University（布朗大学）；Microsoft Research（微软研究院）
Recent research has turned to Reinforcement Learning (RL) to solve challenging decision problems, as an alternative to hand-tuned heuristics. RL can learn good policies without the need for modeling the environment's dynamics. Despite this promise, RL remains an impractical solution for many real-world systems problems. A particularly challenging case occurs when the environment changes over time, i.e. it exhibits non-stationarity. In this work, we characterize the challenges introduced by non-stationarity, shed light on the range of approaches to them and develop a robust framework for addres…
Age and Power Minimization via Meta-Deep Reinforcement Learning in UAV Networks
Centre for Wireless Communications (CWC), University of Oulu（奥卢大学无线通信中心）；CEMSE Division, King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学CEMSE学部）
Neural Operator based Reinforcement Learning for Control of first-order PDEs with Spatially-Varying State Delay
College of Information Science and Technology, Donghua University（东华大学信息科学与技术学院）；College of Information Engineering, Shanghai Maritime University（上海海事大学信息工程学院）
National Laboratory for Scientific Computing (LNCC)（国家科学计算实验室（LNCC））；DASA；DeepMind
Network seeding for efficient information diffusion over time-varying graphs~(TVGs) is a challenging task with many real-world applications. There are several ways to model this spatio-temporal influence maximization problem, but the ultimate goal is to determine the best moment for a node to start the diffusion process. In this context, we propose Spatio-Temporal Influence Maximization~(STIM), a model trained with Reinforcement Learning and Graph Embedding over a set of artificial TVGs that is capable of learning the temporal behavior and connectivity pattern of each node, allowing it to pred…
We consider primal-dual-based reinforcement learning (RL) in episodic constrained Markov decision processes (CMDPs) with non-stationary objectives and constraints, which plays a central role in ensuring the safety of RL in time-varying environments. In this problem, the reward/utility functions and the state transition functions are both allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain known variation budgets. Designing safe RL algorithms in time-varying environments is particularly challenging because of the need to integrate the constraint vi…
United States Navy（美国海军）；University of Maryland（马里兰大学）；United States Naval Academy（美国海军学院）
Data poisoning for reinforcement learning has historically focused on general performance degradation, and targeted attacks have been successful via perturbations that involve control of the victim's policy and rewards. We introduce an insidious poisoning attack for reinforcement learning which causes agent misbehavior only at specific target states - all while minimally modifying a small fraction of training observations without assuming any control over policy or reward. We accomplish this by adapting a recent technique, gradient alignment, to reinforcement learning. We test our method and d…
UC Berkeley（加州大学伯克利分校）；CMU（卡内基梅隆大学）；International Computer Science Institute (ICSI)（国际计算机科学研究所）
Reinforcement learning is a powerful technique to train an agent to perform a task. However, an agent that is trained using reinforcement learning is only capable of achieving the single task that is specified via its reward function. Such an approach does not scale well to settings in which an agent needs to perform a diverse set of tasks, such as navigating to varying positions in a room or moving objects to varying locations. Instead, we propose a method that allows an agent to automatically discover the range of tasks that it is capable of performing. We use a generator network to propose …
We propose a new approach of NoSQL database index selection. For different workloads, we select different indexes and their different parameters to optimize the database performance. The approach builds a deep reinforcement learning model to select an optimal index for a given fixed workload and adapts to a changing workload. Experimental results show that, Deep Reinforcement Learning Index Selection Approach (DRLISA) has improved performance to varying degrees according to traditional single index structures.
University of Washington（华盛顿大学）；Microsoft Research（微软研究院）
We introduce an online popularity prediction and tracking task as a benchmark task for reinforcement learning with a combinatorial, natural language action space. A specified number of discussion threads predicted to be popular are recommended, chosen from a fixed window of recent comments to track. Novel deep reinforcement learning architectures are studied for effective modeling of the value function associated with actions comprised of interdependent sub-actions. The proposed model, which represents dependence between sub-actions through a bi-directional LSTM, gives the best performance acr…
School of Geographic Sciences, East China Normal University（华东师范大学地理科学学院）；University of South Carolina（南卡罗来纳大学）
Conventional simulations on multi-exit indoor evacuation focus primarily on how to determine a reasonable exit based on numerous factors in a changing environment. Results commonly include some congested and other under-utilized exits, especially with massive pedestrians. We propose a multi-exit evacuation simulation based on Deep Reinforcement Learning (DRL), referred to as the MultiExit-DRL, which involves in a Deep Neural Network (DNN) framework to facilitate state-to-action mapping. The DNN framework applies Rainbow Deep Q-Network (DQN), a DRL algorithm that integrates several advanced DQN…
Toward Global Intent Inference for Human Motion by Inverse Reinforcement Learning
Machines in Motion Laboratory, New York University, USA（纽约大学运动机器实验室）；LAAS-CNRS, Université Paul Sabatier, CNRS, Toulouse, France（拉劳斯-CNRS，保罗·萨巴蒂埃大学，法国图卢兹）；Image and Pervasive Access Laboratory (IPAL), CNRS-UMI, Singapore（图像与普及访问实验室（IPAL），CNRS-UMI，新加坡）；Artificial and Natural Intelligence Toulouse Institute (ANITI), Toulouse（图卢兹人工智能与自然智能研究所（ANITI），图卢兹）
A Safe Reinforcement Learning driven Weights-varying Model Predictive Control for Autonomous Vehicle Motion Control
Technical University Munich（慕尼黑工业大学）；TUM School of Engineering and Design（慕尼黑工业大学工程与设计学院）；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与机器智能研究所）
Multi-objective task scheduling (MOTS) is the task scheduling while optimizing multiple and possibly contradicting constraints. A challenging extension of this problem occurs when every individual task is a multi-objective optimization problem by itself. While deep reinforcement learning (DRL) has been successfully applied to complex sequential problems, its application to the MOTS domain has been stymied by two challenges. The first challenge is the inability of the DRL algorithm to ensure that every item is processed identically regardless of its position in the queue. The second challenge i…
The significant components of any successful autonomous flight system are task completion and collision avoidance. Most deep learning algorithms successfully execute these aspects under the environment and conditions they are trained. However, they fail when subjected to novel environments. This paper presents an autonomous multi-rotor flight algorithm, using Deep Reinforcement Learning augmented with Self-Attention Models, that can effectively reason when subjected to varying inputs. In addition to their reasoning ability, they are also interpretable, enabling it to be used under real-world c…
Google Brain（谷歌大脑）；Zhejiang University（浙江大学）；Institut Mines-TELECOM（法国国立高等矿业电信学校联盟）；TELECOM SudParis（南巴黎电信学院）
We consider the problem of learning to behave optimally in a Markov Decision Process when a reward function is not specified, but instead we have access to a set of demonstrators of varying performance. We assume the demonstrators are classified into one of k ranks, and use ideas from ordinal regression to find a reward function that maximizes the margin between the different ranks. This approach is based on the idea that agents should not only learn how to behave from experts, but also how not to behave from non-experts. We show there are MDPs where important differences in the reward functio…
Reactor Optimization Benchmark by Reinforcement Learning
Soreq Nuclear Research Center（索雷克核研究中心）；Ben-Gurion University（本-古里安大学）；Scientific Computing Center, Nuclear Research Center – Negev（内盖夫核研究中心科学计算中心）；Technion – Israel Institute of Technology（以色列理工学院）
Eindhoven University of Technology（埃因霍温理工大学）；Nokia Bell Labs（诺基亚贝尔实验室）
The key challenge in admission control in wireless networks is to strike an optimal trade-off between the blocking probability for new requests while minimizing the dropping probability of ongoing requests. We consider two approaches for solving the admission control problem: i) the typically adopted threshold policy and ii) our proposed policy relying on reinforcement learning with neural networks. Extensive simulation experiments are conducted to analyze the performance of both policies. The results show that the reinforcement learning policy outperforms the threshold-based policies in the s…
A reinforcement-learning-based non-uniform compressed sensing (NCS) framework for time-varying signals is introduced. The proposed scheme, referred to as RL-NCS, aims to boost the performance of signal recovery through an optimal and adaptive distribution of sensing energy among two groups of coefficients of the signal, referred to as the region of interest (ROI) coefficients and non-ROI coefficients. The coefficients in ROI usually have greater importance and need to be reconstructed with higher accuracy compared to non-ROI coefficients. In order to accomplish this task, the ROI is predicted …
We consider the dynamic multichannel access problem, which can be formulated as a partially observable Markov decision process (POMDP). We first propose a model-free actor-critic deep reinforcement learning based framework to explore the sensing policy. To evaluate the performance of the proposed sensing policy and the framework's tolerance against uncertainty, we test the framework in scenarios with different channel switching patterns and consider different switching probabilities. Then, we consider a time-varying environment to identify the adaptive ability of the proposed framework. Additi…
Learning-at-Criticality in Large Language Models for Quantum Field Theory and Beyond
Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China（理论物理研究所，中国科学院，北京）；Hefei National Laboratory, University of Science and Technology of China, Hefei 230088, China（合肥国家实验室，中国科学技术大学，合肥）；Department of Physics, University of Massachusetts, Amherst, MA 01003, USA（物理系，马萨诸塞大学，阿姆赫斯特）；Institute of Physics, Chinese Academy of Sciences, Beijing 100190, China（物理研究所，中国科学院，北京）；School of Fundamental Physics and Mathematical Sciences, Hangzhou Institute for Advanced Study, UCAS, Hangzhou 310024, China（基础物理与数学科学学院，杭州高等研究院，UCAS，杭州）；Hefei National Laboratory, Hefei 230088, China（合肥国家实验室，合肥）；Hefei National Laboratory for Physical Sciences at the Microscale（微尺度物理科学国家实验室，合肥）；Department of Modern Physics, University of Science and Technology of China, Hefei 230026, China（现代物理系，中国科学技术大学，合肥）
Massachusetts Institute of Technology（麻省理工学院）；Rutgers University（罗格斯大学）
Continuous state spaces and stochastic, switching dynamics characterize a number of rich, realworld domains, such as robot navigation across varying terrain. We describe a reinforcementlearning algorithm for learning in these domains and prove for certain environments the algorithm is probably approximately correct with a sample complexity that scales polynomially with the state-space dimension. Unfortunately, no optimal planning techniques exist in general for such problems; instead we use fitted value iteration to solve the learned MDP, and include the error due to approximate planning in ou…
