The Bottleneck Simulator: A Model-based Deep Reinforcement Learning Approach
Montreal Institute for Learning Algorithms (MILA)（蒙特利尔学习算法研究所（MILA））；McGill University（麦吉尔大学）；Facebook Artificial Intelligence Research (FAIR)（Facebook人工智能研究院（FAIR））
Deep reinforcement learning has recently shown many impressive successes. However, one major obstacle towards applying such methods to real-world problems is their lack of data-efficiency. To this end, we propose the Bottleneck Simulator: a model-based reinforcement learning method which combines a learned, factorized transition model of the environment with rollout simulations to learn an effective policy from few examples. The learned transition model employs an abstract, discrete (bottleneck) state, which increases sample efficiency by reducing the number of model parameters and by exploiting structural properties of the environment. We provide a mathematical analysis of the Bottleneck Simulator in terms of fixed points of the learned policy, which reveals how performance is affected by four distinct sources of error: an error related to the abstract space structure, an error related to the transition model estimation variance, an error related to the transition model estimation bias, and an error related to the transition model class bias. Finally, we evaluate the Bottleneck Simulator on two natural language processing tasks: a text adventure game and a real-world, complex dialogue response selection task. On both tasks, the Bottleneck Simulator yields excellent performance beating competing approaches.
East China Normal University（华东师范大学）；Huawei Technologies Co., Ltd.（华为技术有限公司）；The Chinese University of Hong Kong-Shenzhen（香港中文大学（深圳））
A novel simulator called VMAgent is introduced to help RL researchers better explore new methods, especially for virtual machine scheduling. VMAgent is inspired by practical virtual machine (VM) scheduling tasks and provides an efficient simulation platform that can reflect the real situations of cloud computing. Three scenarios (fading, recovering, and expansion) are concluded from practical cloud computing and corresponds to many reinforcement learning challenges (high dimensional state and action spaces, high non-stationarity, and life-long demand). VMAgent provides flexible configurations …
Center for Research on Intelligent System and Engineering（智能系统与工程研究中心）；Institute of Automation, CAS（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Model-based reinforcement learning (MBRL) is believed to have higher sample efficiency compared with model-free reinforcement learning (MFRL). However, MBRL is plagued by dynamics bottleneck dilemma. Dynamics bottleneck dilemma is the phenomenon that the performance of the algorithm falls into the local optimum instead of increasing when the interaction step with the environment increases, which means more data can not bring better performance. In this paper, we find that the trajectory reward estimation error is the main reason that causes dynamics bottleneck dilemma through theoretical analy…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Reinforcement learning (RL) workloads take a notoriously long time to train due to the large number of samples collected at run-time from simulators. Unfortunately, cluster scale-up approaches remain expensive, and commonly used CPU implementations of simulators induce high overhead when switching back and forth between GPU computations. We explore two optimizations that increase RL data collection efficiency by increasing GPU utilization: (1) GPU vectorization: parallelizing simulation on the GPU for increased hardware parallelism, and (2) simulator kernel fusion: fusing multiple simulation s…
Peking University（北京大学）；University of California, Berkeley（加州大学伯克利分校）；Carnegie Mellon University（卡内基梅隆大学）
Microscopic epidemic models are powerful tools for government policy makers to predict and simulate epidemic outbreaks, which can capture the impact of individual behaviors on the macroscopic phenomenon. However, existing models only consider simple rule-based individual behaviors, limiting their applicability. This paper proposes a deep-reinforcement-learning-powered microscopic model named Microscopic Pandemic Simulator (MPS). By replacing rule-based agents with rational agents whose behaviors are driven to maximize rewards, the MPS provides a better approximation of real world dynamics. To …
ETH Zürich（苏黎世联邦理工学院）；University of Konstanz（康斯坦茨大学）；ETH AI Center（苏黎世联邦理工学院人工智能中心）
Robustness to adversarial perturbations has been explored in many areas of computer vision. This robustness is particularly relevant in vision-based reinforcement learning, as the actions of autonomous agents might be safety-critic or impactful in the real world. We investigate the susceptibility of vision-based reinforcement learning agents to gradient-based adversarial attacks and evaluate a potential defense. We observe that Bottleneck Attention Modules (BAM) included in CNN architectures can act as potential tools to increase robustness against adversarial attacks. We show how learned atte…
Despite the significant progress of deep reinforcement learning (RL) in solving sequential decision making problems, RL agents often overfit to training environments and struggle to adapt to new, unseen environments. This prevents robust applications of RL in real world situations, where system dynamics may deviate wildly from the training settings. In this work, our primary contribution is to propose an information theoretic regularization objective and an annealing-based optimization method to achieve better generalization ability in RL agents. We demonstrate the extreme generalization benef…
Deep reinforcement learning (DRL) agents are often sensitive to visual changes that were unseen in their training environments. To address this problem, we leverage the sequential nature of RL to learn robust representations that encode only task-relevant information from observations based on the unsupervised multi-view setting. Specifically, we introduce a novel contrastive version of the Multi-View Information Bottleneck (MIB) objective for temporal data. We train RL agents from pixels with this auxiliary objective to learn robust representations that can compress away task-irrelevant infor…
When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning
Tsinghua University（清华大学）；Indian Institute of Technology（印度理工学院）；Shanghai Jiaotong University（上海交通大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Shanghai AI Laboratory（上海人工智能实验室）
Rule-Bottleneck Reinforcement Learning: Joint Explanation and Decision Optimization for Resource Allocation with Language Agents
Harvard John A. Paulson School of Engineering and Applied Sciences（哈佛大学约翰·A·保尔森工程与应用科学学院）；Harvard T.H. Chan School of Public Health（哈佛大学陈曾熙公共卫生学院）；Google DeepMind（谷歌DeepMind）
Mahjax: A GPU-Accelerated Mahjong Simulator for Reinforcement Learning in JAX
The University of Tokyo（东京大学）；RIKEN AIP（日本理化学研究院AIP）；Nara Institute of Science and Technology（奈良科学技術大學）；Kobe University（Kobe大学）；Kyoto University（京都大学）；ATR；The University of Sydney（悉尼大学）
Reinforcement-learning robotic sailboats: simulator and preliminary results
Fluminense Federal University（弗卢米嫩塞联邦大学）；Center Inria de l’Université de Lille（里尔大学Inria中心）；Federal University of Rio Grande do Norte（北里奥格兰德联邦大学）；Center Inria Chile（智利Inria中心）
We consider offline reinforcement learning (RL) with heterogeneous agents under severe data scarcity, i.e., we only observe a single historical trajectory for every agent under an unknown, potentially sub-optimal policy. We find that the performance of state-of-the-art offline and model-based RL methods degrade significantly given such limited data availability, even for commonly perceived "solved" benchmark settings such as "MountainCar" and "CartPole". To address this challenge, we propose PerSim, a model-based offline RL approach which first learns a personalized simulator for each agent by…
Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy
University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Potsdam Institute for Climate Impact Research（波茨坦气候影响研究所）；Technical University of Berlin（柏林技术大学）；University of Southern California（南加州大学）；University of Hong Kong（香港大学）；Broad Institute of MIT and Harvard（MIT和哈佛大学Broad研究所）
Data-driven simulator of multi-animal behavior with unknown dynamics via offline and online reinforcement learning
Graduate School of Informatics, Nagoya University, Japan（名古屋大学信息学研究科）；RIKEN Center for Advanced Intelligence Project, Japan（RIKEN高级智能项目研究中心）；Graduate School of Arts and Sciences, The University of Tokyo, Japan（东京大学文学系研究科）；Project team for SIP, Japan（SIP项目团队）；Japan Agency for Marine-Earth Science and Technology, Japan（日本海洋地球科学技术机构）；Faculty of Education, Shitennoji University, Japan（世田谷大学教育学部）；Graduate School of Engineering, The University of Tokyo, Japan（东京大学工学研究科）；Graduate School of Science and Technology, Niigata University, Japan（新潟大学科学技术研究科）；Graduate School of Science, Nagoya University, Japan（名古屋大学理学研究科）；Principles of Informatics Research Division, National Institute of Informatics, Japan（信息学原理研究部门，日本信息处理技术研究所）；Graduate School of Information Science, The University of Osaka, Japan（大阪大学信息科学研究科）
We propose an approach for inverse reinforcement learning from hetero-domain which learns a reward function in the simulator, drawing on the demonstrations from the real world. The intuition behind the method is that the reward function should not only be oriented to imitate the experts, but should encourage actions adjusted for the dynamics difference between the simulator and the real world. To achieve this, the widely used GAN-inspired IRL method is adopted, and its discriminator, recognizing policy-generating trajectories, is modified with the quantification of dynamics difference. The tra…
ISAE-SUPAERO, Université de Toulouse（图卢兹大学高等航空航天学院）；Sandbox AQ（Sandbox AQ公司）
Learning a good state representation is a critical skill when dealing with multiple tasks in Reinforcement Learning as it allows for transfer and better generalization between tasks. However, defining what constitute a useful representation is far from simple and there is so far no standard method to find such an encoding. In this paper, we argue that distillation -- a process that aims at imitating a set of given policies with a single neural network -- can be used to learn a state representation displaying favorable characteristics. In this regard, we define three criteria that measure desir…
Addressing a real world sequential decision problem with Reinforcement Learning (RL) usually starts with the use of a simulated environment that mimics real conditions. We present a novel open source RL environment for realistic crop management tasks. gym-DSSAT is a gym interface to the Decision Support System for Agrotechnology Transfer (DSSAT), a high fidelity crop simulator. DSSAT has been developped over the last 30 years and is widely recognized by agronomists. gym-DSSAT comes with predefined simulations based on real world maize experiments. The environment is as easy to use as any gym e…
Sony AI（索尼人工智能）；Sapienza University of Rome（罗马大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
The year 2020 has seen the COVID-19 virus lead to one of the worst global pandemics in history. As a result, governments around the world are faced with the challenge of protecting public health, while keeping the economy running to the greatest extent possible. Epidemiological models provide insight into the spread of these types of diseases and predict the effects of possible intervention policies. However, to date,the even the most data-driven intervention policies rely on heuristics. In this paper, we study how reinforcement learning (RL) can be used to optimize mitigation policies that mi…
We study the optimal sample complexity in large-scale Reinforcement Learning (RL) problems with policy space generalization, i.e. the agent has a prior knowledge that the optimal policy lies in a known policy space. Existing results show that without a generalization model, the sample complexity of an RL algorithm will inevitably depend on the cardinalities of state space and action space, which are intractably large in many practical problems. To avoid such undesirable dependence on the state and action space sizes, this paper proposes a new notion of eluder dimension for the policy space, wh…
University of California, Santa Barbara（加州大学圣巴巴拉分校）
In recent years, fully differentiable rigid body physics simulators have been developed, which can be used to simulate a wide range of robotic systems. In the context of reinforcement learning for control, these simulators theoretically allow algorithms to be applied directly to analytic gradients of the reward function. However, to date, these gradients have proved extremely challenging to use, and are outclassed by algorithms using no gradient information at all. In this work we present a novel algorithm, cross entropy analytic policy gradients, that is able to leverage these gradients to ou…
