Synthesising Reinforcement Learning Policies through Set-Valued Inductive Rule Learning
Vrije Universiteit Brussel（布鲁塞尔自由大学（荷兰语））；Université Libre de Bruxelles（布鲁塞尔自由大学（法语））；Delft University of Technology（代尔夫特理工大学）；Leiden Institute of Advanced Computer Science（莱顿大学高级计算机科学研究所）
Youri Coppens, Denis Steckelmacher, Catholijn M. Jonker, Ann Nowé
Today's advanced Reinforcement Learning algorithms produce black-box policies, that are often difficult to interpret and trust for a person. We introduce a policy distilling algorithm, building on the CN2 rule mining algorithm, that distills the policy into a rule-based decision system. At the core of our approach is the fact that an RL process does not just learn a policy, a mapping from states to actions, but also produces extra meta-information, such as action values indicating the quality of alternative actions. This meta-information can indicate whether more than one action is near-optimal for a certain state. We extend CN2 to make it able to leverage knowledge about equally-good actions to distill the policy into fewer rules, increasing its interpretability by a person. Then, to ensure that the rules explain a valid, non-degenerate policy, we introduce a refinement algorithm that fine-tunes the rules to obtain good performance when executed in the environment. We demonstrate the applicability of our algorithm on the Mario AI benchmark, a complex task that requires modern reinforcement learning algorithms including neural networks. The explanations we produce capture the learned policy in only a few rules, that allow a person to understand what the black-box agent learned. Source code: https://gitlab.ai.vub.ac.be/yocoppen/svcn2
Deep reinforcement learning is an increasingly popular technique for synthesising policies to control an agent's interaction with its environment. There is also growing interest in formally verifying that such policies are correct and execute safely. Progress has been made in this area by building on existing work for verification of deep neural networks and of continuous-state dynamical systems. In this paper, we tackle the problem of verifying probabilistic policies for deep reinforcement learning, which are used to, for example, tackle adversarial environments, break symmetries and manage t…
Decoupled Behavioral Cloning for Scalable Inductive Generalization in RL from Specifications
School of Computer Science, Georgia Institute of Technology, USA（美国佐治亚理工学院计算机科学学院）；Department of Computer Science and Engineering, Indian Institute of Technology Kanpur, India（印度理工学院坎浦尔分校计算机科学与工程系）
Heriot-Watt University（赫瑞-瓦特大学）；University of Edinburgh（爱丁堡大学）
Model-free reinforcement learning (RL) is a powerful tool to learn a broad range of robot skills and policies. However, a lack of policy interpretability can inhibit their successful deployment in downstream applications, particularly when differences in environmental conditions may result in unpredictable behaviour or generalisation failures. As a result, there has been a growing emphasis in machine learning around the inclusion of stronger inductive biases in models to improve generalisation. This paper proposes an alternative strategy, inverse value estimation for interpretable policy certi…
Most reinforcement learning methods for adaptive-traffic-signal-control require training from scratch to be applied on any new intersection or after any modification to the road network, traffic distribution, or behavioral constraints experienced during training. Considering 1) the massive amount of experience required to train such methods, and 2) that experience must be gathered by interacting in an exploratory fashion with real road-network-users, such a lack of transferability limits experimentation and applicability. Recent approaches enable learning policies that generalize for unseen ro…
LCRL is a software tool that implements model-free Reinforcement Learning (RL) algorithms over unknown Markov Decision Processes (MDPs), synthesising policies that satisfy a given linear temporal specification with maximal probability. LCRL leverages partially deterministic finite-state machines known as Limit Deterministic Buchi Automata (LDBA) to express a given linear temporal specification. A reward function for the RL algorithm is shaped on-the-fly, based on the structure of the LDBA. Theoretical guarantees under proper assumptions ensure the convergence of the RL algorithm to an optimal …
Recent breakthroughs in AI have shown the remarkable power of deep learning and deep reinforcement learning. These developments, however, have been tied to specific tasks, and progress in out-of-distribution generalization has been limited. While it is assumed that these limitations can be overcome by incorporating suitable inductive biases, the notion of inductive biases itself is often left vague and does not provide meaningful guidance. In the paper, I articulate a different learning approach where representations do not emerge from biases in a neural architecture but are learned over a giv…
R$^2$PO: Decoupling Rollout and Inference Policies for LLM Reasoning
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国家大学）
Time-Efficient Reinforcement Learning with Stochastic Stateful Policies
Intelligent Autonomous Systems（智能自主系统实验室）；Locomotion Laboratory（运动实验室）；German Research Center for AI (DFKI)（德国人工智能研究中心（DFKI））；Centre for Cognitive Science（认知科学中心）；TU Darmstadt（达姆施塔特工业大学）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
The problem of scheduling of workloads onto heterogeneous processors (e.g., CPUs, GPUs, FPGAs) is of fundamental importance in modern data centers. Current system schedulers rely on application/system-specific heuristics that have to be built on a case-by-case basis. Recent work has demonstrated ML techniques for automating the heuristic search by using black-box approaches which require significant training data and time, which make them challenging to use in practice. This paper presents Symphony, a scheduling framework that addresses the challenge in two ways: (i) a domain-driven Bayesian r…
Morphologically Symmetric Reinforcement Learning for Ambidextrous Bimanual Manipulation
TU Darmstadt（达姆施塔特工业大学）；Honda Research Institute Europe（本田欧洲研究所）；Istituto Italiano di Tecnologia（意大利理工学院）；DFKI（德国人工智能研究中心）；Robotics Institute Germany（德国机器人研究所）
Deepmind（深度思维）；University of Southern California（南加利福尼亚大学）；Google Research（谷歌研究院）
Despite significant progress on multi-agent reinforcement learning (MARL) in recent years, coordination in complex domains remains a challenge. Work in MARL often focuses on solving tasks where agents interact with all other agents and entities in the environment; however, we observe that real-world tasks are often composed of several isolated instances of local agent interactions (subtasks), and each agent can meaningfully focus on one subtask to the exclusion of all else in the environment. In these composite tasks, successful policies can often be decomposed into two levels of decision-maki…
In Reinforcement Learning (RL), Convolutional Neural Networks(CNNs) have been successfully applied as function approximators in Deep Q-Learning algorithms, which seek to learn action-value functions and policies in various environments. However, to date, there has been little work on the learning of symmetry-transformation equivariant representations of the input environment state. In this paper, we propose the use of Equivariant CNNs to train RL agents and study their inductive bias for transformation equivariant Q-value approximation. We demonstrate that equivariant architectures can dramati…
Carnegie Mellon University（卡内基梅隆大学）；Microsoft Research, New York（微软研究院（纽约））；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Microsoft Research, Redmond（微软研究院（雷德蒙德））
We study the exploration problem in episodic MDPs with rich observations generated from a small number of latent states. Under certain identifiability assumptions, we demonstrate how to estimate a mapping from the observations to latent states inductively through a sequence of regression and clustering steps -- where previously decoded latent states provide labels for later regression problems -- and use it to construct good exploration policies. We provide finite-sample guarantees on the quality of the learned state decoding function and exploration policies, and complement our theory with an…
Learning Index Selection with Structured Action Spaces
University of Cambridge（剑桥大学）
计算机系统的配置空间对于传统的和自动的调整策略来说是具有挑战性的。将任务特定的知识注入到用于任务的调谐器中可以允许更有效地探索候选配置。我们将这个想法应用到索引集选择任务中，以加速数据库工作负载。索引集选择已经适合于最近的vanilla deep RL应用，但真正的部署仍然遥不可及。在本文中，我们探讨了如何学习索引的选择，可以加强与特定任务的归纳偏见，特别是通过编码这些归纳偏见更好的动作结构。索引选择特定的动作表示出现时，问题是重新制定的排列学习，我们依靠最近的工作学习RL的政策排列。通过这种方法，我们建立了一个索引代理，能够实现改进的索引和验证其行为与特定任务的统计。早期的实验表明，与其他方法相比，我们的代理可以在相同的延迟水平下找到最多小40%的配置，并指示更直观的索引行为。
We present a novel neural architecture to solve graph optimization problems where the solution consists of arbitrary node labels, allowing us to solve hard problems like graph coloring. We train our model using reinforcement learning, specifically policy gradients, which gives us both a greedy and a probabilistic policy. Our architecture builds on a graph attention network and uses several inductive biases to improve solution quality. Our learned deterministic heuristics for graph coloring give better solutions than classical degree-based greedy heuristics and only take seconds to apply to gra…
Québec AI institute (Mila)（魁北克人工智能研究所（Mila））；Polytechnique Montréal（蒙特利尔高等理工学院）；McGill University（麦吉尔大学）；Element AI
In multi-agent reinforcement learning, discovering successful collective behaviors is challenging as it requires exploring a joint action space that grows exponentially with the number of agents. While the tractability of independent agent-wise exploration is appealing, this approach fails on tasks that require elaborate group strategies. We argue that coordinating the agents' policies can guide their exploration and we investigate techniques to promote such an inductive bias. We propose two policy regularization methods: TeamReg, which is based on inter-agent action predictability and CoachRe…
Large-scale finite element simulations of complex physical systems governed by partial differential equations (PDE) crucially depend on adaptive mesh refinement (AMR) to allocate computational budget to regions where higher resolution is required. Existing scalable AMR methods make heuristic refinement decisions based on instantaneous error estimation and thus do not aim for long-term optimality over an entire simulation. We propose a novel formulation of AMR as a Markov decision process and apply deep reinforcement learning (RL) to train refinement policies directly from simulation. AMR poses…
Imperial College London（帝国理工学院）；Universitat Pompeu Fabra（庞培法布拉大学）
In this paper we present ISA, an approach for learning and exploiting subgoals in episodic reinforcement learning (RL) tasks. ISA interleaves reinforcement learning with the induction of a subgoal automaton, an automaton whose edges are labeled by the task's subgoals expressed as propositional logic formulas over a set of high-level events. A subgoal automaton also consists of two special states: a state indicating the successful completion of the task, and a state indicating that the task has finished without succeeding. A state-of-the-art inductive logic programming system is used to learn a…
Agile Reinforcement Learning through Separable Neural Architecture and Applications
Department of Mechanical Engineering, Bangladesh University of Engineering and Technology（孟加拉国工程技术大学机械工程系）；Kevin T. Crofton Department of Aerospace and Ocean Engineering, Virginia Polytechnic Institute and State University（弗吉尼亚理工大学凯文·T·克罗夫顿航空航天与海洋工程系）
Shanghai Jiao Tong University（上海交通大学）；University of California San Diego（加利福尼亚大学圣迭戈分校）
A prominent approach to visual Reinforcement Learning (RL) is to learn an internal state representation using self-supervised methods, which has the potential benefit of improved sample-efficiency and generalization through additional learning signal and inductive biases. However, while the real world is inherently 3D, prior efforts have largely been focused on leveraging 2D computer vision techniques as auxiliary self-supervision. In this work, we present a unified framework for self-supervised learning of 3D representations for motor control. Our proposed framework consists of two phases: a …
RIDGECUT: Learning Graph Partitioning with Rings and Wedges
College of Computer Science and Artificial Intelligence, Shanghai Key Laboratory of Data Science（计算机科学与人工智能学院，上海数据科学重点实验室）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）；PayPal Inc.（PayPal公司）；Center for AI Safety（人工智能安全中心）；Qatar Computing Research Institute（卡塔尔计算研究所）；Hamad Bin Khalifa University（哈马德·本·卡西姆大学）；Computing and Mathematical Sciences (CMS) Division（计算与数学科学（CMS）部门）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（Mohamed bin Zayed人工智能大学（MBZUAI））；Fudan University（复旦大学）
