Towards sample-efficient episodic control with DAC-ML
Institute for Bioengineering of Catalonia (IBEC)（加泰罗尼亚生物工程研究所）；Universitat Pompeu Fabra (UPF)（庞培法布拉大学）；Catalan Institute for Research and Advanced Studies (ICREA)（加泰罗尼亚研究与高等研究院）
Ismael T. Freire, Adrián F. Amil, Vasiliki Vouloutsi, Paul F. M. J. Verschure
The sample-inefficiency problem in Artificial Intelligence refers to the inability of current Deep Reinforcement Learning models to optimize action policies within a small number of episodes. Recent studies have tried to overcome this limitation by adding memory systems and architectural biases to improve learning speed, such as in Episodic Reinforcement Learning. However, despite achieving incremental improvements, their performance is still not comparable to how humans learn behavioral policies. In this paper, we capitalize on the design principles of the Distributed Adaptive Control (DAC) theory of mind and brain to build a novel cognitive architecture (DAC-ML) that, by incorporating a hippocampus-inspired sequential memory system, can rapidly converge to effective action policies that maximize reward acquisition in a challenging foraging task.
Applied AI Institute, Deakin University（应用人工智能研究所，迪肯大学）
Episodic control enables sample efficiency in reinforcement learning by recalling past experiences from an episodic memory. We propose a new model-based episodic memory of trajectories addressing current limitations of episodic control. Our memory estimates trajectory values, guiding the agent towards good policies. Built upon the memory, we construct a complementary learning model via a dynamic hybrid control unifying model-based, episodic and habitual learning into a single architecture. Experiments demonstrate that our model allows significantly faster and better learning than other strong …
Laboratory of Computational Neuroscience（计算神经科学实验室）；École polytechnique fédérale de Lausanne（洛桑联邦理工学院）；Brain Mind Institute, School of Life Sciences, École polytechnique fédérale de Lausanne（洛桑联邦理工学院生命科学学院脑思维研究所）
Episodic control has been proposed as a third approach to reinforcement learning, besides model-free and model-based control, by analogy with the three types of human memory. i.e. episodic, procedural and semantic memory. But the theoretical properties of episodic control are not well investigated. Here I show that in deterministic tree Markov decision processes, episodic control is equivalent to a form of prioritized sweeping in terms of sample efficiency as well as memory and computation demands. For general deterministic and stochastic environments, prioritized sweeping performs better even…
Instituto Federal de Educação, Ciência e Tecnologia do Rio Grande do Sul（南里奥格兰德联邦教育、科学与技术学院）
Episodic control provides a highly sample-efficient method for reinforcement learning while enforcing high memory and computational requirements. This work proposes a simple heuristic for reducing these requirements, and an application to Model-Free Episodic Control (MFEC) is presented. Experiments on Atari games show that this heuristic successfully reduces MFEC computational demands while producing no significant loss of performance when conservative choices of hyperparameters are used. Consequently, episodic control becomes a more feasible option when dealing with reinforcement learning tas…
Kyushu University（九州大学）；Technical University of Munich（慕尼黑工业大学）；NetEase Fuxi AI Lab（网易伏羲人工智能实验室）；Singapore Management University（新加坡管理大学）；University of Alberta（阿尔伯塔大学）；The University of Tokyo（东京大学）；Tianjin University（天津大学）
Existing Deep Reinforcement Learning (DRL) algorithms suffer from sample inefficiency. Generally, episodic control-based approaches are solutions that leverage highly-rewarded past experiences to improve sample efficiency of DRL algorithms. However, previous episodic control-based approaches fail to utilize the latent information from the historical behaviors (e.g., state transitions, topological similarities, etc.) and lack scalability during DRL training. This work introduces Neural Episodic Control with State Abstraction (NECSA), a simple but effective state abstraction-based episodic contr…
End-to-end deep reinforcement learning has enabled agents to learn with little preprocessing by humans. However, it is still difficult to learn stably and efficiently because the learning method usually uses a nonlinear function approximation. Neural Episodic Control (NEC), which has been proposed in order to improve sample efficiency, is able to learn stably by estimating action values using a non-parametric method. In this paper, we propose an architecture that incorporates random projection into NEC to train with more stability. In addition, we verify the effectiveness of our architecture b…
State representation learning (SRL) in partially observable Markov decision processes has been studied to learn abstract features of data useful for robot control tasks. For SRL, acquiring domain-agnostic states is essential for achieving efficient imitation learning. Without these states, imitation learning is hampered by domain-dependent information useless for control. However, existing methods fail to remove such disturbances from the states when the data from experts and agents show large domain shifts. To overcome this issue, we propose a domain-adversarial and conditional state space mo…
East China Normal University（华东师范大学）；Tongji University（同济大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai University of International Business and Economics（上海对外经贸大学）；Shenzhen Loop Area Institute（深圳河套学院）
The research on deep reinforcement learning which estimates Q-value by deep learning has been attracted the interest of researchers recently. In deep reinforcement learning, it is important to efficiently learn the experiences that an agent has collected by exploring environment. We propose NEC2DQN that improves learning speed of a poor sample efficiency algorithm such as DQN by using good one such as NEC at the beginning of learning. We show it is able to learn faster than Double DQN or N-step DQN in the experiments of Pong.
Episodic self-imitation learning, a novel self-imitation algorithm with a trajectory selection module and an adaptive loss function, is proposed to speed up reinforcement learning. Compared to the original self-imitation learning algorithm, which samples good state-action pairs from the experience replay buffer, our agent leverages entire episodes with hindsight to aid self-imitation learning. A selection module is introduced to filter uninformative samples from each episode of the update. The proposed method overcomes the limitations of the standard self-imitation learning algorithm, a transi…
Model-Free Adaptive Optimal Control of Episodic Fixed-Horizon Manufacturing Processes using Reinforcement Learning
Institute Intelligent Systems Research Group, Karlsruhe University of Applied Sciences（智能系统研究组，卡尔斯鲁厄应用科学大学）；Institute for Applied Materials (IAM-CMS), Karlsruhe Institute of Technology（应用材料研究所（IAM-CMS），卡尔斯鲁厄理工学院）
Flexible Prefrontal Control over Hippocampal Episodic Memory for Goal-Directed Generalization
Department of Psychology, University of California, Davis, California, United States of America（心理学系，加州大学戴维斯分校）；Center for Neuroscience, University of California, Davis, California, United States of America（神经科学中心，加州大学戴维斯分校）；Astera Institute（Astera研究所）；Department of Computer Science, University of California, Davis, California, United States of America（计算机科学系，加州大学戴维斯分校）
VTT Technical Research Centre of Finland（芬兰VTT技术研究中心）；Zhejiang Lab（之江实验室）；Zhejiang University（浙江大学）；Auburn University（奥本大学）；University of Electro-Communications（电气通信大学）；University of Oulu（奥卢大学）
The age of information metric fails to correctly describe the intrinsic semantics of a status update. In an intelligent reflecting surface-aided cooperative relay communication system, we propose the age of semantics (AoS) for measuring semantics freshness of the status updates. Specifically, we focus on the status updating from a source node (SN) to the destination, which is formulated as a Markov decision process (MDP). The objective of the SN is to maximize the expected satisfaction of AoS and energy consumption under the maximum transmit power constraint. To seek the optimal control policy…
Tsinghua University（清华大学）；Peking University（北京大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Episodic memory-based methods can rapidly latch onto past successful strategies by a non-parametric memory and improve sample efficiency of traditional reinforcement learning. However, little effort is put into the continuous domain, where a state is never visited twice, and previous episodic methods fail to efficiently aggregate experience across trajectories. To address this problem, we propose Generalizable Episodic Memory (GEM), which effectively organizes the state-action values of episodic memory in a generalizable manner and supports implicit planning on memorized trajectories. GEM util…
Sensory inference under conditions of uncertainty is a major problem in both machine learning and computational neuroscience. An important but poorly understood aspect of sensory processing is the role of active sensing. Here, we present a Bayes-optimal inference and control framework for active sensing, C-DAC (Context-Dependent Active Controller). Unlike previously proposed algorithms that optimize abstract statistical objectives such as information maximization (Infomax) [Butko & Movellan, 2010] or one-step look-ahead accuracy [Najemnik & Geisler, 2005], our active sensing model directly min…
Sensory inference under conditions of uncertainty is a major problem in both machine learning and computational neuroscience. An important but poorly understood aspect of sensory processing is the role of active sensing. Here, we present a Bayes-optimal inference and control framework for active sensing, C-DAC (Context-Dependent Active Controller). Unlike previously proposed algorithms that optimize abstract statistical objectives such as information maximization (Infomax) [Butko & Movellan, 2010] or one-step look-ahead accuracy [Najemnik & Geisler, 2005], our active sensing model directly min…
Meta AI Research（Meta人工智能研究院）；University College London（伦敦大学学院）
In recent years, a number of reinforcement learning (RL) methods have been proposed to explore complex environments which differ across episodes. In this work, we show that the effectiveness of these methods critically relies on a count-based episodic term in their exploration bonus. As a result, despite their success in relatively simple, noise-free settings, these methods fall short in more realistic scenarios where the state space is vast and prone to noise. To address this limitation, we introduce Exploration via Elliptical Episodic Bonuses (E3B), a new method which extends count-based epi…
We consider scenarios from the real-time strategy game StarCraft as new benchmarks for reinforcement learning algorithms. We propose micromanagement tasks, which present the problem of the short-term, low-level control of army members during a battle. From a reinforcement learning point of view, these scenarios are challenging because the state-action space is very large, and because there is no obvious feature representation for the state-action evaluation function. We describe our approach to tackle the micromanagement scenarios with deep neural network controllers from raw state features gi…
While reinforcement learning (RL) provides a framework for learning through trial and error, translating RL algorithms into the real world has remained challenging. A major hurdle to real-world application arises from the development of algorithms in an episodic setting where the environment is reset after every trial, in contrast with the continual and non-episodic nature of the real-world encountered by embodied agents such as humans and robots. Prior works have considered an alternating approach where a forward policy learns to solve the task and the backward policy learns to reset the envi…
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与学院）；School of Mechanical Engineering, University of Science and Technology Beijing（北京科技大学机械工程学院）
Institute for Bioengineering of Catalonia (IBEC)（加泰罗尼亚生物工程研究所）；Barcelona Institute of Science and Technology (BIST)（巴塞罗那科学技术研究院）；Catalan Institution for Research and Advanced Studies (ICREA)（加泰罗尼亚高级研究学院）
In the past decade, society has experienced notable growth in a variety of technological areas. However, the Fourth Industrial Revolution has not been embraced yet. Industry 4.0 imposes several challenges which include the necessity of new architectural models to tackle the uncertainty that open environments represent to cyber-physical systems (CPS). Waste Electrical and Electronic Equipment (WEEE) recycling plants stand for one of such open environments. Here, CPSs must work harmoniously in a changing environment, interacting with similar and not so similar CPSs, and adaptively collaborating …
CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video
University College London（伦敦大学学院）；Imperial College London（帝国理工学院）；Durham University（杜伦大学）；Shanghai Jiao Tong University（上海交通大学）；Queen Mary University of London（伦敦玛丽女王大学）
High-fidelity social learning via shared episodic memories enhances collaborative foraging through mnemonic convergence
Donders Institute for Brain, Cognition and Behaviour（唐德斯脑、认知与行为研究所）；Radboud University（拉德堡德大学）；Alicante Institute of Neuroscience（阿利坎特神经科学研究所）；Universidad Miguel Hernandez de Elche（埃尔切米格尔·埃尔南德斯大学）
Lessons from Neuroscience for AI: How integrating Actions, Compositional Structure and Episodic Memory could enable Safe, Interpretable and Human-Like AI
