Reinforcement learning (RL) is a foundation of learning in biological systems and provides a framework to address numerous challenges with real-world artificial intelligence applications. Efficient implementations of RL techniques could allow for agents deployed in edge-use cases to gain novel abilities, such as improved navigation, understanding complex situations and critical decision making. Towards this goal, we describe a flexible architecture to carry out reinforcement learning on neuromorphic platforms. This architecture was implemented using an Intel neuromorphic processor and demonstrated solving a variety of tasks using spiking dynamics. Our study proposes a usable energy efficient solution for real-world RL applications and demonstrates applicability of the neuromorphic platforms for RL problems.
The process of continuously reallocating funds into financial assets, aiming to increase the expected return of investment and minimizing the risk, is known as portfolio management. Processing speed and energy consumption of portfolio management have become crucial as the complexity of their real-world applications increasingly involves high-dimensional observation and action spaces and environment uncertainty, which their limited onboard resources cannot offset. Emerging neuromorphic chips inspired by the human brain increase processing speed by up to 1000 times and reduce power consumption b…
Quantum Computing and Neuromorphic Computing for Safe, Reliable, and explainable Multi-Agent Reinforcement Learning: Optimal Control in Autonomous Robotics
Computational Brain Lab, Department of Computer Science, Rutgers University（罗格斯大学计算机科学系计算大脑实验室）
Energy-efficient mapless navigation is crucial for mobile robots as they explore unknown environments with limited on-board resources. Although the recent deep reinforcement learning (DRL) approaches have been successfully applied to navigation, their high energy consumption limits their use in several robotic applications. Here, we propose a neuromorphic approach that combines the energy-efficiency of spiking neural networks with the optimality of DRL and benchmark it in learning control policies for mapless navigation. Our hybrid framework, spiking deep deterministic policy gradient (SDDPG),…
Self-training superconducting neuromorphic circuits using reinforcement learning rules
National Institute of Standards and Technology（美国国家标准与技术研究院）；University of Colorado Boulder（科罗拉多大学博尔德分校）；Colgate University（科尔盖特大学）；Colorado State University（科罗拉多州立大学）
A Neuromorphic Architecture for Reinforcement Learning from Real-Valued Observations
Universidade Federal de Pernambuco（伯南布哥联邦大学）；Centro de Informática（信息中心）；Western Sydney University（西悉尼大学）；MARCS Institute（MARCS研究所）；International Centre for Neuromorphic Engineering（国际神经形态工程中心）
Recently, significant progress has been made in solving sophisticated problems among various domains by using reinforcement learning (RL), which allows machines or agents to learn from interactions with environments rather than explicit supervision. As the end of Moore's law seems to be imminent, emerging technologies that enable high performance neuromorphic hardware systems are attracting increasing attention. Namely, neuromorphic architectures that leverage memristors, the programmable and nonvolatile two-terminal devices, as synaptic weights in hardware neural networks, are candidates of c…
We introduce AndroidEnv, an open-source platform for Reinforcement Learning (RL) research built on top of the Android ecosystem. AndroidEnv allows RL agents to interact with a wide variety of apps and services commonly used by humans through a universal touchscreen interface. Since agents train on a realistic simulation of an Android device, they have the potential to be deployed on real devices. In this report, we give an overview of the environment, highlighting the significant features it provides for research, and we present an empirical evaluation of some popular reinforcement learning ag…
Low precision networks in the reinforcement learning (RL) setting are relatively unexplored because of the limitations of binary activations for function approximation. Here, in the discrete action ATARI domain, we demonstrate, for the first time, that low precision policy distillation from a high precision network provides a principled, practical way to train an RL agent. As an application, on 10 different ATARI games, we demonstrate real-time end-to-end game playing on low-power neuromorphic hardware by converting a sequence of game frames into discrete actions.
Neuromorphic dreaming: A pathway to efficient learning in artificial agents
Institute of Neuroinformatics, University of Zurich and ETH Zurich（苏黎世大学与苏黎世联邦理工学院神经信息学研究所）；Technical University of Munich（慕尼黑工业大学）；Istituto Superiore di Sanità（意大利高等卫生研究院）
In this work we explore recurrent representations of leaky integrate and fire neurons operating at a timescale equal to their absolute refractory period. Our coarse time scale approximation is obtained using a probability distribution function for spike arrivals that is homogeneously distributed over this time interval. This leads to a discrete representation that exhibits the same dynamics as the continuous model, enabling efficient large scale simulations and backpropagation through the recurrent implementation. We use this approach to explore the training of deep spiking neural networks inc…
The energy-efficient control of mobile robots is crucial as the complexity of their real-world applications increasingly involves high-dimensional observation and action spaces, which cannot be offset by limited on-board resources. An emerging non-Von Neumann model of intelligence, where spiking neural networks (SNNs) are run on neuromorphic processors, is regarded as an energy-efficient and robust alternative to the state-of-the-art real-time robotic controllers for low dimensional control tasks. The challenge now for this new computing paradigm is to scale so that it can keep up with real-wo…
Neuromorphic Energy-Aware Learning for Adaptive Deep Brain Stimulation
Dept. of Electrical and Computer Engineering, University of California, Santa Cruz, CA, USA（加州大学圣克鲁兹分校电气与计算机工程系）；Dept. of Bioengineering, University of California, San Diego, La Jolla, CA, USA（加州大学圣地亚哥分校生物工程系）
Training slow silicon neurons to control extremely fast robots with spiking reinforcement learning
Institute of Neuroinformatics, UZH and ETH Zurich（神经信息研究所，苏黎世联邦理工学院和苏黎世联邦理工学院）；Istituto Italiano di Tecnologia（意大利技术研究所）；Technical University of Munich（慕尼黑技术大学）；Natl. Center for Radiation Protection and Computational Physics, Istituto Superiore di Sanità（辐射防护与计算物理国家中心，意大利卫生超级研究所）；Digital Society Initiative, University of Zurich（数字社会倡议，苏黎世大学）
With the development of industry, drones are appearing in various field. In recent years, deep reinforcement learning has made impressive gains in games, and we are committed to applying deep reinforcement learning algorithms to the field of robotics, moving reinforcement learning algorithms from game scenarios to real-world application scenarios. We are inspired by the LunarLander of OpenAI Gym, we decided to make a bold attempt in the field of reinforcement learning to control drones. At present, there is still a lack of work applying reinforcement learning algorithms to robot control, the p…
Reinforcement Learning (RL) is a research area that has blossomed tremendously in recent years and has shown remarkable potential in among others successfully playing computer games. However, there only exists a few game platforms that provide diversity in tasks and state-space needed to advance RL algorithms. The existing platforms offer RL access to Atari- and a few web-based games, but no platform fully expose access to Flash games. This is unfortunate because applying RL to Flash games have potential to push the research of RL algorithms. This paper introduces the Flash Reinforcement Learn…
Baidu Apollo Autonomous Driving（百度阿波罗自动驾驶）；University of Cambridge（剑桥大学）
We introduce ApolloRL, an open platform for research in reinforcement learning for autonomous driving. The platform provides a complete closed-loop pipeline with training, simulation, and evaluation components. It comes with 300 hours of real-world data in driving scenarios and popular baselines such as Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) agents. We elaborate in this paper on the architecture and the environment defined in the platform. In addition, we discuss the performance of the baseline agents in the ApolloRL environment.
New Jersey Institute of Technology（新泽西理工学院）；King’s College London（伦敦国王学院）
Artificial Neural Networks (ANNs) are currently being used as function approximators in many state-of-the-art Reinforcement Learning (RL) algorithms. Spiking Neural Networks (SNNs) have been shown to drastically reduce the energy consumption of ANNs by encoding information in sparse temporal binary spike streams, hence emulating the communication mechanism of biological neurons. Due to their low energy consumption, SNNs are considered to be important candidates as co-processors to be implemented in mobile devices. In this work, the use of SNNs as stochastic policies is explored under an energy…
Laser interferometry as a robust neuromorphic platform for machine learning
Department of Physics, University of Virginia（物理系，弗吉尼亚大学）；Department of Mathematics, The University of British Columbia（数学系，不列颠哥伦比亚大学）；Department of Physics, Syracuse University（物理系，苏利文大学）；Max Planck Institute for the Physics of Complex Systems（复杂系统物理研究所）；Charles L. Brown Department of Electrical and Computer Engineering, University of Virginia（查尔斯·L·布朗电子与计算机工程系，弗吉尼亚大学）
SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence
Peking University（北京大学）；Peng Cheng Laboratory（鹏城实验室）；Shanghai Jiao Tong University（上海交通大学）；Chinese Academy of Sciences（中国科学院）；Centre de Recherche Cerveau et Cognition (CERCO), UMR5549 CNRS - Univ. Toulouse 3（大脑与认知研究中心（CERCO），CNRS UMR5549 - 图卢兹第三大学）；University of Chinese Academy of Sciences（中国科学院大学）
A Realistic Simulation Framework for Analog/Digital Neuromorphic Architectures
Systems Lab, Zernike Institute for Advanced Materials, University of Groningen, Netherlands（格罗宁根大学系统实验室，泽尼克先进材料研究所）；Materials Center, University of Groningen, Netherlands（格罗宁根大学材料中心）；School of Engineering, University of Cádiz, Puerto Real, Cádiz, Spain（卡迪斯大学工程学院）；Institute of Neuroinformatics, University of Zurich（苏黎世大学神经信息研究所）；ETH Zurich, Zurich, Switzerland（苏黎世联邦理工学院）
