Recurrent neural network (RNN) based reinforcement learning (RL) is used for learning context-dependent tasks and has also attracted attention as a method with remarkable learning performance in recent research. However, RNN-based RL has some issues that the learning procedures tend to be more computationally expensive, and training with backpropagation through time (BPTT) is unstable because of vanishing/exploding gradients problem. An approach with replay memory introducing reservoir computing has been proposed, which trains an agent without BPTT and avoids these issues. The basic idea of this approach is that observations from the environment are input to the reservoir network, and both the observation and the reservoir output are stored in the memory. This paper shows that the performance of this method improves by using a multi-layered neural network for the readout layer, which regularly consists of a single linear layer. The experimental results show that using multi-layered readout improves the learning performance of four classical control tasks that require time-series processing.
Université Libre de Bruxelles（布鲁塞尔自由大学）；Ghent University（根特大学）
Reservoir computing is a new, powerful and flexible machine learning technique that is easily implemented in hardware. Recently, by using a time-multiplexed architecture, hardware reservoir computers have reached performance comparable to digital implementations. Operating speeds allowing for real time information operation have been reached using optoelectronic systems. At present the main performance bottleneck is the readout layer which uses slow, digital postprocessing. We have designed an analog readout suitable for time-multiplexed optoelectronic reservoir computers, capable of working i…
As Moore's law comes to an end, neuromorphic approaches to computing are on the rise. One of these, passive photonic reservoir computing, is a strong candidate for computing at high bitrates (> 10 Gbps) and with low energy consumption. Currently though, both benefits are limited by the necessity to perform training and readout operations in the electrical domain. Thus, efforts are currently underway in the photonic community to design an integrated optical readout, which allows to perform all operations in the optical domain. In addition to the technological challenge of designing such a reado…
University of Vienna（维也纳大学）；Ludwig-Maximilians-Universität München（路德维希-马克西米利安慕尼黑大学）；Universität Konstanz（康斯坦茨大学）；Universität Sankt Gallen（圣加仑大学）；Centre National de la Recherche Scientifique (CNRS)（法国国家科学研究中心）；ETH Zürich（苏黎世联邦理工学院）
A new explanation of geometric nature of the reservoir computing phenomenon is presented. Reservoir computing is understood in the literature as the possibility of approximating input/output systems with randomly chosen recurrent neural systems and a trained linear readout layer. Light is shed on this phenomenon by constructing what is called strongly universal reservoir systems as random projections of a family of state-space systems that generate Volterra series expansions. This procedure yields a state-affine reservoir system with randomly generated coefficients in a dimension that is logar…
Università Degli Studi Di Pavia（帕维亚大学）；STMicroelectronics（意法半导体）
Echo State Networks (ESN) are a class of Recurrent Neural Networks (RNN) that has gained substantial popularity due to their effectiveness, ease of use and potential for compact hardware implementation. An ESN contains the three network layers input, reservoir and readout where the reservoir is the truly recurrent network. The input and reservoir layers of an ESN are initialized at random and never trained afterwards and the training of the ESN is applied to the readout layer only. The alternative of Recursive Neural Gas (RNG) is one of the many proposals of fully-trainable reservoirs that can…
Using optical hardware for neuromorphic computing has become more and more popular recently due to its efficient high-speed data processing capabilities and low power consumption. However, there are still some remaining obstacles to realizing the vision of a completely optical neuromorphic computer. One of them is that, depending on the technology used, optical weighting elements may not share the same resolution as in the electrical domain. Moreover, noise and drift are important considerations as well. In this article, we investigate a new method for improving the performance of optical weig…
The mushroom body is the key network for the representation of learned olfactory stimuli in Drosophila and insects. The sparse activity of Kenyon cells, the principal neurons in the mushroom body, plays a key role in the learned classification of different odours. In the specific case of the fruit fly, the sparseness of the network is enforced by an inhibitory feedback neuron called APL, and by an intrinsic high firing threshold of the Kenyon cells. In this work we took inspiration from the fruit fly brain to formulate a novel machine learning algorithm that is able to optimize the sparsity le…
University of New Mexico（新墨西哥大学）；Portland State University（波特兰州立大学）
Reservoir computing (RC) is a novel approach to time series prediction using recurrent neural networks. In RC, an input signal perturbs the intrinsic dynamics of a medium called a reservoir. A readout layer is then trained to reconstruct a target output from the reservoir's state. The multitude of RC architectures and evaluation metrics poses a challenge to both practitioners and theorists who study the task-solving performance and computational power of RC. In addition, in contrast to traditional computation models, the reservoir is a dynamical system in which computation and memory are insep…
Reservoir Computing (RC) refers to a Recurrent Neural Networks (RNNs) framework, frequently used for sequence learning and time series prediction. The RC system consists of a random fixed-weight RNN (the input-hidden reservoir layer) and a classifier (the hidden-output readout layer). Here we focus on the sequence learning problem, and we explore a different approach to RC. More specifically, we remove the non-linear neural activation function, and we consider an orthogonal reservoir acting on normalized states on the unit hypersphere. Surprisingly, our numerical results show that the system's…
University of Maryland（马里兰大学）；Department of Computer Science（计算机科学系）；Department of Physics（物理系）；Institute for Adv. Computer Studies（高级计算机研究所）
In many real-world applications, fully-differentiable RNNs such as LSTMs and GRUs have been widely deployed to solve time series learning tasks. These networks train via Backpropagation Through Time, which can work well in practice but involves a biologically unrealistic unrolling of the network in time for gradient updates, are computationally expensive, and can be hard to tune. A second paradigm, Reservoir Computing, keeps the recurrent weight matrix fixed and random. Here, we propose a novel hybrid network, which we call Hybrid Backpropagation Parallel Echo State Network (HBP-ESN) which com…
We analyze the memory capacity of a delay based reservoir computer with a Hopf normal form as nonlinearity and numerically compute the linear as well as the higher order recall capabilities. A possible physical realisation could be a laser with external cavity, for which the information is fed via electrical injection. A task independent quantification of the computational capability of the reservoir system is done via a complete orthonormal set of basis functions. Our results suggest that even for constant readout dimension the total memory capacity is dependent on the ratio between the infor…
Insect-Wing Structured Microfluidic System for Reservoir Computing
School of Computing, University of Nebraska-Lincoln（计算机学院，内布拉斯加大学林肯分校）；Department of Mechanical and Materials Engineering, University of Nebraska-Lincoln（机械与材料工程学院，内布拉斯加大学林肯分校）
Quantum Reservoir Computing: Recent Advances and Future Directions
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg（安全、可靠性与信任跨学科中心（SnT），卢森堡大学）；School of Computing, Engineering and Digital Technologies, Teesside University（计算、工程与数字技术学院，泰赛大学）
Centre National de la Recherche Scientifique (CNRS)（法国国家科学研究中心）
This paper shows that echo state networks are universal uniform approximants in the context of discrete-time fading memory filters with uniformly bounded inputs defined on negative infinite times. This result guarantees that any fading memory input/output system in discrete time can be realized as a simple finite-dimensional neural network-type state-space model with a static linear readout map. This approximation is valid for infinite time intervals. The proof of this statement is based on fundamental results, also presented in this work, about the topological nature of the fading memory prop…
Wyant College of Optical Sciences, University of Arizona（亚利桑那大学怀恩特光学科学学院）；School of Electrical Engineering and Telecommunications, University of New South Wales（新南威尔士大学电气工程与电信学院）
University Medical Center Hamburg-Eppendorf (UKE)（汉堡-埃彭多夫大学医学中心（UKE））；Jacobs University gGmbH（雅各布大学gGmbH）
The reservoir computing paradigm is employed to classify heartbeat anomalies online based on electrocardiogram signals. Inspired by the principles of information processing in the brain, reservoir computing provides a framework to design, train, and analyze recurrent neural networks (RNNs) for processing time-dependent information. Due to its computational efficiency and the fact that training amounts to a simple linear regression, this supervised learning algorithm has been variously considered as a strategy to implement useful computations not only on digital computers but also on emerging u…
Data-Specific Hyper-Parameter Design: A Paradigm Shift in Reservoir Computing
University of Pretoria. Department of Mathematics and Applied Mathematics（南非大学 Pretoria分校 数学与应用数学系）；Nanyang Technological University. Division of Mathematical Sciences. School of Physical and Mathematical Sciences（南洋理工大学 数学科学学院 物理与数学科学学院）
Airbus Central Research & Technology（空客中央研究与技术中心）；Department of Computer Science（计算机科学系）；University of Pisa, Italy（意大利比萨大学）；Bielefeld University, Germany（德国比勒菲尔德大学）
Interfacial and bulk switching MoS2 memristors for an all-2D reservoir computing framework
Department of Electrical Engineering, Indian Institute of Technology Madras, Chennai, India（电子工程系，印度理工学院马德拉斯学院，钦奈，印度）；Department of Physics, Indian Institute of Technology Madras, Chennai, India（物理系，印度理工学院马德拉斯学院，钦奈，印度）；Center for Nano Science and Engineering, Indian Institute of Science, Bangalore, India（纳米科学与工程中心，印度科学研究院，班加罗尔，印度）；The Pennsylvania State University, University Park, PA, USA（宾夕法尼亚州立大学，大学公园，宾夕法尼亚州，美国）
Multivariate Time Series Forecasting with Gate-Based Quantum Reservoir Computing on NISQ Hardware
Polytechnique Montréal（波士顿理工学院）；Institut quantique, Université de Sherbrooke（量子研究所， Sherbrooke 大学）；Thales Digital Solutions（泰雷兹数字解决方案）；Université de Sherbrooke（Sherbrooke 大学）
University Sapienza（萨皮恩扎大学）；National Research Council (ISC-CNR)（国家研究委员会（复杂系统研究所））；University of Twente（特文特大学）；Heriot-Watt University（赫瑞-瓦特大学）
Novel machine learning computational tools open new perspectives for quantum information systems. Here we adopt the open-source programming library TensorFlow to design multi-level quantum gates including a computing reservoir represented by a random unitary matrix. In optics, the reservoir is a disordered medium or a multi-modal fiber. We show that trainable operators at the input and the readout enable one to realize multi-level gates. We study various qudit gates, including the scaling properties of the algorithms with the size of the reservoir. Despite an initial low slop learning stage, T…
