Recently, reinforcement learning models have achieved great success, completing complex tasks such as mastering Go and other games with higher scores than human players. Many of these models collect considerable data on the tasks and improve accuracy by extracting visual and time-series features using convolutional neural networks (CNNs) and recurrent neural networks, respectively. However, these networks have very high computational costs because they need to be trained by repeatedly using a large volume of past playing data. In this study, we propose a novel practical approach called reinforcement learning with convolutional reservoir computing (RCRC) model. The RCRC model has several desirable features: 1. it can extract visual and time-series features very fast because it uses random fixed-weight CNN and the reservoir computing model; 2. it does not require the training data to be stored because it extracts features without training and decides action with evolution strategy. Furthermore, the model achieves state of the art score in the popular reinforcement learning task. Incredibly, we find the random weight-fixed simple networks like only one dense layer network can also reach high score in the RL task.
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Yerevan State University（埃里温国立大学）
Reservoir Computing is a class of Recurrent Neural Networks with internal weights fixed at random. Stability relates to the sensitivity of the network state to perturbations. It is an important property in Reservoir Computing as it directly impacts performance. In practice, it is desirable to stay in a stable regime, where the effect of perturbations does not explode exponentially, but also close to the chaotic frontier where reservoir dynamics are rich. Open questions remain today regarding input regularization and discontinuous activation functions. In this work, we use the recurrent kernel …
Graduate school of Systems and Information Engineering（系统与信息工程研究生院）；University of Tsukuba（筑波大学）
Recently, reinforcement learning models have achieved great success, mastering complex tasks such as Go and other games with higher scores than human players. Many of these models store considerable data on the tasks and achieve high performance by extracting visual and time-series features using convolutional neural networks (CNNs) and recurrent neural networks, respectively. However, these networks have very high computational costs because they need to be trained by repeatedly using the stored data. In this study, we propose a novel practical approach called reinforcement learning with conv…
University of Vienna（维也纳大学）；Ludwig-Maximilians-Universität München（路德维希-马克西米利安慕尼黑大学）；Universität Konstanz（康斯坦茨大学）；Universität Sankt Gallen（圣加仑大学）；Centre National de la Recherche Scientifique (CNRS)（法国国家科学研究中心）；ETH Zürich（苏黎世联邦理工学院）
A new explanation of geometric nature of the reservoir computing phenomenon is presented. Reservoir computing is understood in the literature as the possibility of approximating input/output systems with randomly chosen recurrent neural systems and a trained linear readout layer. Light is shed on this phenomenon by constructing what is called strongly universal reservoir systems as random projections of a family of state-space systems that generate Volterra series expansions. This procedure yields a state-affine reservoir system with randomly generated coefficients in a dimension that is logar…
Universität Sankt Gallen（圣加仑大学）；Universität Konstanz（康斯坦茨大学）；Centre National de la Recherche Scientifique (CNRS)（法国国家科学研究中心）
We analyze the practices of reservoir computing in the framework of statistical learning theory. In particular, we derive finite sample upper bounds for the generalization error committed by specific families of reservoir computing systems when processing discrete-time inputs under various hypotheses on their dependence structure. Non-asymptotic bounds are explicitly written down in terms of the multivariate Rademacher complexities of the reservoir systems and the weak dependence structure of the signals that are being handled. This allows, in particular, to determine the minimal number of obs…
Faculty of Engineering（工学院）；University of Tsukuba（筑波大学）；Toyota Motor Corporation（丰田汽车公司）
Reservoir computing derived from recurrent neural networks is more applicable to real world systems than deep learning because of its low computational cost and potential for physical implementation. Specifically, physical reservoir computing, which replaces the dynamics of reservoir units with physical phenomena, has recently received considerable attention. In this study, we propose a method of exploiting the dynamics of road traffic as a reservoir, and numerically confirm its feasibility by applying several prediction tasks based on a simple mathematical model of the traffic flow.
Reservoir Computing is an emerging machine learning framework which is a versatile option for utilising physical systems for computation. In this paper, we demonstrate how a single node reservoir, made of a simple electronic circuit, can be employed for computation and explore the available options to improve the computational capability of the physical reservoirs. We build a reservoir computing system using a memristive chaotic oscillator as the reservoir. We choose two of the available hyperparameters to find the optimal working regime for the reservoir, resulting in two reservoir versions. …
Ecole Normale Supérieure（巴黎高等师范学院）；Université PSL（巴黎文理研究大学）；CNRS（法国国家科学研究中心）；Sorbonne Université（索邦大学）；Université de Paris（巴黎大学）；Collège de France（法兰西公学院）；LightOn（LightOn公司）；Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Reservoir Computing is a class of simple yet efficient Recurrent Neural Networks where internal weights are fixed at random and only a linear output layer is trained. In the large size limit, such random neural networks have a deep connection with kernel methods. Our contributions are threefold: a) We rigorously establish the recurrent kernel limit of Reservoir Computing and prove its convergence. b) We test our models on chaotic time series prediction, a classic but challenging benchmark in Reservoir Computing, and show how the Recurrent Kernel is competitive and computationally efficient whe…
University of New Mexico（新墨西哥大学）；Portland State University（波特兰州立大学）
Reservoir computing (RC) is a novel approach to time series prediction using recurrent neural networks. In RC, an input signal perturbs the intrinsic dynamics of a medium called a reservoir. A readout layer is then trained to reconstruct a target output from the reservoir's state. The multitude of RC architectures and evaluation metrics poses a challenge to both practitioners and theorists who study the task-solving performance and computational power of RC. In addition, in contrast to traditional computation models, the reservoir is a dynamical system in which computation and memory are insep…
Photonic reservoir computing is a promising candidate for low-energy computing at high bandwidths. Despite recent successes, there are bounds to what one can achieve simply by making photonic reservoirs larger. Therefore, a switch from single-reservoir computing to multi-reservoir and even deep physical reservoir computing is desirable. Given that backpropagation can not be used directly to train multi-reservoir systems in our targeted setting, we propose an alternative approach that still uses its power to derive intermediate targets. In this work we report our findings on a conducted experim…
Reservoir computing(RC) is a brain-inspired computing framework that employs a transient dynamical system whose reaction to an input signal is transformed to a target output. One of the central problems in RC is to find a reliable reservoir with a large criticality, since computing performance of a reservoir is maximized near the phase transition. In this work, we propose a continuous reservoir that utilizes transient dynamics of coupled chaotic oscillators in a critical regime where sudden amplitude death occurs. This "explosive death" not only brings the system a large criticality which prov…
Recently, studies on deep Reservoir Computing (RC) highlighted the role of layering in deep recurrent neural networks (RNNs). In this paper, the use of linear recurrent units allows us to bring more evidence on the intrinsic hierarchical temporal representation in deep RNNs through frequency analysis applied to the state signals. The potentiality of our approach is assessed on the class of Multiple Superimposed Oscillator tasks. Furthermore, our investigation provides useful insights to open a discussion on the main aspects that characterize the deep learning framework in the temporal domain.
In this work, we propose a new approach towards the efficient optimization and implementation of reservoir computing hardware reducing the required domain expert knowledge and optimization effort. First, we adapt the reservoir input mask to the structure of the data via linear autoencoders. We therefore incorporate the advantages of dimensionality reduction and dimensionality expansion achieved by conventional algorithmically efficient linear algebra procedures of principal component analysis. Second, we employ evolutionary-inspired genetic algorithm techniques resulting in a highly efficient …
Wyant College of Optical Sciences, University of Arizona（亚利桑那大学怀恩特光学科学学院）；School of Electrical Engineering and Telecommunications, University of New South Wales（新南威尔士大学电气工程与电信学院）
In this paper we give a profound insight into the computation capability of delay-based reservoir computing via an eigenvalue analysis. We concentrate on the task-independent memory capacity to quantify the reservoir performance and compare these with the eigenvalue spectrum of the dynamical system. We show that these two quantities are deeply connected, and thus the reservoir computing performance is predictable by analyzing the small signal response of the reservoir. Our results suggest that any dynamical system used as a reservoir can be analyzed in this way. We apply our method exemplarily…
Reservoir Computing (RC) refers to a Recurrent Neural Networks (RNNs) framework, frequently used for sequence learning and time series prediction. The RC system consists of a random fixed-weight RNN (the input-hidden reservoir layer) and a classifier (the hidden-output readout layer). Here we focus on the sequence learning problem, and we explore a different approach to RC. More specifically, we remove the non-linear neural activation function, and we consider an orthogonal reservoir acting on normalized states on the unit hypersphere. Surprisingly, our numerical results show that the system's…
Reservoir computing is a recently introduced machine learning paradigm that has been shown to be well-suited for the processing of spatiotemporal data. Rather than training the network node connections and weights via backpropagation in traditional recurrent neural networks, reservoirs instead have fixed connections and weights among the `hidden layer' nodes, and traditionally only the weights to the output layer of neurons are trained using linear regression. We claim that for signal classification tasks one may forgo the weight training step entirely and instead use a simple supervised clust…
Institute of Fundamental and Frontier Sciences, University of Electronic Science and Technology of China（电子科技大学基础与前沿科学研究所）；Potsdam Institute for Climate Impact Research（波茨坦气候影响研究所）；Department of Physics, Humboldt University Berlin（柏林洪堡大学物理系）；Research Institute of Intelligent Complex Systems, Fudan University（复旦大学智能复杂系统研究所）；School of Physics, University of Electronic Science and Technology of China（电子科技大学物理学院）；Key Laboratory of Quantum Physics and Photonic Quantum Information, Ministry of Education, University of Electronic Science and Technology of China（电子科技大学教育部量子物理与光子量子信息重点实验室）；Non-classical Information Science Basic Discipline Research Center of Sichuan Province, University of Electronic Science and Technology of China（电子科技大学四川省非经典信息科学基础学科研究中心）
Insect-Wing Structured Microfluidic System for Reservoir Computing
School of Computing, University of Nebraska-Lincoln（计算机学院，内布拉斯加大学林肯分校）；Department of Mechanical and Materials Engineering, University of Nebraska-Lincoln（机械与材料工程学院，内布拉斯加大学林肯分校）
Reservoir computing is a recently introduced, highly efficient bio-inspired approach for processing time dependent data. The basic scheme of reservoir computing consists of a non linear recurrent dynamical system coupled to a single input layer and a single output layer. Within these constraints many implementations are possible. Here we report an opto-electronic implementation of reservoir computing based on a recently proposed architecture consisting of a single non linear node and a delay line. Our implementation is sufficiently fast for real time information processing. We illustrate its p…
The Ohio State University（俄亥俄州立大学）；ResCon Technologies, LLC（ResCon Technologies公司）；Clarkson University（克拉克森大学）
Reservoir computing is a best-in-class machine learning algorithm for processing information generated by dynamical systems using observed time-series data. Importantly, it requires very small training data sets, uses linear optimization, and thus requires minimal computing resources. However, the algorithm uses randomly sampled matrices to define the underlying recurrent neural network and has a multitude of metaparameters that must be optimized. Recent results demonstrate the equivalence of reservoir computing to nonlinear vector autoregression, which requires no random matrices, fewer metap…
Technical University of Denmark（丹麦技术大学）；Nokia Bell Labs（诺基亚贝尔实验室）
A receiver with shared complexity between optical and digital domains is experimentally demonstrated. Reservoir computing is used to equalize up to 4 directly-detected optically filtered spectral slices of a 32 GBd OOK signal over up to 80 km of SMF.
DTU Electro, Technical University of Denmark（丹麦技术大学DTU Electro）
We propose spectrum-sliced reservoir computer-based (RC) multi-symbol equalization for 32-GBd PAM4 transmission. RC with 17 symbols at the output achieves an order of magnitude reduction in multiplications/symbol versus single output case while maintaining simple training.
Reservoir computers (RC) have proven useful as surrogate models in forecasting and replicating systems of chaotic dynamics. The quality of surrogate models based on RCs is crucially dependent on their optimal implementation that involves selecting optimal reservoir topology and hyperparameters. By systematically applying Bayesian hyperparameter optimization and using ensembles of reservoirs of various topology we show that connectednes of reservoirs is of significance only in forecasting and replication of chaotic system of sufficient complexity. By applying RCs of different topology in foreca…
Jacobs University Bremen（不来梅雅各布大学）；University Medical Center Hamburg-Eppendorf（汉堡-埃彭多夫大学医学中心）
This chapter provides a comprehensive survey of the researches and motivations for hardware implementation of reservoir computing (RC) on neuromorphic electronic systems. Due to its computational efficiency and the fact that training amounts to a simple linear regression, both spiking and non-spiking implementations of reservoir computing on neuromorphic hardware have been developed. Here, a review of these experimental studies is provided to illustrate the progress in this area and to address the technical challenges which arise from this specific hardware implementation. Moreover, to deal wi…
Reservoir computing (RC) offers efficient temporal data processing with a low training cost by separating recurrent neural networks into a fixed network with recurrent connections and a trainable linear network. The quality of the fixed network, called reservoir, is the most important factor that determines the performance of the RC system. In this paper, we investigate the influence of the hierarchical reservoir structure on the properties of the reservoir and the performance of the RC system. Analogous to deep neural networks, stacking sub-reservoirs in series is an efficient way to enhance …
Saitama University（埼玉大学）；University of Tokyo（东京大学）
Photonic reservoir computing is an emergent technology toward beyond-Neumann computing. Although photonic reservoir computing provides superior performance in environments whose characteristics are coincident with the training datasets for the reservoir, the performance is significantly degraded if these characteristics deviate from the original knowledge used in the training phase. Here, we propose a scheme of adaptive model selection in photonic reservoir computing using reinforcement learning. In this scheme, a temporal waveform is generated by different dynamic source models that change ov…
School of Electrical, Computer, and Energy Engineering, Arizona State University（亚利桑那州立大学电气、计算机与能源工程学院）；Department of Electrical and Computer Engineering, Colorado State University（科罗拉多州立大学电气与计算机工程系）；DEVCOM Army Research Office（DEVCOM陆军研究办公室）；Department of Physics, Arizona State University（亚利桑那州立大学物理系）
Harnessing Synthetic Active Particles for Physical Reservoir Computing
Peter Debye Institute for Soft Matter Physics, Leipzig University（莱比锡大学彼得·德拜软物质物理研究所）；Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Dresden/Leipzig（德累斯顿/莱比锡可扩展数据分析与人工智能中心（ScaDS.AI））
Technische Universität Ilmenau（伊尔默瑙工业大学）；New York University（纽约大学）
Reservoir computing is applied to model the large-scale evolution and the resulting low-order turbulence statistics of a two-dimensional turbulent Rayleigh-Bénard convection flow at a Rayleigh number ${\rm Ra}=10^7$ and a Prandtl number ${\rm Pr}=7$ in an extended domain with an aspect ratio of 6. Our data-driven approach which is based on a long-term direct numerical simulation of the convection flow comprises a two-step procedure. (1) Reduction of the original simulation data by a Proper Orthogonal Decomposition (POD) snapshot analysis and subsequent truncation to the first 150 POD modes whi…
ChemReservoir -- An Open-Source Framework for Chemically-Inspired Reservoir Computing
Department of Mathematics and Computer Science（数学与计算机科学系）；University of Southern Denmark（南部丹麦大学）；Algorithmic Cheminformatics（算法化学信息学）；Faculty of Technology, Bielefeld University（技术学院，比勒菲尔德大学）
Institute of Information；Communication Technologies Bulgarian Academy of Sciences Sofia, Bulgaria ORCID 0000-0002-0447-9667；Communication Technologies Bulgarian Academy of Sciences, Sofia, Bulgaria
The paper proposes a novel approach for gray scale images segmentation. It is based on multiple features extraction from single feature per image pixel, namely its intensity value, using Echo state network. The newly extracted features - reservoir equilibrium states - reveal hidden image characteristics that improve its segmentation via a clustering algorithm. Moreover, it was demonstrated that the intrinsic plasticity tuning of reservoir fits its equilibrium states to the original image intensity distribution thus allowing for its better segmentation. The proposed approach is tested on the be…
Understanding the fundamental relationships between physics and its information-processing capability has been an active research topic for many years. Physical reservoir computing is a recently introduced framework that allows one to exploit the complex dynamics of physical systems as information-processing devices. This framework is particularly suited for edge computing devices, in which information processing is incorporated at the edge (e.g., into sensors) in a decentralized manner to reduce the adaptation delay caused by data transmission overhead. This paper aims to illustrate the poten…
National Engineering School of Sousse, University of Sousse, LATIS – Laboratory of Advanced Technology and Intelligent Systems（苏塞国家工程学校，苏塞大学，先进技术和智能系统实验室）；BASIRA Lab, Imperial-X and Department of Computing, Imperial College London（BASIRA实验室，Imperial-X和伦敦帝国理工学院计算系）
