Input-to-State Representation in linear reservoirs dynamics
Università della Svizzera Italiana（瑞士意大利语大学）；Politecnico di Milano（米兰理工大学）；University of Manitoba（曼尼托巴大学）；University of Exeter（埃克塞特大学）；University of Birmingham（伯明翰大学）
Pietro Verzelli, Cesare Alippi, Lorenzo Livi, Peter Tino
Reservoir computing is a popular approach to design recurrent neural networks, due to its training simplicity and approximation performance. The recurrent part of these networks is not trained (e.g., via gradient descent), making them appealing for analytical studies by a large community of researchers with backgrounds spanning from dynamical systems to neuroscience. However, even in the simple linear case, the working principle of these networks is not fully understood and their design is usually driven by heuristics. A novel analysis of the dynamics of such networks is proposed, which allows the investigator to express the state evolution using the controllability matrix. Such a matrix encodes salient characteristics of the network dynamics; in particular, its rank represents an input-indepedent measure of the memory capacity of the network. Using the proposed approach, it is possible to compare different reservoir architectures and explain why a cyclic topology achieves favourable results as verified by practitioners.
Data-Specific Hyper-Parameter Design: A Paradigm Shift in Reservoir Computing
University of Pretoria. Department of Mathematics and Applied Mathematics（南非大学 Pretoria分校 数学与应用数学系）；Nanyang Technological University. Division of Mathematical Sciences. School of Physical and Mathematical Sciences（南洋理工大学 数学科学学院 物理与数学科学学院）
Chiba Institute of Technology（千叶工业大学）；The University of Tokyo（东京大学）；National Institute of Mental Health, National Center of Neurology and Psychiatry（国立神经精神研究中心国立精神卫生研究所）；Oita University（大分大学）；Kyushu Institute of Technology（九州工业大学）
Reservoir computing (RC) can efficiently process time-series data by transferring the input signal to randomly connected recurrent neural networks (RNNs), which are referred to as a reservoir. The high-dimensional representation of time-series data in the reservoir significantly simplifies subsequent learning tasks. Although this simple architecture allows fast learning and facile physical implementation, the learning performance is inferior to that of other state-of-the-art RNN models. In this paper, to improve the learning ability of RC, we propose self-modulated RC (SM-RC), which extends RC…
Università di Pisa（比萨大学）；Biobeats Ltd（Biobeats 有限公司）
The paper presents a novel, principled approach to train recurrent neural networks from the Reservoir Computing family that are robust to missing part of the input features at prediction time. By building on the ensembling properties of Dropout regularization, we propose a methodology, named DropIn, which efficiently trains a neural model as a committee machine of subnetworks, each capable of predicting with a subset of the original input features. We discuss the application of the DropIn methodology in the context of Reservoir Computing models and targeting applications characterized by input…
Reservoir Computing (RC) is a well-known strategy for designing Recurrent Neural Networks featured by striking efficiency of training. The crucial aspect of RC is to properly instantiate the hidden recurrent layer that serves as dynamical memory to the system. In this respect, the common recipe is to create a pool of randomly and sparsely connected recurrent neurons. While the aspect of sparsity in the design of RC systems has been debated in the literature, it is nowadays understood mainly as a way to enhance the efficiency of computation, exploiting sparse matrix operations. In this paper, w…
University of New Mexico（新墨西哥大学）；Portland State University（波特兰州立大学）
Reservoir computing (RC) is a novel approach to time series prediction using recurrent neural networks. In RC, an input signal perturbs the intrinsic dynamics of a medium called a reservoir. A readout layer is then trained to reconstruct a target output from the reservoir's state. The multitude of RC architectures and evaluation metrics poses a challenge to both practitioners and theorists who study the task-solving performance and computational power of RC. In addition, in contrast to traditional computation models, the reservoir is a dynamical system in which computation and memory are insep…
Techniques for Enhancing Memory Capacity of Reservoir Computing
Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology（九州工业大学生命科学与系统工程学院）；Research Center for Neuromorphic AI Hardware, Kyushu Institute of Technology（九州工业大学神经形态人工智能硬件研究中心）；Faculty of Informatics, The University of Fukuchiyama（福知山公立大学信息学部）
University of Sheffield（谢菲尔德大学）；The University of Sheffield（谢菲尔德大学）
"Sparse" neural networks, in which relatively few neurons or connections are active, are common in both machine learning and neuroscience. Whereas in machine learning, "sparsity" is related to a penalty term that leads to some connecting weights becoming small or zero, in biological brains, sparsity is often created when high spiking thresholds prevent neuronal activity. Here we introduce sparsity into a reservoir computing network via neuron-specific learnable thresholds of activity, allowing neurons with low thresholds to contribute to decision-making but suppressing information from neurons…
Recently, studies on deep Reservoir Computing (RC) highlighted the role of layering in deep recurrent neural networks (RNNs). In this paper, the use of linear recurrent units allows us to bring more evidence on the intrinsic hierarchical temporal representation in deep RNNs through frequency analysis applied to the state signals. The potentiality of our approach is assessed on the class of Multiple Superimposed Oscillator tasks. Furthermore, our investigation provides useful insights to open a discussion on the main aspects that characterize the deep learning framework in the temporal domain.
University of California San Diego（加州大学圣迭戈分校）；Cooperative Institute for Research in Environmental Sciences at the University of Colorado Boulder（科罗拉多大学博尔德分校环境科学合作研究所）；NOAA Physical Sciences Laboratory（美国国家海洋和大气管理局物理科学实验室）；Scripps Institution of Oceanography, University of California San Diego（加州大学圣迭戈分校斯克里普斯海洋研究所）
Reservoir computers (RC) are a form of recurrent neural network (RNN) used for forecasting time series data. As with all RNNs, selecting the hyperparameters presents a challenge when training on new inputs. We present a method based on generalized synchronization (GS) that gives direction in designing and evaluating the architecture and hyperparameters of a RC. The 'auxiliary method' for detecting GS provides a pre-training test that guides hyperparameter selection. Furthermore, we provide a metric for a "well trained" RC using the reproduction of the input system's Lyapunov exponents.
University of Maryland（马里兰大学）；Institute for Adv. Computer Studies（高级计算机研究所）
Deep Reservoir Computing has emerged as a new paradigm for deep learning, which is based around the reservoir computing principle of maintaining random pools of neurons combined with hierarchical deep learning. The reservoir paradigm reflects and respects the high degree of recurrence in biological brains, and the role that neuronal dynamics play in learning. However, one issue hampering deep reservoir network development is that one cannot backpropagate through the reservoir layers. Recent deep reservoir architectures do not learn hidden or hierarchical representations in the same manner as d…
Current radio frequency (RF) sensors at the Edge lack the computational resources to support practical, in-situ training for intelligent spectrum monitoring, and sensor data classification in general. We propose a solution via Deep Delay Loop Reservoir Computing (DLR), a processing architecture that supports general machine learning algorithms on compact mobile devices by leveraging delay-loop reservoir computing in combination with innovative electrooptical hardware. With both digital and photonic realizations of our design of the loops, DLR delivers reductions in form factor, hardware comple…
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Yerevan State University（埃里温国立大学）
Reservoir Computing is a class of Recurrent Neural Networks with internal weights fixed at random. Stability relates to the sensitivity of the network state to perturbations. It is an important property in Reservoir Computing as it directly impacts performance. In practice, it is desirable to stay in a stable regime, where the effect of perturbations does not explode exponentially, but also close to the chaotic frontier where reservoir dynamics are rich. Open questions remain today regarding input regularization and discontinuous activation functions. In this work, we use the recurrent kernel …
Reservoir Computation with Networks of Differentiating Neuron Ring Oscillators
College of Computer \& Information Sciences, University of Massachusetts Amherst；Department of Mathematics, Virginia Tech；Allen Discovery Center, Tufts University
Università Degli Studi Di Pavia（帕维亚大学）；STMicroelectronics（意法半导体）
Echo State Networks (ESN) are a class of Recurrent Neural Networks (RNN) that has gained substantial popularity due to their effectiveness, ease of use and potential for compact hardware implementation. An ESN contains the three network layers input, reservoir and readout where the reservoir is the truly recurrent network. The input and reservoir layers of an ESN are initialized at random and never trained afterwards and the training of the ESN is applied to the readout layer only. The alternative of Recursive Neural Gas (RNG) is one of the many proposals of fully-trainable reservoirs that can…
Reservoir Computing (RC) provides an efficient way for designing dynamical recurrent neural models. While training is restricted to a simple output component, the recurrent connections are left untrained after initialization, subject to stability constraints specified by the Echo State Property (ESP). Literature conditions for the ESP typically fail to properly account for the effects of driving input signals, often limiting the potentialities of the RC approach. In this paper, we study the fundamental aspect of asymptotic stability of RC models in presence of driving input, introducing an emp…
Echo-state networks are simple models of discrete dynamical systems driven by a time series. By selecting network parameters such that the dynamics of the network is contractive, characterized by a negative maximal Lyapunov exponent, the network may synchronize with the driving signal. Exploiting this synchronization, the echo-state network may be trained to autonomously reproduce the input dynamics, enabling time-series prediction. However, while synchronization is a necessary condition for prediction, it is not sufficient. Here, we study what other conditions are necessary for successful tim…
Reservoir computing(RC) is a brain-inspired computing framework that employs a transient dynamical system whose reaction to an input signal is transformed to a target output. One of the central problems in RC is to find a reliable reservoir with a large criticality, since computing performance of a reservoir is maximized near the phase transition. In this work, we propose a continuous reservoir that utilizes transient dynamics of coupled chaotic oscillators in a critical regime where sudden amplitude death occurs. This "explosive death" not only brings the system a large criticality which prov…
In this work, we propose a new approach towards the efficient optimization and implementation of reservoir computing hardware reducing the required domain expert knowledge and optimization effort. First, we adapt the reservoir input mask to the structure of the data via linear autoencoders. We therefore incorporate the advantages of dimensionality reduction and dimensionality expansion achieved by conventional algorithmically efficient linear algebra procedures of principal component analysis. Second, we employ evolutionary-inspired genetic algorithm techniques resulting in a highly efficient …
University of Vienna（维也纳大学）；Ludwig-Maximilians-Universität München（路德维希-马克西米利安慕尼黑大学）；Universität Konstanz（康斯坦茨大学）；Universität Sankt Gallen（圣加仑大学）；Centre National de la Recherche Scientifique (CNRS)（法国国家科学研究中心）；ETH Zürich（苏黎世联邦理工学院）
A new explanation of geometric nature of the reservoir computing phenomenon is presented. Reservoir computing is understood in the literature as the possibility of approximating input/output systems with randomly chosen recurrent neural systems and a trained linear readout layer. Light is shed on this phenomenon by constructing what is called strongly universal reservoir systems as random projections of a family of state-space systems that generate Volterra series expansions. This procedure yields a state-affine reservoir system with randomly generated coefficients in a dimension that is logar…
Reservoir computing is a recently introduced, highly efficient bio-inspired approach for processing time dependent data. The basic scheme of reservoir computing consists of a non linear recurrent dynamical system coupled to a single input layer and a single output layer. Within these constraints many implementations are possible. Here we report an opto-electronic implementation of reservoir computing based on a recently proposed architecture consisting of a single non linear node and a delay line. Our implementation is sufficiently fast for real time information processing. We illustrate its p…
Institute of Fundamental and Frontier Sciences, University of Electronic Science and Technology of China（电子科技大学基础与前沿科学研究所）；Potsdam Institute for Climate Impact Research（波茨坦气候影响研究所）；Department of Physics, Humboldt University Berlin（柏林洪堡大学物理系）；Research Institute of Intelligent Complex Systems, Fudan University（复旦大学智能复杂系统研究所）；School of Physics, University of Electronic Science and Technology of China（电子科技大学物理学院）；Key Laboratory of Quantum Physics and Photonic Quantum Information, Ministry of Education, University of Electronic Science and Technology of China（电子科技大学教育部量子物理与光子量子信息重点实验室）；Non-classical Information Science Basic Discipline Research Center of Sichuan Province, University of Electronic Science and Technology of China（电子科技大学四川省非经典信息科学基础学科研究中心）
Universität Sankt Gallen（圣加仑大学）；Universität Konstanz（康斯坦茨大学）；Centre National de la Recherche Scientifique (CNRS)（法国国家科学研究中心）
We analyze the practices of reservoir computing in the framework of statistical learning theory. In particular, we derive finite sample upper bounds for the generalization error committed by specific families of reservoir computing systems when processing discrete-time inputs under various hypotheses on their dependence structure. Non-asymptotic bounds are explicitly written down in terms of the multivariate Rademacher complexities of the reservoir systems and the weak dependence structure of the signals that are being handled. This allows, in particular, to determine the minimal number of obs…
Efficient Time Series Clustering from Multiscale Reservoir Dynamics with Granular-Ball Anchoring Graph Optimization
Chongqing Key Laboratory of Computational Intelligence, Key Laboratory of Cyberspace Big Data Intelligent Security, Ministry of Education, Sichuan-Chongqing Co-construction Key Laboratory of Digital Economy Intelligence and Key Laboratory of Big Data Intelligent Computing, College of Computer Science and Technology, Chongqing University of Posts and Telecommunications（重庆邮电大学计算机科学与技术学院，计算智能重庆市重点实验室，网络空间大数据智能安全教育部重点实验室，川渝共建数字经济智能重点实验室，大数据智能计算重点实验室）；Chongqing Ant Consumer Finance Co,. Ltd , Ant Group（蚂蚁集团，重庆蚂蚁消费金融有限公司）
Reservoir Computing (RC) refers to a Recurrent Neural Networks (RNNs) framework, frequently used for sequence learning and time series prediction. The RC system consists of a random fixed-weight RNN (the input-hidden reservoir layer) and a classifier (the hidden-output readout layer). Here we focus on the sequence learning problem, and we explore a different approach to RC. More specifically, we remove the non-linear neural activation function, and we consider an orthogonal reservoir acting on normalized states on the unit hypersphere. Surprisingly, our numerical results show that the system's…
Recurrent Quantum Feature Maps for Reservoir Computing
Department of Physics, University of Ottawa（渥太华大学物理系）；National Research Council of Canada（加拿大国家研究委员会）；Institute for Quantum Science and Technology, Department of Physics and Astronomy, University of Calgary（卡尔加里大学物理与天文学系量子科学与技术研究所）；Hotchkiss Brain Institute, University of Calgary（卡尔加里大学霍奇基斯脑研究所）
Graduate School of Information Science and Technology, The University of Tokyo（东京大学信息科学与技术研究生院）；Next Generation Artificial Intelligence Research Center (AI Center), The University of Tokyo（东京大学下一代人工智能研究中心）；National Institute of Advanced Industrial Science and Technology（日本国立先进工业科学技术研究院）
South China University of Technology（华南理工大学）；University of California, San Diego（加州大学圣迭戈分校）
As an efficient recurrent neural network (RNN) model, reservoir computing (RC) models, such as Echo State Networks, have attracted widespread attention in the last decade. However, while they have had great success with time series data [1], [2], many time series have a multiscale structure, which a single-hidden-layer RC model may have difficulty capturing. In this paper, we propose a novel hierarchical reservoir computing framework we call Deep Echo State Networks (Deep-ESNs). The most distinctive feature of a Deep-ESN is its ability to deal with time series through hierarchical projections.…
We analyze the memory capacity of a delay based reservoir computer with a Hopf normal form as nonlinearity and numerically compute the linear as well as the higher order recall capabilities. A possible physical realisation could be a laser with external cavity, for which the information is fed via electrical injection. A task independent quantification of the computational capability of the reservoir system is done via a complete orthonormal set of basis functions. Our results suggest that even for constant readout dimension the total memory capacity is dependent on the ratio between the infor…
Usually, reservoir computing shows an exponential memory decay. This paper investigates under which circumstances echo state networks can show a power law forgetting. That means traces of earlier events can be found in the reservoir for very long time spans. Such a setting requires critical connectivity exactly at the limit of what is permissible according the echo state condition. However, for general matrices the limit cannot be determined exactly from theory. In addition, the behavior of the network is strongly influenced by the input flow. Results are presented that use certain types of re…
A new class of non-homogeneous state-affine systems is introduced for use in reservoir computing. Sufficient conditions are identified that guarantee first, that the associated reservoir computers with linear readouts are causal, time-invariant, and satisfy the fading memory property and second, that a subset of this class is universal in the category of fading memory filters with stochastic almost surely uniformly bounded inputs. This means that any discrete-time filter that satisfies the fading memory property with random inputs of that type can be uniformly approximated by elements in the n…
The universal approximation properties with respect to $L ^p $-type criteria of three important families of reservoir computers with stochastic discrete-time semi-infinite inputs is shown. First, it is proved that linear reservoir systems with either polynomial or neural network readout maps are universal. More importantly, it is proved that the same property holds for two families with linear readouts, namely, trigonometric state-affine systems and echo state networks, which are the most widely used reservoir systems in applications. The linearity in the readouts is a key feature in supervise…
We introduce a novel framework of reservoir computing. Cellular automaton is used as the reservoir of dynamical systems. Input is randomly projected onto the initial conditions of automaton cells and nonlinear computation is performed on the input via application of a rule in the automaton for a period of time. The evolution of the automaton creates a space-time volume of the automaton state space, and it is used as the reservoir. The proposed framework is capable of long short-term memory and it requires orders of magnitude less computation compared to Echo State Networks. Also, for additive …
This paper addresses the reservoir design problem in the context of delay-based reservoir computers for multidimensional input signals, parallel architectures, and real-time multitasking. First, an approximating reservoir model is presented in those frameworks that provides an explicit functional link between the reservoir parameters and architecture and its performance in the execution of a specific task. Second, the inference properties of the ridge regression estimator in the multivariate context is used to assess the impact of finite sample training on the decrease of the reservoir capacit…
