Reservoir Computing (RC) models, a subclass of recurrent neural networks, are distinguished by their fixed, non-trainable input layer and dynamically coupled reservoir, with only the static readout layer being trained. This design circumvents the issues associated with backpropagating error signals through time, thereby enhancing both stability and training efficiency. RC models have been successfully applied across a broad range of application domains. Crucially, they have been demonstrated to be universal approximators of time-invariant dynamic filters with fading memory, under various settings of approximation norms and input driving sources.
Simple Cycle Reservoirs (SCR) represent a specialized class of RC models with a highly constrained reservoir architecture, characterized by uniform ring connectivity and binary input-to-reservoir weights with an aperiodic sign pattern. For linear reservoirs, given the reservoir size, the reservoir construction has only one degree of freedom -- the reservoir cycle weight. Such architectures are particularly amenable to hardware implementations without significant performance degradation in many practical tasks. In this study we endow these observations with solid theoretical foundations by proving that SCRs operating in real domain are universal approximators of time-invariant dynamic filters with fading memory. Our results supplement recent research showing that SCRs in the complex domain can approximate, to arbitrary precision, any unrestricted linear reservoir with a non-linear readout. We furthermore introduce a novel method to drastically reduce the number of SCR units, making such highly constrained architectures natural candidates for low-complexity hardware implementations. Our findings are supported by empirical studies on real-world time series datasets.
School of Computer Science University of St. Gallen（圣加尔登大学计算机科学学院）；Donostia International Physics Center（多尼亚国际物理中心）；Department of Mathematics, Imperial College（帝国理工学院数学系）；School of Physical and Mathematical Sciences Nanyang Technological University（南洋理工大学物理与数学科学学院）
Minimal Deterministic Echo State Networks Outperform Random Reservoirs in Learning Chaotic Dynamics
Center for Scalable Data Analytics；Artificial Intelligence (ScaDS.AI), Leipzig University, Leipzig, Germany；Institute for Earth System Science \& Remote Sensing, Leipzig University, Leipzig, Germany
Wyant College of Optical Sciences, University of Arizona（亚利桑那大学怀恩特光学科学学院）；School of Electrical Engineering and Telecommunications, University of New South Wales（新南威尔士大学电气工程与电信学院）
Structuring Multiple Simple Cycle Reservoirs with Particle Swarm Optimization
International Research Center for Neurointelligence, The University of Tokyo（国际神经智能研究中心，东京大学）；School of Computer Science, University of Birmingham（伯明翰大学计算机科学学院）；Graduate School of Medicine, The University of Tokyo（东京大学医学研究生院）；Department of Computer Science, Graduate School of Engineering, Nagoya Institute of Technology（名古屋工业大学工学研究生院计算机科学系）
Universität Sankt Gallen（圣加仑大学）；Universität Konstanz（康斯坦茨大学）；Centre National de la Recherche Scientifique (CNRS)（法国国家科学研究中心）
We analyze the practices of reservoir computing in the framework of statistical learning theory. In particular, we derive finite sample upper bounds for the generalization error committed by specific families of reservoir computing systems when processing discrete-time inputs under various hypotheses on their dependence structure. Non-asymptotic bounds are explicitly written down in terms of the multivariate Rademacher complexities of the reservoir systems and the weak dependence structure of the signals that are being handled. This allows, in particular, to determine the minimal number of obs…
The Ohio State University（俄亥俄州立大学）；ResCon Technologies, LLC（ResCon Technologies公司）；Clarkson University（克拉克森大学）
Reservoir computing is a best-in-class machine learning algorithm for processing information generated by dynamical systems using observed time-series data. Importantly, it requires very small training data sets, uses linear optimization, and thus requires minimal computing resources. However, the algorithm uses randomly sampled matrices to define the underlying recurrent neural network and has a multitude of metaparameters that must be optimized. Recent results demonstrate the equivalence of reservoir computing to nonlinear vector autoregression, which requires no random matrices, fewer metap…
Oscillations enhance time-series prediction in reservoir computing with feedback
Symbiotic Intelligent Systems Research Center, Institute for Open and Transdisciplinary Research Initiatives, Osaka University（共生智能系统研究中心，开放与跨学科研究倡议所，大阪大学）；Academy of Emerging Sciences, Chubu University（新兴科学学院，名古屋大学）；Center for Information and Neural Networks, National Institute of Information and Communications Technology（信息与神经网络中心，信息与通信技术国家研究所）；International Professional University of Technology in Osaka（大阪国际技术专业大学）
A recurrent neural network (RNN) is a widely used deep-learning network for dealing with sequential data. Imitating a dynamical system, an infinite-width RNN can approximate any open dynamical system in a compact domain. In general, deep networks with bounded widths are more effective than wide networks in practice; however, the universal approximation theorem for deep narrow structures has yet to be extensively studied. In this study, we prove the universality of deep narrow RNNs and show that the upper bound of the minimum width for universality can be independent of the length of the data. …
G.B Pant Institute of Technology（G.B.潘特技术学院）；Netaji Subhas Institute of Technology（内塔吉·苏巴斯技术学院）；University of Delhi（德里大学）
A reliable forecast of inflows to the reservoir is a key factor in the optimal operation of reservoirs. Real-time operation of the reservoir based on forecasts of inflows can lead to substantial economic gains. However, the forecast of inflow is an intricate task as it has to incorporate the impacts of climate and hydrological changes. Therefore, the major objective of the present work is to develop a novel approach based on long short-term memory (LSTM) for the forecast of inflows. Real-time inflow forecast, in other words, daily inflow at the reservoir helps in efficient operation of water r…
W. M. Keck Science Department of Pitzer, Scripps, and Claremont McKenna College（皮泽学院、斯克里普斯学院和克莱蒙特·麦肯纳学院W.M.凯克科学系）；Nanyang Technological University（南洋理工大学）；University of California at Davis（加州大学戴维斯分校）
Recurrent neural networks are used to forecast time series in finance, climate, language, and from many other domains. Reservoir computers are a particularly easily trainable form of recurrent neural network. Recently, a "next-generation" reservoir computer was introduced in which the memory trace involves only a finite number of previous symbols. We explore the inherent limitations of finite-past memory traces in this intriguing proposal. A lower bound from Fano's inequality shows that, on highly non-Markovian processes generated by large probabilistic state machines, next-generation reservoi…
