Differentiable neural computers extend artificial neural networks with an explicit memory without interference, thus enabling the model to perform classic computation tasks such as graph traversal. However, such models are difficult to train, requiring long training times and large datasets. In this work, we achieve some of the computational capabilities of differentiable neural computers with a model that can be trained very efficiently, namely an echo state network with an explicit memory without interference. This extension enables echo state networks to recognize all regular languages, including those that contractive echo state networks provably can not recognize. Further, we demonstrate experimentally that our model performs comparably to its fully-trained deep version on several typical benchmark tasks for differentiable neural computers.
Reservoir computing is a neural network approach for processing time-dependent signals that has seen rapid development in recent years. Physical implementations of the technique using optical reservoirs have demonstrated remarkable accuracy and processing speed at benchmark tasks. However, these approaches require an electronic output layer to maintain high performance, which limits their use in tasks such as time-series prediction, where the output is fed back into the reservoir. We present here a reservoir computing scheme that has rapid processing speed both by the reservoir and the output …
Wyant College of Optical Sciences, University of Arizona（亚利桑那大学怀恩特光学科学学院）；School of Electrical Engineering and Telecommunications, University of New South Wales（新南威尔士大学电气工程与电信学院）
In recent years, Neural Turing Machines have gathered attention by joining the flexibility of neural networks with the computational capabilities of Turing machines. However, Neural Turing Machines are notoriously hard to train, which limits their applicability. We propose reservoir memory machines, which are still able to solve some of the benchmark tests for Neural Turing Machines, but are much faster to train, requiring only an alignment algorithm and linear regression. Our model can also be seen as an extension of echo state networks with an external memory, enabling arbitrarily long stora…
University College Cork（科克大学）；Raytheon Technologies Research Center Ireland（雷神技术爱尔兰研究中心）
Multifunctionality is a well observed phenomenological feature of biological neural networks and considered to be of fundamental importance to the survival of certain species over time. These multifunctional neural networks are capable of performing more than one task without changing any network connections. In this paper we investigate how this neurological idiosyncrasy can be achieved in an artificial setting with a modern machine learning paradigm known as `Reservoir Computing'. A training technique is designed to enable a Reservoir Computer to perform tasks of a multifunctional nature. We…
Université Libre de Bruxelles（布鲁塞尔自由大学）；Ghent University（根特大学）
Reservoir computing is a new, powerful and flexible machine learning technique that is easily implemented in hardware. Recently, by using a time-multiplexed architecture, hardware reservoir computers have reached performance comparable to digital implementations. Operating speeds allowing for real time information operation have been reached using optoelectronic systems. At present the main performance bottleneck is the readout layer which uses slow, digital postprocessing. We have designed an analog readout suitable for time-multiplexed optoelectronic reservoir computers, capable of working i…
INRIA Bordeaux Sud-Ouest（法国国家信息与自动化研究所波尔多西南中心）；LaBRI, Université de Bordeaux（波尔多大学LaBRI实验室）；IMN, Université de Bordeaux（波尔多大学IMN研究所）
The prefrontal cortex is known to be involved in many high-level cognitive functions, in particular, working memory. Here, we study to what extent a group of randomly connected units (namely an Echo State Network, ESN) can store and maintain (as output) an arbitrary real value from a streamed input, i.e. can act as a sustained working memory unit. Furthermore, we explore to what extent such an architecture can take advantage of the stored value in order to produce non-linear computations. Comparison between different architectures (with and without feedback, with and without a working memory u…
University of New Mexico（新墨西哥大学）；Portland State University（波特兰州立大学）
Reservoir computing (RC) is a novel approach to time series prediction using recurrent neural networks. In RC, an input signal perturbs the intrinsic dynamics of a medium called a reservoir. A readout layer is then trained to reconstruct a target output from the reservoir's state. The multitude of RC architectures and evaluation metrics poses a challenge to both practitioners and theorists who study the task-solving performance and computational power of RC. In addition, in contrast to traditional computation models, the reservoir is a dynamical system in which computation and memory are insep…
Techniques for Enhancing Memory Capacity of Reservoir Computing
Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology（九州工业大学生命科学与系统工程学院）；Research Center for Neuromorphic AI Hardware, Kyushu Institute of Technology（九州工业大学神经形态人工智能硬件研究中心）；Faculty of Informatics, The University of Fukuchiyama（福知山公立大学信息学部）
Faculty of Physics, University of Duisburg-Essen（物理系，杜伊斯堡-埃森大学）；DyNaMat, Department of Solid State Sciences, Ghent University（DyNaMat，固态科学系，根特大学）；Senior Principle Engineer, Intel Corp. (Retired)（高级原则工程师，英特尔公司（退休））；Faculty of Physics（物理系）；Center for Nanointegration Duisburg-Essen (CENIDE), University of Duisburg-Essen（杜伊斯堡-埃森纳米集成中心（CENIDE），杜伊斯堡-埃森大学）
Institute of Theoretical Physics, Technische Universität Berlin（柏林工业大学理论物理研究所）；Department of Mathematics, Humboldt-Universität zu Berlin（柏林洪堡大学数学系）；Institute of Mathematics, Technische Universität Berlin（柏林工业大学数学研究所）
The Deep Time-Delay Reservoir Computing concept utilizes unidirectionally connected systems with time-delays for supervised learning. We present how the dynamical properties of a deep Ikeda-based reservoir are related to its memory capacity (MC) and how that can be used for optimization. In particular, we analyze bifurcations of the corresponding autonomous system and compute conditional Lyapunov exponents, which measure the generalized synchronization between the input and the layer dynamics. We show how the MC is related to the systems distance to bifurcations or magnitude of the conditional…
School of Electrical, Computer, and Energy Engineering, Arizona State University（亚利桑那州立大学电气、计算机与能源工程学院）；Department of Electrical and Computer Engineering, Colorado State University（科罗拉多州立大学电气与计算机工程系）；DEVCOM Army Research Office（DEVCOM陆军研究办公室）；Department of Physics, Arizona State University（亚利桑那州立大学物理系）
Optimizing the Network Topology of a Linear Reservoir Computer
K. N. Toosi University of Technology, Tehran, Iran（德黑兰技术大学）；Catholic University of America, Washington, D.C., United States（美国天主教大学）；University of New Mexico, Albuquerque, NM 87131, United States（新墨西哥大学）；Max Planck Institute for the Physics of Complex Systems, 01187 Dresden, Germany（复杂系统物理研究所）
Instituto de Física Interdisciplinar y Sistemas Complejos, IFISC (CSIC-UIB)（跨学科物理与复杂系统研究所 (IFISC (CSIC-UIB))）；The University of Tokyo（东京大学）；The Ohio State University（俄亥俄州立大学）；ResCon Technologies, LLC（ResCon Technologies公司）
Reservoir computers are powerful tools for chaotic time series prediction. They can be trained to approximate phase space flows and can thus both predict future values to a high accuracy, as well as reconstruct the general properties of a chaotic attractor without requiring a model. In this work, we show that the ability to learn the dynamics of a complex system can be extended to systems with co-existing attractors, here a 4-dimensional extension of the well-known Lorenz chaotic system. We demonstrate that a reservoir computer can infer entirely unexplored parts of the phase space: a properly…
Potomac Research, LLC（波托马克研究有限责任公司）；The Ohio State University（俄亥俄州立大学）
We propose and demonstrate a nonlinear control method that can be applied to unknown, complex systems where the controller is based on a type of artificial neural network known as a reservoir computer. In contrast to many modern neural-network-based control techniques, which are robust to system uncertainties but require a model nonetheless, our technique requires no prior knowledge of the system and is thus model-free. Further, our approach does not require an initial system identification step, resulting in a relatively simple and efficient learning process. Reservoir computers are well-suit…
Identifying Systems with Symmetries using Equivariant Autoregressive Reservoir Computers
Department of Statistics and Research, CNBS（统计与研究系，CNBS）；Department of Applied Mathematics, UNAH（应用数学系，UNAH）；Department of Chemical Engineering, Norwegian University of Science and Technology（化学工程系，挪威科学技术大学）
Modeling Nonlinear Oscillator Networks Using Physics-Informed Hybrid Reservoir Computing
University of Bristol, School of Computer Science（布里斯托大学计算机科学学院）；University of Bristol, School of Engineering Mathematics and Technology（布里斯托大学工程数学与技术学院）
DTU Electro, Technical University of Denmark（丹麦技术大学DTU Electro）
We propose spectrum-sliced reservoir computer-based (RC) multi-symbol equalization for 32-GBd PAM4 transmission. RC with 17 symbols at the output achieves an order of magnitude reduction in multiplications/symbol versus single output case while maintaining simple training.
We analyze the memory capacity of a delay based reservoir computer with a Hopf normal form as nonlinearity and numerically compute the linear as well as the higher order recall capabilities. A possible physical realisation could be a laser with external cavity, for which the information is fed via electrical injection. A task independent quantification of the computational capability of the reservoir system is done via a complete orthonormal set of basis functions. Our results suggest that even for constant readout dimension the total memory capacity is dependent on the ratio between the infor…
A reservoir computer is a complex dynamical system, often created by coupling nonlinear nodes in a network. The nodes are all driven by a common driving signal. In this work, three dimension estimation methods, false nearest neighbor, covariance and Kaplan-Yorke dimensions, are used to estimate the dimension of the reservoir dynamical system. It is shown that the signals in the reservoir system exist on a relatively low dimensional surface. Changing the spectral radius of the reservoir network can increase the fractal dimension of the reservoir signals, leading to an increase in testing error.
Reservoir computers are a type of neuromorphic computer that may be built a an analog system, potentially creating powerful computers that are small, light and consume little power. Typically a reservoir computer is build by connecting together a set of nonlinear nodes into a network; connecting the nonlinear nodes may be difficult or expensive, however. This work shows how a reservoir computer may be expanded by adding functions to its output. The particular functions described here are linear filters, but other functions are possible. The design and construction of linear filters is well kno…
The Ohio State University（俄亥俄州立大学）；Potomac Research LLC（波托马克研究有限责任公司）
We explore the hyperparameter space of reservoir computers used for forecasting of the chaotic Lorenz '63 attractor with Bayesian optimization. We use a new measure of reservoir performance, designed to emphasize learning the global climate of the forecasted system rather than short-term prediction. We find that optimizing over this measure more quickly excludes reservoirs that fail to reproduce the climate. The results of optimization are surprising: the optimized parameters often specify a reservoir network with very low connectivity. Inspired by this observation, we explore reservoir design…
We show that many delay-based reservoir computers considered in the literature can be characterized by a universal master memory function (MMF). Once computed for two independent parameters, this function provides linear memory capacity for any delay-based single-variable reservoir with small inputs. Moreover, we propose an analytical description of the MMF that enables its efficient and fast computation. Our approach can be applied not only to reservoirs governed by known dynamical rules such as Mackey-Glass or Ikeda-like systems but also to reservoirs whose dynamical model is not available. …
W. M. Keck Science Department of Pitzer, Scripps, and Claremont McKenna College（皮泽学院、斯克里普斯学院和克莱蒙特·麦肯纳学院W.M.凯克科学系）；Nanyang Technological University（南洋理工大学）；University of California at Davis（加州大学戴维斯分校）
Recurrent neural networks are used to forecast time series in finance, climate, language, and from many other domains. Reservoir computers are a particularly easily trainable form of recurrent neural network. Recently, a "next-generation" reservoir computer was introduced in which the memory trace involves only a finite number of previous symbols. We explore the inherent limitations of finite-past memory traces in this intriguing proposal. A lower bound from Fano's inequality shows that, on highly non-Markovian processes generated by large probabilistic state machines, next-generation reservoi…
University of Alberta（阿尔伯塔大学）；Tsinghua University（清华大学）
Episodic memory is a psychology term which refers to the ability to recall specific events from the past. We suggest one advantage of this particular type of memory is the ability to easily assign credit to a specific state when remembered information is found to be useful. Inspired by this idea, and the increasing popularity of external memory mechanisms to handle long-term dependencies in deep learning systems, we propose a novel algorithm which uses a reservoir sampling procedure to maintain an external memory consisting of a fixed number of past states. The algorithm allows a deep reinforc…
University of New Mexico（新墨西哥大学）；U.S. Naval Research Laboratory（美国海军研究实验室）
The topology of a network associated with a reservoir computer is often taken so that the connectivity and the weights are chosen randomly. Optimization is hardly considered as the parameter space is typically too large. Here we investigate this problem for a class of reservoir computers for which we obtain a decomposition of the reservoir dynamics into modes, which can be computed independently of one another. Each mode depends on an eigenvalue of the network adjacency matrix. We then take a parametric approach in which the eigenvalues are parameters that can be appropriately designed and opt…
Petrozavodsk State University（彼得罗扎沃茨克国立大学）；K-SkAI LLC（K-SkAI有限责任公司）
The presented compact algorithm for recognizing handwritten digits of the MNIST database, created on the LogNNet reservoir neural network, reaches the recognition accuracy of 82%. The algorithm was tested on a low-memory Arduino board with 2 Kb static RAM low-power microcontroller. The dependences of the accuracy and time of image recognition on the number of neurons in the reservoir have been investigated. The memory allocation demonstrates that the algorithm stores all the necessary information in RAM without using additional data storage, and operates with original images without preliminar…
Exploring the origins of switching dynamics in a multifunctional reservoir computer
School of Mathematical Sciences, University College Cork（科克大学学院数学科学学院）；INFANT Research Centre, University College Cork（科克大学学院INFANT研究中心）；Potsdam Institute for Climate Impact Research（波茨坦气候影响研究所）
G.B Pant Institute of Technology（G.B.潘特技术学院）；Netaji Subhas Institute of Technology（内塔吉·苏巴斯技术学院）；University of Delhi（德里大学）
A reliable forecast of inflows to the reservoir is a key factor in the optimal operation of reservoirs. Real-time operation of the reservoir based on forecasts of inflows can lead to substantial economic gains. However, the forecast of inflow is an intricate task as it has to incorporate the impacts of climate and hydrological changes. Therefore, the major objective of the present work is to develop a novel approach based on long short-term memory (LSTM) for the forecast of inflows. Real-time inflow forecast, in other words, daily inflow at the reservoir helps in efficient operation of water r…
University College Cork（科克大学学院）；Ludwig-Maximilians University Munich（慕尼黑路德维希-马克西米利安大学）；Collins Aerospace（柯林斯航空航天公司）；Deutsches Zentrum für Luft-und Raumfahrt（德国航空航天中心）
Multifunctional neural networks are capable of performing more than one task without changing any network connections. In this paper we explore the performance of a continuous-time, leaky-integrator, and next-generation `reservoir computer' (RC), when trained on tasks which test the limits of multifunctionality. In the first task we train each RC to reconstruct a coexistence of chaotic attractors from different dynamical systems. By moving the data describing these attractors closer together, we find that the extent to which each RC can reconstruct both attractors diminishes as they begin to o…
School of Physics and Information Technology, Shaanxi Normal University（陕西师范大学物理与信息技术学院）
Reconstructing the KAM dynamics diagram of Hamiltonian system from the time series of a limited number of parameters is an outstanding question in nonlinear science, especially when the Hamiltonian governing the system dynamics are unknown. Here, we demonstrate that this question can be addressed by the machine learning approach knowing as reservoir computer (RC). Specifically, we show that without prior knowledge about the Hamilton's equations of motion, the trained RC is able to not only predict the short-term evolution of the system state, but also replicate the long-term ergodic properties…
Optimal training of finitely-sampled quantum reservoir computers for forecasting of chaotic dynamics
Imperial College London（伦敦帝国理工学院）；City St George’s, University of London（伦敦大学城市圣乔治学院）；The Alan Turing Institute（艾伦·图灵研究所）；Politecnico di Torino（都灵理工大学）
