Deep Time-Delay Reservoir Computing: Dynamics and Memory Capacity
Institute of Theoretical Physics, Technische Universität Berlin（柏林工业大学理论物理研究所）；Department of Mathematics, Humboldt-Universität zu Berlin（柏林洪堡大学数学系）；Institute of Mathematics, Technische Universität Berlin（柏林工业大学数学研究所）
Mirko Goldmann, Felix Köster, Kathy Lüdge, Serhiy Yanchuk
The Deep Time-Delay Reservoir Computing concept utilizes unidirectionally connected systems with time-delays for supervised learning. We present how the dynamical properties of a deep Ikeda-based reservoir are related to its memory capacity (MC) and how that can be used for optimization. In particular, we analyze bifurcations of the corresponding autonomous system and compute conditional Lyapunov exponents, which measure the generalized synchronization between the input and the layer dynamics. We show how the MC is related to the systems distance to bifurcations or magnitude of the conditional Lyapunov exponent. The interplay of different dynamical regimes leads to a adjustable distribution between linear and nonlinear MC. Furthermore, numerical simulations show resonances between clock cycle and delays of the layers in all degrees of the MC. Contrary to MC losses in a single-layer reservoirs, these resonances can boost separate degrees of the MC and can be used, e.g., to design a system with maximum linear MC. Accordingly, we present two configurations that empower either high nonlinear MC or long time linear MC.
In this paper we give a profound insight into the computation capability of delay-based reservoir computing via an eigenvalue analysis. We concentrate on the task-independent memory capacity to quantify the reservoir performance and compare these with the eigenvalue spectrum of the dynamical system. We show that these two quantities are deeply connected, and thus the reservoir computing performance is predictable by analyzing the small signal response of the reservoir. Our results suggest that any dynamical system used as a reservoir can be analyzed in this way. We apply our method exemplarily…
Embedding Theory of Reservoir Computing and Reducing Reservoir Network Using Time Delays
Soochow University（苏州大学）；Fudan University（复旦大学）；Potsdam Institute for Climate Impact Research (PIK)（波茨坦气候影响研究所（PIK））；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Technische Universität Berlin（柏林工业大学）；Humboldt-Universität zu Berlin（柏林洪堡大学）；Instituto de Física Interdisciplinar y Sistemas Complejos, IFISC (CSIC-UIB)（跨学科物理与复杂系统研究所（IFISC，西班牙国家研究委员会-巴利阿里群岛大学））
The time-delay-based reservoir computing setup has seen tremendous success in both experiment and simulation. It allows for the construction of large neuromorphic computing systems with only few components. However, until now the interplay of the different timescales has not been investigated thoroughly. In this manuscript, we investigate the effects of a mismatch between the time-delay and the clock cycle for a general model. Typically, these two time scales are considered to be equal. Here we show that the case of equal or resonant time-delay and clock cycle could be actively detrimental and…
We analyze the memory capacity of a delay based reservoir computer with a Hopf normal form as nonlinearity and numerically compute the linear as well as the higher order recall capabilities. A possible physical realisation could be a laser with external cavity, for which the information is fed via electrical injection. A task independent quantification of the computational capability of the reservoir system is done via a complete orthonormal set of basis functions. Our results suggest that even for constant readout dimension the total memory capacity is dependent on the ratio between the infor…
Techniques for Enhancing Memory Capacity of Reservoir Computing
Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology（九州工业大学生命科学与系统工程学院）；Research Center for Neuromorphic AI Hardware, Kyushu Institute of Technology（九州工业大学神经形态人工智能硬件研究中心）；Faculty of Informatics, The University of Fukuchiyama（福知山公立大学信息学部）
To perform temporal and sequential machine learning tasks, the use of conventional Recurrent Neural Networks (RNNs) has been dwindling due to the training complexities of RNNs. To this end, accelerators for delayed feedback reservoir computing (DFRC) have attracted attention in lieu of RNNs, due to their simple hardware implementations. A typical implementation of a DFRC accelerator consists of a delay loop and a single nonlinear neuron, together acting as multiple virtual nodes for computing. In prior work, photonic DFRC accelerators have shown an undisputed advantage of fast computation over…
School of Electrical, Computer, and Energy Engineering, Arizona State University（亚利桑那州立大学电气、计算机与能源工程学院）；Department of Electrical and Computer Engineering, Colorado State University（科罗拉多州立大学电气与计算机工程系）；DEVCOM Army Research Office（DEVCOM陆军研究办公室）；Department of Physics, Arizona State University（亚利桑那州立大学物理系）
Institute of Fundamental and Frontier Sciences, University of Electronic Science and Technology of China（电子科技大学基础与前沿科学研究所）；Potsdam Institute for Climate Impact Research（波茨坦气候影响研究所）；Department of Physics, Humboldt University Berlin（柏林洪堡大学物理系）；Research Institute of Intelligent Complex Systems, Fudan University（复旦大学智能复杂系统研究所）；School of Physics, University of Electronic Science and Technology of China（电子科技大学物理学院）；Key Laboratory of Quantum Physics and Photonic Quantum Information, Ministry of Education, University of Electronic Science and Technology of China（电子科技大学教育部量子物理与光子量子信息重点实验室）；Non-classical Information Science Basic Discipline Research Center of Sichuan Province, University of Electronic Science and Technology of China（电子科技大学四川省非经典信息科学基础学科研究中心）
Harnessing Synthetic Active Particles for Physical Reservoir Computing
Peter Debye Institute for Soft Matter Physics, Leipzig University（莱比锡大学彼得·德拜软物质物理研究所）；Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Dresden/Leipzig（德累斯顿/莱比锡可扩展数据分析与人工智能中心（ScaDS.AI））
Understanding the fundamental relationships between physics and its information-processing capability has been an active research topic for many years. Physical reservoir computing is a recently introduced framework that allows one to exploit the complex dynamics of physical systems as information-processing devices. This framework is particularly suited for edge computing devices, in which information processing is incorporated at the edge (e.g., into sensors) in a decentralized manner to reduce the adaptation delay caused by data transmission overhead. This paper aims to illustrate the poten…
Current radio frequency (RF) sensors at the Edge lack the computational resources to support practical, in-situ training for intelligent spectrum monitoring, and sensor data classification in general. We propose a solution via Deep Delay Loop Reservoir Computing (DLR), a processing architecture that supports general machine learning algorithms on compact mobile devices by leveraging delay-loop reservoir computing in combination with innovative electrooptical hardware. With both digital and photonic realizations of our design of the loops, DLR delivers reductions in form factor, hardware comple…
The University of Tokyo（东京大学）；Keio University（庆应义塾大学）
Quantum computing has been moving from a theoretical phase to practical one, presenting daunting challenges in implementing physical qubits, which are subjected to noises from the surrounding environment. These quantum noises are ubiquitous in quantum devices and generate adverse effects in the quantum computational model, leading to extensive research on their correction and mitigation techniques. But do these quantum noises always provide disadvantages? We tackle this issue by proposing a framework called quantum noise-induced reservoir computing and show that some abstract quantum noise mod…
Reservoir computing is a neural network approach for processing time-dependent signals that has seen rapid development in recent years. Physical implementations of the technique using optical reservoirs have demonstrated remarkable accuracy and processing speed at benchmark tasks. However, these approaches require an electronic output layer to maintain high performance, which limits their use in tasks such as time-series prediction, where the output is fed back into the reservoir. We present here a reservoir computing scheme that has rapid processing speed both by the reservoir and the output …
Reservoir computing(RC) is a brain-inspired computing framework that employs a transient dynamical system whose reaction to an input signal is transformed to a target output. One of the central problems in RC is to find a reliable reservoir with a large criticality, since computing performance of a reservoir is maximized near the phase transition. In this work, we propose a continuous reservoir that utilizes transient dynamics of coupled chaotic oscillators in a critical regime where sudden amplitude death occurs. This "explosive death" not only brings the system a large criticality which prov…
We introduce a novel design for in-situ training of machine learning algorithms built into smart sensors, and illustrate distributed training scenarios using radio frequency (RF) spectrum sensors. Current RF sensors at the Edge lack the computational resources to support practical, in-situ training for intelligent signal classification. We propose a solution using Deepdelay Loop Reservoir Computing (DLR), a processing architecture that supports machine learning algorithms on resource-constrained edge-devices by leveraging delayloop reservoir computing in combination with innovative hardware. D…
We show that many delay-based reservoir computers considered in the literature can be characterized by a universal master memory function (MMF). Once computed for two independent parameters, this function provides linear memory capacity for any delay-based single-variable reservoir with small inputs. Moreover, we propose an analytical description of the MMF that enables its efficient and fast computation. Our approach can be applied not only to reservoirs governed by known dynamical rules such as Mackey-Glass or Ikeda-like systems but also to reservoirs whose dynamical model is not available. …
Reservoir computing is a machine learning paradigm that uses a high-dimensional dynamical system, or \emph{reservoir}, to approximate and predict time series data. The scale, speed and power usage of reservoir computers could be enhanced by constructing reservoirs out of electronic circuits, and several experimental studies have demonstrated promise in this direction. However, designing quality reservoirs requires a precise understanding of how such circuits process and store information. We analyze the feasibility and optimal design of electronic reservoirs that include both linear elements (…
The role of the feedback effect on physical reservoir computing is studied theoretically by solving the vortex-core dynamics in a nanostructured ferromagnet. Although the spin-transfer torque due to the feedback current makes the vortex dynamics complex, it is clarified that the feedback effect does not always contribute to the enhancement of the memory function in a physical reservoir. The memory function, characterized by the correlation coefficient between the input data and the dynamical response of the vortex core, becomes large when the delay time of the feedback current is not an integr…
The influence of array geometry on synchronization properties of a 2-D oscillator array is investigated based on a comparison between a rectangular and a hexagonal grid. The Kuramoto model is solved for a nearest-neighbor case with periodic boundary conditions and for a small-scale, realistic coupling case with 1/r^3 decay characteristic of spintronic oscillators. In both cases, it is found that the hexagonal grid choice leads to lower synchronization threshold and higher emission power than its rectangular counterpart, which results from increased connectivity, as well as, in the realistic-co…
A numerical analysis on the computational capability of physical reservoir computing utilizing a spin-torque oscillator with two free layers is reported. Conventional spintronics devices usually consist of two ferromagnets, where the direction of magnetization in one layer, called the free layer, can move while that of the other, the reference layer, is fixed. Recently, however, devices with two free layers, where the reference layer is replaced by another free layer, have been developed for various practical applications. Adding another free layer drastically changes the dynamical response of…
