We extend the Physics-Informed Echo State Network (PI-ESN) framework to reconstruct the evolution of an unmeasured state (hidden state) in a chaotic system. The PI-ESN is trained by using (i) data, which contains no information on the unmeasured state, and (ii) the physical equations of a prototypical chaotic dynamical system. Non-noisy and noisy datasets are considered. First, it is shown that the PI-ESN can accurately reconstruct the unmeasured state. Second, the reconstruction is shown to be robust with respect to noisy data, which means that the PI-ESN acts as a denoiser. This paper opens up new possibilities for leveraging the synergy between physical knowledge and machine learning to enhance the reconstruction and prediction of unmeasured states in chaotic dynamical systems.
University of Cambridge（剑桥大学）；The Alan Turing Institute（阿兰·图灵研究所）；Technical University of Munich（慕尼黑工业大学）
An approach to the time-accurate prediction of chaotic solutions is by learning temporal patterns from data. Echo State Networks (ESNs), which are a class of Reservoir Computing, can accurately predict the chaotic dynamics well beyond the predictability time. Existing studies, however, also showed that small changes in the hyperparameters may markedly affect the network's performance. The aim of this paper is to assess and improve the robustness of Echo State Networks for the time-accurate prediction of chaotic solutions. The goal is three-fold. First, we investigate the robustness of routinel…
University of Cambridge（剑桥大学）；The Alan Turing Institute（艾伦·图灵研究所）；Imperial College London（帝国理工学院）；Technical University of Munich（慕尼黑工业大学）
We propose the Automatic-differentiated Physics-Informed Echo State Network (API-ESN). The network is constrained by the physical equations through the reservoir's exact time-derivative, which is computed by automatic differentiation. As compared to the original Physics-Informed Echo State Network, the accuracy of the time-derivative is increased by up to seven orders of magnitude. This increased accuracy is key in chaotic dynamical systems, where errors grows exponentially in time. The network is showcased in the reconstruction of unmeasured (hidden) states of a chaotic system. The API-ESN el…
Advanced Research Center, National Fusion Research Institute (NFRI)（国家聚变研究所高级研究中心）
Inspired by recent theoretical arguments, physics-informed echo state network (ESN) is discussed on the attempt to train a reservoir model absolutely in physics-informed manner. As the plainest work on such a purpose, an ODE (ordinary differential equation) approximator is designed to replicate the solution in sequence with respect to the recurrent evaluations. On the principal invariance of differential equations, the constraint in recurrence just takes shape to secure a proper regression method for the ESN-based ODE approximator. After then, the actual training process is established on the …
Minimal Deterministic Echo State Networks Outperform Random Reservoirs in Learning Chaotic Dynamics
Center for Scalable Data Analytics；Artificial Intelligence (ScaDS.AI), Leipzig University, Leipzig, Germany；Institute for Earth System Science \& Remote Sensing, Leipzig University, Leipzig, Germany
Technical University of Munich（慕尼黑工业大学）；University of Cambridge（剑桥大学）
We propose a physics-informed Echo State Network (ESN) to predict the evolution of chaotic systems. Compared to conventional ESNs, the physics-informed ESNs are trained to solve supervised learning tasks while ensuring that their predictions do not violate physical laws. This is achieved by introducing an additional loss function during the training, which is based on the system's governing equations. The additional loss function penalizes non-physical predictions without the need of any additional training data. This approach is demonstrated on a chaotic Lorenz system and a truncation of the …
Technical University of Munich（慕尼黑工业大学）；Institute for Advanced Study, Technical University of Munich（慕尼黑工业大学高等研究院）；University of Cambridge（剑桥大学）
We propose a physics-informed Echo State Network (ESN) to predict the evolution of chaotic systems. Compared to conventional ESNs, the physics-informed ESNs are trained to solve supervised learning tasks while ensuring that their predictions do not violate physical laws. This is achieved by introducing an additional loss function during the training of the ESNs, which penalizes non-physical predictions without the need of any additional training data. This approach is demonstrated on a chaotic Lorenz system, where the physics-informed ESNs improve the predictability horizon by about two Lyapun…
Artificial Intelligence Center（人工智能中心）；Czech Technical University（捷克技术大学）；Niels Bohr Institute（尼尔斯·玻尔研究所）；University of Copenhagen（哥本哈根大学）
This work builds an automated anomaly detection method for chaotic time series, and more concretely for turbulent, high-dimensional, ocean simulations. We solve this task by extending the Echo State Network by spatially aware input maps, such as convolutions, gradients, cosine transforms, et cetera, as well as a spatially aware loss function. The spatial ESN is used to create predictions which reduce the detection problem to thresholding of the prediction error. We benchmark our detection framework on different tasks of increasing difficulty to show the generality of the framework before apply…
University of Cambridge（剑桥大学）；Imperial College London（帝国理工学院）；The Alan Turing Institute（艾伦·图灵研究所）；TU Munich（慕尼黑工业大学）
We develop a versatile optimization method, which finds the design parameters that minimize time-averaged acoustic cost functionals. The method is gradient-free, model-informed, and data-driven with reservoir computing based on echo state networks. First, we analyse the predictive capabilities of echo state networks both in the short- and long-time prediction of the dynamics. We find that both fully data-driven and model-informed architectures learn the chaotic acoustic dynamics, both time-accurately and statistically. Informing the training with a physical reduced-order model with one acousti…
In the framework of physical reservoir computing (RC), machine learning algorithms designed for digital computers are executed using analog computer-like nonlinear physical systems that can provide energy-efficient computational power for predicting time-dependent quantities that can be found using nonlinear differential equations. We suggest a bubble-based RC (BRC) system that combines the nonlinearity of an acoustic response of a cluster of oscillating gas bubbles in water with a standard Echo State Network (ESN) algorithm that is well-suited to forecast chaotic time series. We confirm the p…
University of Cambridge（剑桥大学）；Technical University of Munich（慕尼黑工业大学）
We propose a physics-informed machine learning method to predict the time average of a chaotic attractor. The method is based on the hybrid echo state network (hESN). We assume that the system is ergodic, so the time average is equal to the ergodic average. Compared to conventional echo state networks (ESN) (purely data-driven), the hESN uses additional information from an incomplete, or imperfect, physical model. We evaluate the performance of the hESN and compare it to that of an ESN. This approach is demonstrated on a chaotic time-delayed thermoacoustic system, where the inclusion of a phys…
We report an improvement to the conventional Echo State Network (ESN) across three benchmark chaotic time-series prediction tasks using fruit fly connectome data alone. We also investigate the impact of key connectome-derived structural features on prediction performance -- uniquely bridging neurobiological structure and machine learning function; and find that both increasing the global average clustering coefficient and modifying the position of weights -- by permuting their synapse-synapse partners -- can lead to increased model variance and (in some cases) degraded performance. In all we c…
Echo State Networks for Bitcoin Time Series Prediction
DFKI, Saarland Informatics Campus, Germany（德国达姆施塔特研究所，萨尔兰信息技术校园）；Politecnico di Milano, Italy（意大利米兰理工学院）；University of Stirling, UK（英国斯特林大学）；National Institute of Informatics, Japan（日本信息学研究院）
Can connectome-derived constraints inform computation? In this paper we investigate the contribution of a fruit fly connectome's topology on the performance of an Echo State Network (ESN) -- a subset of Reservoir Computing which is state of the art in chaotic time series prediction. Specifically, we replace the reservoir layer of a classical ESN -- normally a fixed, random graph represented as a 2-d matrix -- with a particular (female) fruit fly connectome-derived connectivity matrix. We refer to this experimental class of models (with connectome-derived reservoirs) as "Fruit Fly ESNs" (FFESNs…
Echo State Networks for Spatio-Temporal Area-Level Data
Department of Statistics, University of Missouri（统计学系，密苏里大学）；Office of the Associate Director for Research and Methodology, U.S. Census Bureau（研究与方法副主任办公室，美国人口普查局）
Laboratoire Kastler Brossel（卡斯特勒布罗塞尔实验室）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）；Laboratoire de Physique Statistique（统计物理实验室）；LightOn（LightOn公司）
Echo-State Networks and Reservoir Computing have been studied for more than a decade. They provide a simpler yet powerful alternative to Recurrent Neural Networks, every internal weight is fixed and only the last linear layer is trained. They involve many multiplications by dense random matrices. Very large networks are difficult to obtain, as the complexity scales quadratically both in time and memory. Here, we present a novel optical implementation of Echo-State Networks using light-scattering media and a Digital Micromirror Device. As a proof of concept, binary networks have been successful…
Massachusetts Institute of Technology（麻省理工学院）；Julia Computing（Julia Computing公司）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
Modern design, control, and optimization often requires simulation of highly nonlinear models, leading to prohibitive computational costs. These costs can be amortized by evaluating a cheap surrogate of the full model. Here we present a general data-driven method, the continuous-time echo state network (CTESN), for generating surrogates of nonlinear ordinary differential equations with dynamics at widely separated timescales. We empirically demonstrate near-constant time performance using our CTESNs on a physically motivated scalable model of a heating system whose full execution time increase…
University of Tromsø - The Arctic University of Norway（特罗姆瑟大学-挪威北极大学）
In this paper we introduce a new framework to train an Echo State Network to predict real valued time-series. The method consists in projecting the output of the internal layer of the network on a space with lower dimensionality, before training the output layer to learn the target task. Notably, we enforce a regularization constraint that leads to better generalization capabilities. We evaluate the performances of our approach on several benchmark tests, using different techniques to train the readout of the network, achieving superior predictive performance when using the proposed framework.…
University of Cambridge（剑桥大学）；Imperial College London（帝国理工学院）；The Alan Turing Institute（艾伦·图灵研究所）；TU Munich（慕尼黑工业大学）
An extreme event is a sudden and violent change in the state of a nonlinear system. In fluid dynamics, extreme events can have adverse effects on the system's optimal design and operability, which calls for accurate methods for their prediction and control. In this paper, we propose a data-driven methodology for the prediction and control of extreme events in a chaotic shear flow. The approach is based on echo state networks, which are a type of reservoir computing that learn temporal correlations within a time-dependent dataset. The objective is five-fold. First, we exploit ad-hoc metrics fro…
Università della Svizzera Italiana（瑞士意大利语区大学）；Politecnico di Milano（米兰理工大学）；University of Manitoba（曼尼托巴大学）；University of Exeter（埃克塞特大学）
Among the various architectures of Recurrent Neural Networks, Echo State Networks (ESNs) emerged due to their simplified and inexpensive training procedure. These networks are known to be sensitive to the setting of hyper-parameters, which critically affect their behaviour. Results show that their performance is usually maximized in a narrow region of hyper-parameter space called edge of chaos. Finding such a region requires searching in hyper-parameter space in a sensible way: hyper-parameter configurations marginally outside such a region might yield networks exhibiting fully developed chaos…
Faculty of Electrical Engineering and Computer Science（电气工程与计算机科学学院）；VŠB-Technical University of Ostrava（俄斯特拉发技术大学）；INRIA Rennes – Bretagne Atlantique（法国国家信息与自动化研究所雷恩-布列塔尼大西洋中心）
The Echo State Network (ESN) is a class of Recurrent Neural Network with a large number of hidden-hidden weights (in the so-called reservoir). Canonical ESN and its variations have recently received significant attention due to their remarkable success in the modeling of non-linear dynamical systems. The reservoir is randomly connected with fixed weights that don't change in the learning process. Only the weights from reservoir to output are trained. Since the reservoir is fixed during the training procedure, we may wonder if the computational power of the recurrent structure is fully harnesse…
TECNALIA（泰克纳利亚）；Basque Research and Technology Alliance (BRTA)（巴斯克研究与技术联盟（BRTA））；University of the Basque Country (UPV/EHU)（巴斯克大学（UPV/EHU））
Since their inception, learning techniques under the Reservoir Computing paradigm have shown a great modeling capability for recurrent systems without the computing overheads required for other approaches. Among them, different flavors of echo state networks have attracted many stares through time, mainly due to the simplicity and computational efficiency of their learning algorithm. However, these advantages do not compensate for the fact that echo state networks remain as black-box models whose decisions cannot be easily explained to the general audience. This work addresses this issue by co…
University of Cambridge（剑桥大学）；Delft University of Technology（代尔夫特理工大学）；Imperial College London（帝国理工学院）；The Alan Turing Institute（艾伦·图灵研究所）
The spatiotemporal dynamics of turbulent flows is chaotic and difficult to predict. This makes the design of accurate and stable reduced-order models challenging. The overarching objective of this paper is to propose a nonlinear decomposition of the turbulent state for a reduced-order representation of the dynamics. We divide the turbulent flow into a spatial problem and a temporal problem. First, we compute the latent space, which is the manifold onto which the turbulent dynamics live (i.e., it is a numerical approximation of the turbulent attractor). The latent space is found by a series of …
In this paper, the performance of three deep learning methods for predicting short-term evolution and for reproducing the long-term statistics of a multi-scale spatio-temporal Lorenz 96 system is examined. The methods are: echo state network (a type of reservoir computing, RC-ESN), deep feed-forward artificial neural network (ANN), and recurrent neural network with long short-term memory (RNN-LSTM). This Lorenz 96 system has three tiers of nonlinearly interacting variables representing slow/large-scale ($X$), intermediate ($Y$), and fast/small-scale ($Z$) processes. For training or testing, on…
Prediction of Nonlinear Oscillations in a Jumping Quarter-Car Model Using Reservoir Computing
Tokyo University of Agriculture and Technology（东京农工大学）；Amity University Haryana - Gurugram（Amity大学哈里亚纳邦-古鲁格拉姆校区）；Nagoya Institute of Technology（名古屋工业大学）；Central University of Rajasthan（中央拉贾斯坦大学）
Adaptive Nonlinear Vector Autoregression: Robust Forecasting for Noisy Chaotic Time Series
Department of Mathematics, Pusan National University, Republic of Korea（釜山国立大学数学系）；Pusan National University, Republic of Korea（釜山国立大学）；Industrial Mathematics Center, Pusan National University, Republic of Korea（釜山国立大学工业数学中心）；Humanoid Olfactory Display Center, Pusan National University, Republic of Korea（釜山国立大学人形嗅觉显示中心）；Department of Mathematics, Yonsei University, Republic of Korea（成均馆大学数学系）；Innovation Center for MathScience Research&Education, Pusan National University, Republic of Korea（釜山国立大学数学科学研究与教育创新中心）；Marine Natural Disaster Research Department, Korea Institute of Ocean Science and Technology, Republic of Korea（韩国海洋科学与技术研究院海洋自然灾害研究部）；Institute for Future Earth, Pusan National University, Republic of Korea（未来地球研究所）
In some Internet of Things (IoT) applications, multi-path propagation is a main constraint of communication channel. Recently, the chaotic baseband wireless communication system (CBWCS) is promising to eliminate the inter-symbol interference (ISI) caused by multipath propagation. However, the current technique is only capable of removing the partial effect of ISI, due to only past decoded bits are available for the suboptimal decoding threshold calculation. However, the future transmitting bits also contribute to the threshold. The unavailable future information bits needed by the optimal deco…
Generalized synchronization between coupled dynamical systems is a phenomenon of relevance in applications that range from secure communications to physiological modelling. Here we test the capabilities of reservoir computing and, in particular, echo state networks for the detection of generalized synchronization. A nonlinear dynamical system consisting of two coupled Rössler chaotic attractors is used to generate temporal series consisting of time-locked generalized synchronized sequences interleaved by unsynchronized ones. Correctly tuned, echo state networks are able to efficiently discrimi…
