针对初始活动自发混沌的回声状态网络，本文提出基于递归最小二乘并结合动态回归器扩展与记忆数据利用的复合 FORCE 学习方法，在 Mackey-Glass 混沌时间序列预测中显著提升性能。
摘要：
回声状态网络（Echo state network，ESN）是一种循环神经网络，由一个神经元以随机且递归方式连接的固定储备池组成，并且仅通过训练输出连接权重来获得期望输出。一阶约简与受控误差（First-order reduced and controlled error，FORCE）学习是一种在线监督训练方法，能够将 ESN 的混沌活动转变为指定的活动模式。本文提出一种基于递归最小二乘的复合 FORCE 学习方法，用于训练初始活动自发混沌的 ESN；其中采用以动态回归器扩展和记忆数据利用为特征的复合学习技术来增强参数收敛。所提方法被应用于一个预测由 Mackey-Glass 系统生成的混沌时间序列的基准问题，数值结果表明，与现有方法相比，该方法显著提升了学习和预测性能。
英文摘要：
Echo state network (ESN), a kind of recurrent neural networks, consists of a fixed reservoir in which neurons are connected randomly and recursively and obtains the desired output only by training output connection weights. First-order reduced and controlled error (FORCE) learning is an online supervised training approach that can change the chaotic activity of ESNs into specified activity patterns. This paper proposes a composite FORCE learning method based on recursive least squares to train ESNs whose initial activity is spontaneously chaotic, where a composite learning technique featured by dynamic regressor extension and memory data exploitation is applied to enhance parameter convergence. The proposed method is applied to a benchmark problem about predicting chaotic time series generated by the Mackey-Glass system, and numerical results have shown that it significantly improves learning and prediction performances compared with existing methods.
University of Cambridge（剑桥大学）；The Alan Turing Institute（阿兰·图灵研究所）；Technical University of Munich（慕尼黑工业大学）
An approach to the time-accurate prediction of chaotic solutions is by learning temporal patterns from data. Echo State Networks (ESNs), which are a class of Reservoir Computing, can accurately predict the chaotic dynamics well beyond the predictability time. Existing studies, however, also showed that small changes in the hyperparameters may markedly affect the network's performance. The aim of this paper is to assess and improve the robustness of Echo State Networks for the time-accurate prediction of chaotic solutions. The goal is three-fold. First, we investigate the robustness of routinel…
Artificial Intelligence Center（人工智能中心）；Czech Technical University（捷克技术大学）；Niels Bohr Institute（尼尔斯·玻尔研究所）；University of Copenhagen（哥本哈根大学）
This work builds an automated anomaly detection method for chaotic time series, and more concretely for turbulent, high-dimensional, ocean simulations. We solve this task by extending the Echo State Network by spatially aware input maps, such as convolutions, gradients, cosine transforms, et cetera, as well as a spatially aware loss function. The spatial ESN is used to create predictions which reduce the detection problem to thresholding of the prediction error. We benchmark our detection framework on different tasks of increasing difficulty to show the generality of the framework before apply…
Technical University of Munich（慕尼黑工业大学）；Institute for Advanced Study, Technical University of Munich（慕尼黑工业大学高等研究院）；University of Cambridge（剑桥大学）
We propose a physics-informed Echo State Network (ESN) to predict the evolution of chaotic systems. Compared to conventional ESNs, the physics-informed ESNs are trained to solve supervised learning tasks while ensuring that their predictions do not violate physical laws. This is achieved by introducing an additional loss function during the training of the ESNs, which penalizes non-physical predictions without the need of any additional training data. This approach is demonstrated on a chaotic Lorenz system, where the physics-informed ESNs improve the predictability horizon by about two Lyapun…
Technical University of Munich（慕尼黑工业大学）；University of Cambridge（剑桥大学）
We extend the Physics-Informed Echo State Network (PI-ESN) framework to reconstruct the evolution of an unmeasured state (hidden state) in a chaotic system. The PI-ESN is trained by using (i) data, which contains no information on the unmeasured state, and (ii) the physical equations of a prototypical chaotic dynamical system. Non-noisy and noisy datasets are considered. First, it is shown that the PI-ESN can accurately reconstruct the unmeasured state. Second, the reconstruction is shown to be robust with respect to noisy data, which means that the PI-ESN acts as a denoiser. This paper opens …
Minimal Deterministic Echo State Networks Outperform Random Reservoirs in Learning Chaotic Dynamics
Center for Scalable Data Analytics；Artificial Intelligence (ScaDS.AI), Leipzig University, Leipzig, Germany；Institute for Earth System Science \& Remote Sensing, Leipzig University, Leipzig, Germany
University of Cambridge（剑桥大学）；The Alan Turing Institute（艾伦·图灵研究所）；Imperial College London（帝国理工学院）；Technical University of Munich（慕尼黑工业大学）
We propose the Automatic-differentiated Physics-Informed Echo State Network (API-ESN). The network is constrained by the physical equations through the reservoir's exact time-derivative, which is computed by automatic differentiation. As compared to the original Physics-Informed Echo State Network, the accuracy of the time-derivative is increased by up to seven orders of magnitude. This increased accuracy is key in chaotic dynamical systems, where errors grows exponentially in time. The network is showcased in the reconstruction of unmeasured (hidden) states of a chaotic system. The API-ESN el…
We report an improvement to the conventional Echo State Network (ESN) across three benchmark chaotic time-series prediction tasks using fruit fly connectome data alone. We also investigate the impact of key connectome-derived structural features on prediction performance -- uniquely bridging neurobiological structure and machine learning function; and find that both increasing the global average clustering coefficient and modifying the position of weights -- by permuting their synapse-synapse partners -- can lead to increased model variance and (in some cases) degraded performance. In all we c…
Can connectome-derived constraints inform computation? In this paper we investigate the contribution of a fruit fly connectome's topology on the performance of an Echo State Network (ESN) -- a subset of Reservoir Computing which is state of the art in chaotic time series prediction. Specifically, we replace the reservoir layer of a classical ESN -- normally a fixed, random graph represented as a 2-d matrix -- with a particular (female) fruit fly connectome-derived connectivity matrix. We refer to this experimental class of models (with connectome-derived reservoirs) as "Fruit Fly ESNs" (FFESNs…
Echo State Networks for Bitcoin Time Series Prediction
DFKI, Saarland Informatics Campus, Germany（德国达姆施塔特研究所，萨尔兰信息技术校园）；Politecnico di Milano, Italy（意大利米兰理工学院）；University of Stirling, UK（英国斯特林大学）；National Institute of Informatics, Japan（日本信息学研究院）
Technical University of Munich（慕尼黑工业大学）；University of Cambridge（剑桥大学）
We propose a physics-informed Echo State Network (ESN) to predict the evolution of chaotic systems. Compared to conventional ESNs, the physics-informed ESNs are trained to solve supervised learning tasks while ensuring that their predictions do not violate physical laws. This is achieved by introducing an additional loss function during the training, which is based on the system's governing equations. The additional loss function penalizes non-physical predictions without the need of any additional training data. This approach is demonstrated on a chaotic Lorenz system and a truncation of the …
In the framework of physical reservoir computing (RC), machine learning algorithms designed for digital computers are executed using analog computer-like nonlinear physical systems that can provide energy-efficient computational power for predicting time-dependent quantities that can be found using nonlinear differential equations. We suggest a bubble-based RC (BRC) system that combines the nonlinearity of an acoustic response of a cluster of oscillating gas bubbles in water with a standard Echo State Network (ESN) algorithm that is well-suited to forecast chaotic time series. We confirm the p…
University of Cambridge（剑桥大学）；Technical University of Munich（慕尼黑工业大学）
We propose a physics-informed machine learning method to predict the time average of a chaotic attractor. The method is based on the hybrid echo state network (hESN). We assume that the system is ergodic, so the time average is equal to the ergodic average. Compared to conventional echo state networks (ESN) (purely data-driven), the hESN uses additional information from an incomplete, or imperfect, physical model. We evaluate the performance of the hESN and compare it to that of an ESN. This approach is demonstrated on a chaotic time-delayed thermoacoustic system, where the inclusion of a phys…
University of Cambridge（剑桥大学）；Imperial College London（帝国理工学院）；The Alan Turing Institute（艾伦·图灵研究所）；TU Munich（慕尼黑工业大学）
We develop a versatile optimization method, which finds the design parameters that minimize time-averaged acoustic cost functionals. The method is gradient-free, model-informed, and data-driven with reservoir computing based on echo state networks. First, we analyse the predictive capabilities of echo state networks both in the short- and long-time prediction of the dynamics. We find that both fully data-driven and model-informed architectures learn the chaotic acoustic dynamics, both time-accurately and statistically. Informing the training with a physical reduced-order model with one acousti…
University of Tromsø - The Arctic University of Norway（特罗姆瑟大学-挪威北极大学）
In this paper we introduce a new framework to train an Echo State Network to predict real valued time-series. The method consists in projecting the output of the internal layer of the network on a space with lower dimensionality, before training the output layer to learn the target task. Notably, we enforce a regularization constraint that leads to better generalization capabilities. We evaluate the performances of our approach on several benchmark tests, using different techniques to train the readout of the network, achieving superior predictive performance when using the proposed framework.…
Laboratoire Kastler Brossel（卡斯特勒布罗塞尔实验室）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）；Laboratoire de Physique Statistique（统计物理实验室）；LightOn（LightOn公司）
Echo-State Networks and Reservoir Computing have been studied for more than a decade. They provide a simpler yet powerful alternative to Recurrent Neural Networks, every internal weight is fixed and only the last linear layer is trained. They involve many multiplications by dense random matrices. Very large networks are difficult to obtain, as the complexity scales quadratically both in time and memory. Here, we present a novel optical implementation of Echo-State Networks using light-scattering media and a Digital Micromirror Device. As a proof of concept, binary networks have been successful…
Università della Svizzera Italiana（瑞士意大利语区大学）；Politecnico di Milano（米兰理工大学）；University of Manitoba（曼尼托巴大学）；University of Exeter（埃克塞特大学）
Among the various architectures of Recurrent Neural Networks, Echo State Networks (ESNs) emerged due to their simplified and inexpensive training procedure. These networks are known to be sensitive to the setting of hyper-parameters, which critically affect their behaviour. Results show that their performance is usually maximized in a narrow region of hyper-parameter space called edge of chaos. Finding such a region requires searching in hyper-parameter space in a sensible way: hyper-parameter configurations marginally outside such a region might yield networks exhibiting fully developed chaos…
Faculty of Electrical Engineering and Computer Science（电气工程与计算机科学学院）；VŠB-Technical University of Ostrava（俄斯特拉发技术大学）；INRIA Rennes – Bretagne Atlantique（法国国家信息与自动化研究所雷恩-布列塔尼大西洋中心）
The Echo State Network (ESN) is a class of Recurrent Neural Network with a large number of hidden-hidden weights (in the so-called reservoir). Canonical ESN and its variations have recently received significant attention due to their remarkable success in the modeling of non-linear dynamical systems. The reservoir is randomly connected with fixed weights that don't change in the learning process. Only the weights from reservoir to output are trained. Since the reservoir is fixed during the training procedure, we may wonder if the computational power of the recurrent structure is fully harnesse…
University of Cambridge（剑桥大学）；Imperial College London（帝国理工学院）；The Alan Turing Institute（艾伦·图灵研究所）；TU Munich（慕尼黑工业大学）
An extreme event is a sudden and violent change in the state of a nonlinear system. In fluid dynamics, extreme events can have adverse effects on the system's optimal design and operability, which calls for accurate methods for their prediction and control. In this paper, we propose a data-driven methodology for the prediction and control of extreme events in a chaotic shear flow. The approach is based on echo state networks, which are a type of reservoir computing that learn temporal correlations within a time-dependent dataset. The objective is five-fold. First, we exploit ad-hoc metrics fro…
University of Cambridge（剑桥大学）；Delft University of Technology（代尔夫特理工大学）；Imperial College London（帝国理工学院）；The Alan Turing Institute（艾伦·图灵研究所）
The spatiotemporal dynamics of turbulent flows is chaotic and difficult to predict. This makes the design of accurate and stable reduced-order models challenging. The overarching objective of this paper is to propose a nonlinear decomposition of the turbulent state for a reduced-order representation of the dynamics. We divide the turbulent flow into a spatial problem and a temporal problem. First, we compute the latent space, which is the manifold onto which the turbulent dynamics live (i.e., it is a numerical approximation of the turbulent attractor). The latent space is found by a series of …
In this paper, the performance of three deep learning methods for predicting short-term evolution and for reproducing the long-term statistics of a multi-scale spatio-temporal Lorenz 96 system is examined. The methods are: echo state network (a type of reservoir computing, RC-ESN), deep feed-forward artificial neural network (ANN), and recurrent neural network with long short-term memory (RNN-LSTM). This Lorenz 96 system has three tiers of nonlinearly interacting variables representing slow/large-scale ($X$), intermediate ($Y$), and fast/small-scale ($Z$) processes. For training or testing, on…
Prediction of Nonlinear Oscillations in a Jumping Quarter-Car Model Using Reservoir Computing
Tokyo University of Agriculture and Technology（东京农工大学）；Amity University Haryana - Gurugram（Amity大学哈里亚纳邦-古鲁格拉姆校区）；Nagoya Institute of Technology（名古屋工业大学）；Central University of Rajasthan（中央拉贾斯坦大学）
Adaptive Nonlinear Vector Autoregression: Robust Forecasting for Noisy Chaotic Time Series
Department of Mathematics, Pusan National University, Republic of Korea（釜山国立大学数学系）；Pusan National University, Republic of Korea（釜山国立大学）；Industrial Mathematics Center, Pusan National University, Republic of Korea（釜山国立大学工业数学中心）；Humanoid Olfactory Display Center, Pusan National University, Republic of Korea（釜山国立大学人形嗅觉显示中心）；Department of Mathematics, Yonsei University, Republic of Korea（成均馆大学数学系）；Innovation Center for MathScience Research&Education, Pusan National University, Republic of Korea（釜山国立大学数学科学研究与教育创新中心）；Marine Natural Disaster Research Department, Korea Institute of Ocean Science and Technology, Republic of Korea（韩国海洋科学与技术研究院海洋自然灾害研究部）；Institute for Future Earth, Pusan National University, Republic of Korea（未来地球研究所）
Motion-robust 2D Radial Turbo Spin Echo (RADTSE) pulse sequence can provide a high-resolution composite image, T2-weighted images at multiple echo times (TEs), and a quantitative T2 map, all from a single k-space acquisition. In this work, we use a deep-learning convolutional neural network (CNN) for the segmentation of liver in abdominal RADTSE images. A modified UNET architecture with generalized dice loss objective function was implemented. Three 2D CNNs were trained, one for each image type obtained from the RADTSE sequence. On evaluating the performance of the CNNs on the validation set, …
University of Waterloo（滑铁卢大学）；ATS Automation（ATS自动化公司）
Causal discovery between collections of time-series data can help diagnose causes of symptoms and hopefully prevent faults before they occur. However, reliable causal discovery can be very challenging, especially when the data acquisition rate varies (i.e., non-uniform data sampling), or in the presence of missing data points (e.g., sparse data sampling). To address these issues, we proposed a new system comprised of two parts, the first part fills missing data with a Gaussian Process Regression, and the second part leverages an Echo State Network, which is a type of reservoir computer (i.e., …
