Machine learning approaches have recently been leveraged as a substitute or an aid for physical/mathematical modeling approaches to dynamical systems. To develop an efficient machine learning method dedicated to modeling and prediction of multiscale dynamics, we propose a reservoir computing (RC) model with diverse timescales by using a recurrent network of heterogeneous leaky integrator (LI) neurons. We evaluate computational performance of the proposed model in two time series prediction tasks related to four chaotic fast-slow dynamical systems. In a one-step-ahead prediction task where input data are provided only from the fast subsystem, we show that the proposed model yields better performance than the standard RC model with identical LI neurons. Our analysis reveals that the timescale required for producing each component of target multiscale dynamics is appropriately and flexibly selected from the reservoir dynamics by model training. In a long-term prediction task, we demonstrate that a closed-loop version of the proposed model can achieve longer-term predictions compared to the counterpart with identical LI neurons depending on the hyperparameter setting.
Echo state networks are computationally lightweight reservoir models inspired by the random projections observed in cortical circuitry. As interest in reservoir computing has grown, networks have become deeper and more intricate. While these networks are increasingly applied to nontrivial forecasting tasks, there is a need for comprehensive performance analysis of deep reservoirs. In this work, we study the influence of partitioning neurons given a budget and the effect of parallel reservoir pathways across different datasets exhibiting multi-scale and nonlinear dynamics.
Efficient Time Series Clustering from Multiscale Reservoir Dynamics with Granular-Ball Anchoring Graph Optimization
Chongqing Key Laboratory of Computational Intelligence, Key Laboratory of Cyberspace Big Data Intelligent Security, Ministry of Education, Sichuan-Chongqing Co-construction Key Laboratory of Digital Economy Intelligence and Key Laboratory of Big Data Intelligent Computing, College of Computer Science and Technology, Chongqing University of Posts and Telecommunications（重庆邮电大学计算机科学与技术学院，计算智能重庆市重点实验室，网络空间大数据智能安全教育部重点实验室，川渝共建数字经济智能重点实验室，大数据智能计算重点实验室）；Chongqing Ant Consumer Finance Co,. Ltd , Ant Group（蚂蚁集团，重庆蚂蚁消费金融有限公司）
Università “Sapienza”（罗马第一大学）；Istituto dei Sistemi Complessi, CNR（意大利国家研究委员会复杂系统研究所）
We scrutinize the use of machine learning, based on reservoir computing, to build data-driven effective models of multiscale chaotic systems. We show that, for a wide scale separation, machine learning generates effective models akin to those obtained using multiscale asymptotic techniques and, remarkably, remains effective in predictability also when the scale separation is reduced. We also show that predictability can be improved by hybridizing the reservoir with an imperfect model.
University of Sheffield（谢菲尔德大学）；Italian Institute of Health（意大利卫生研究院）
Echo state networks (ESNs) are a powerful form of reservoir computing that only require training of linear output weights whilst the internal reservoir is formed of fixed randomly connected neurons. With a correctly scaled connectivity matrix, the neurons' activity exhibits the echo-state property and responds to the input dynamics with certain timescales. Tuning the timescales of the network can be necessary for treating certain tasks, and some environments require multiple timescales for an efficient representation. Here we explore the timescales in hierarchical ESNs, where the reservoir is …
Prediction of Unobserved Bifurcation by Unsupervised Extraction of Slowly Time-Varying System Parameter Dynamics from Time Series Using Reservoir Computing
Faculty of Health Data science, Juntendo University（顺天堂大学健康数据科学学部）；The School of Systems Information Science, Future University Hakodate（函馆未来大学系统信息科学学院）
Representing the reservoir as a network of discrete compartments with neighbor and non-neighbor connections is a fast, yet accurate method for analyzing oil and gas reservoirs. Automatic and rapid detection of coarse-scale compartments with distinct static and dynamic properties is an integral part of such high-level reservoir analysis. In this work, we present a hybrid framework specific to reservoir analysis for an automatic detection of clusters in space using spatial and temporal field data, coupled with a physics-based multiscale modeling approach. In this work a novel hybrid approach is …
Technische Universität Berlin（柏林工业大学）；Humboldt-Universität zu Berlin（柏林洪堡大学）；Instituto de Física Interdisciplinar y Sistemas Complejos, IFISC (CSIC-UIB)（跨学科物理与复杂系统研究所（IFISC，西班牙国家研究委员会-巴利阿里群岛大学））
The time-delay-based reservoir computing setup has seen tremendous success in both experiment and simulation. It allows for the construction of large neuromorphic computing systems with only few components. However, until now the interplay of the different timescales has not been investigated thoroughly. In this manuscript, we investigate the effects of a mismatch between the time-delay and the clock cycle for a general model. Typically, these two time scales are considered to be equal. Here we show that the case of equal or resonant time-delay and clock cycle could be actively detrimental and…
Nonlinear differential equations rarely admit closed-form solutions, thus requiring numerical time-stepping algorithms to approximate solutions. Further, many systems characterized by multiscale physics exhibit dynamics over a vast range of timescales, making numerical integration computationally expensive due to numerical stiffness. In this work, we develop a hierarchy of deep neural network time-steppers to approximate the flow map of the dynamical system over a disparate range of time-scales. The resulting model is purely data-driven and leverages features of the multiscale dynamics, enabli…
The mushroom body is the key network for the representation of learned olfactory stimuli in Drosophila and insects. The sparse activity of Kenyon cells, the principal neurons in the mushroom body, plays a key role in the learned classification of different odours. In the specific case of the fruit fly, the sparseness of the network is enforced by an inhibitory feedback neuron called APL, and by an intrinsic high firing threshold of the Kenyon cells. In this work we took inspiration from the fruit fly brain to formulate a novel machine learning algorithm that is able to optimize the sparsity le…
In this paper, the performance of three deep learning methods for predicting short-term evolution and for reproducing the long-term statistics of a multi-scale spatio-temporal Lorenz 96 system is examined. The methods are: echo state network (a type of reservoir computing, RC-ESN), deep feed-forward artificial neural network (ANN), and recurrent neural network with long short-term memory (RNN-LSTM). This Lorenz 96 system has three tiers of nonlinearly interacting variables representing slow/large-scale ($X$), intermediate ($Y$), and fast/small-scale ($Z$) processes. For training or testing, on…
Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inference
Department of Electrical and Computer Engineering（电气与计算机工程系）；University of Southern California（南加州大学）；Department of Computer Science（计算机科学系）；Department of Biomedical Engineering（生物医学工程系）；Neuroscience Graduate Program（神经科学研究生项目）
University of Victoria（维多利亚大学）；University of Washington（华盛顿大学）
Multiscale phenomena that evolve on multiple distinct timescales are prevalent throughout the sciences. It is often the case that the governing equations of the persistent and approximately periodic fast scales are prescribed, while the emergent slow scale evolution is unknown. Yet the course-grained, slow scale dynamics is often of greatest interest in practice. In this work we present an accurate and efficient method for extracting the slow timescale dynamics from signals exhibiting multiple timescales that are amenable to averaging. The method relies on tracking the signal at evenly-spaced …
Language exhibits structure at different scales, ranging from subwords to words, sentences, paragraphs, and documents. To what extent do deep models capture information at these scales, and can we force them to better capture structure across this hierarchy? We approach this question by focusing on individual neurons, analyzing the behavior of their activations at different timescales. We show that signal processing provides a natural framework for separating structure across scales, enabling us to 1) disentangle scale-specific information in existing embeddings and 2) train models to learn mo…
Machine Learning-driven Multiscale MD Workflows: The Mini-MuMMI Experience
Center for Applied Scientific Computing, Lawrence Livermore National Laboratory（应用科学计算中心，劳伦斯利弗莫尔国家实验室）；Physical and Life Sciences Directorate, Lawrence Livermore National Laboratory（物理与生命科学 Directorate，劳伦斯利弗莫尔国家实验室）；RAS Initiative, The Cancer Research Technology Program, Frederick National Laboratory（RAS计划，癌症研究技术计划，弗雷德里克国家实验室）；Frederick National Laboratory（弗雷德里克国家实验室）
Machine learning enhanced data assimilation framework for multiscale carbonate rock characterization
Institute of GeoEnergy Engineering, Heriot-Watt University（赫瑞-瓦特大学地质能源工程研究所）；Department of Geoscience & Engineering, Technical University of Delft（代尔夫特理工大学地球科学与工程系）；PETRONAS Research Sdn Bhd（马来西亚国家石油公司研究有限公司）
