Echo state networks are computationally lightweight reservoir models inspired by the random projections observed in cortical circuitry. As interest in reservoir computing has grown, networks have become deeper and more intricate. While these networks are increasingly applied to nontrivial forecasting tasks, there is a need for comprehensive performance analysis of deep reservoirs. In this work, we study the influence of partitioning neurons given a budget and the effect of parallel reservoir pathways across different datasets exhibiting multi-scale and nonlinear dynamics.
University of Cambridge（剑桥大学）；Delft University of Technology（代尔夫特理工大学）；Imperial College London（帝国理工学院）；The Alan Turing Institute（艾伦·图灵研究所）
The spatiotemporal dynamics of turbulent flows is chaotic and difficult to predict. This makes the design of accurate and stable reduced-order models challenging. The overarching objective of this paper is to propose a nonlinear decomposition of the turbulent state for a reduced-order representation of the dynamics. We divide the turbulent flow into a spatial problem and a temporal problem. First, we compute the latent space, which is the manifold onto which the turbulent dynamics live (i.e., it is a numerical approximation of the turbulent attractor). The latent space is found by a series of …
South China University of Technology（华南理工大学）；University of California, San Diego（加州大学圣迭戈分校）
As an efficient recurrent neural network (RNN) model, reservoir computing (RC) models, such as Echo State Networks, have attracted widespread attention in the last decade. However, while they have had great success with time series data [1], [2], many time series have a multiscale structure, which a single-hidden-layer RC model may have difficulty capturing. In this paper, we propose a novel hierarchical reservoir computing framework we call Deep Echo State Networks (Deep-ESNs). The most distinctive feature of a Deep-ESN is its ability to deal with time series through hierarchical projections.…
Air Force Research Laboratory Information Directorate（空军研究实验室信息局）
This work introduces a tensor-based method to perform supervised classification on spatiotemporal data processed in an echo state network. Typically when performing supervised classification tasks on data processed in an echo state network, the entire collection of hidden layer node states from the training dataset is shaped into a matrix, allowing one to use standard linear algebra techniques to train the output layer. However, the collection of hidden layer states is multidimensional in nature, and representing it as a matrix may lead to undesirable numerical conditions or loss of spatial an…
Technion - Israel Institute of Technology（以色列理工学院）；Duke University（杜克大学）
Multivariate time-series have become abundant in recent years, as many data-acquisition systems record information through multiple sensors simultaneously. In this paper, we assume the variables pertain to some geometry and present an operator-based approach for spatiotemporal analysis. Our approach combines three components that are often considered separately: (i) manifold learning for building operators representing the geometry of the variables, (ii) Riemannian geometry of symmetric positive-definite matrices for multiscale composition of operators corresponding to different time samples, …
Modeling Biological Multifunctionality with Echo State Networks
Department of Informatics & Computer Engineering, Internet Computing and Cloud Technologies Lab, University of West Attica（信息与计算机工程系，互联网计算与云计算技术实验室，西阿提卡大学）
本研究构建结合可激发系统动力学与扩散过程的三维多组分反应扩散模型，通过数值求解生成生物时空行为时序数据，用以训练Echo State Network（ESN），验证了数据驱动的多功能ESN模拟生物动力学的可行性与有效性。
Natural spatiotemporal processes can be highly non-stationary in many ways, e.g. the low-level non-stationarity such as spatial correlations or temporal dependencies of local pixel values; and the high-level variations such as the accumulation, deformation or dissipation of radar echoes in precipitation forecasting. From Cramer's Decomposition, any non-stationary process can be decomposed into deterministic, time-variant polynomials, plus a zero-mean stochastic term. By applying differencing operations appropriately, we may turn time-variant polynomials into a constant, making the deterministi…
Jupiter Intelligence；University of Missouri（密苏里大学）
Long-lead forecasting for spatio-temporal systems can often entail complex nonlinear dynamics that are difficult to specify it a priori. Current statistical methodologies for modeling these processes are often highly parameterized and thus, challenging to implement from a computational perspective. One potential parsimonious solution to this problem is a method from the dynamical systems and engineering literature referred to as an echo state network (ESN). ESN models use so-called {\it reservoir computing} to efficiently compute recurrent neural network (RNN) forecasts. Moreover, so-called "d…
MIT Media Lab（麻省理工学院媒体实验室）；Philips Research（飞利浦研究院）；IBM Research（IBM研究院）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
Event detection has been one of the most important research topics in social media analysis. Most of the traditional approaches detect events based on fixed temporal and spatial resolutions, while in reality events of different scales usually occur simultaneously, namely, they span different intervals in time and space. In this paper, we propose a novel approach towards multiscale event detection using social media data, which takes into account different temporal and spatial scales of events in the data. Specifically, we explore the properties of the wavelet transform, which is a well-develop…
AFD-STA: Adaptive Filtering Denoising with Spatiotemporal Attention for Chaotic System Prediction
University of Minnesota Twin Cities（明尼苏达大学双城分校）；School of Statistics and Mathematics, Shandong University of Finance and Economics（山东财经大学统计与数学学院）；Jinan Fengdi Intelligent Electronics Co., Ltd.（济南风迪智能电子有限公司）；Shandong Zhike Intelligence Computing Co., Ltd.（山东智科智能计算有限公司）；School of Mechanical,Electrical and Information Engineering, Shandong University（山东大学机械电子信息工程学院）
ASTEROID: A Spatiotemporal Information Transformer for Forecasting Multi-Step Time Series of Molecular Dynamics
Department of Medicinal Chemistry, School of Pharmaceutical Sciences, Fudan University（药学院药物化学系，复旦大学）；School of Mathematical Sciences and School of AI, Shanghai Jiao Tong University（数学科学学院和人工智能学院，上海交通大学）
A spatial curve dynamical model framework is adopted for functional prediction of counts in a spatiotemporal log-Gaussian Cox process model. Our spatial functional estimation approach handles both wavelet-based heterogeneity analysis in time, and spectral analysis in space. Specifically, model fitting is achieved by minimising the information divergence or relative entropy between the multiscale model underlying the data and the corresponding candidates in the spatial spectral domain. A simulation study is carried out within the family of log-Gaussian Spatial Autoregressive l2-valued processes…
A spatiotemporal calibration and resolution refinement model was fitted to calibrate nitrogen dioxide (NO$_2$) concentration estimates from the Community Multiscale Air Quality (CMAQ) model, using two sources of observed data on NO$_2$ that differed in their spatial and temporal resolutions. To refine the spatial resolution of the CMAQ model estimates, we leveraged information using additional local covariates including total traffic volume within 2 km, population density, elevation, and land use characteristics. Predictions from this model greatly improved the bias in the CMAQ estimates, as o…
Since December 2019, the world has been witnessing the gigantic effect of an unprecedented global pandemic called Severe Acute Respiratory Syndrome Coronavirus (SARS-CoV-2) - COVID-19. So far, 38,619,674 confirmed cases and 1,093,522 confirmed deaths due to COVID-19 have been reported. In the United States (US), the cases and deaths are recorded as 7,833,851 and 215,199. Several timely researches have discussed the local and global effects of the confounding factors on COVID-19 casualties in the US. However, most of these studies considered little about the time varying associations between an…
Deconstructing Recurrence, Attention, and Gating: Investigating the transferability of Transformers and Gated Recurrent Neural Networks in forecasting of dynamical systems
School of Engineering and Applied Sciences, Harvard University（哈佛大学工程与应用科学学院）；Institute of Structural Engineering, ETH Zurich（苏黎世联邦理工学院结构工程研究所）
Natural disasters caused by heavy rainfall often cost huge loss of life and property. To avoid it, the task of precipitation nowcasting is imminent. To solve the problem, increasingly deep learning methods are proposed to forecast future radar echo images and then the predicted maps have converted the distribution of rainfall. The prevailing spatiotemporal sequence prediction methods apply ConvRNN structure which combines the Convolution and Recurrent neural network. Although improvements based on ConvRNN achieve remarkable success, these methods ignore capturing both local and global spatial …
