首页/论文/2001.07694

arXiv 2001.076942020-01math.DS

The Echo Index and multistability in input-driven recurrent neural networks

Andrea Ceni, Peter Ashwin, Lorenzo Livi, Claire Postlethwaite

备注：: Revised version, 43 pages, 6 figures
DOI：: 10.1016/j.physd.2020.132609

英文摘要：

A recurrent neural network (RNN) possesses the echo state property (ESP) if, for a given input sequence, it ``forgets'' any internal states of the driven (nonautonomous) system and asymptotically follows a unique, possibly complex trajectory. The lack of ESP is conventionally understood as a lack of reliable behaviour in RNNs. Here, we show that RNNs can reliably perform computations under a more general principle that accounts only for their local behaviour in phase space. To this end, we formulate a generalisation of the ESP and introduce an echo index to characterise the number of simultaneously stable responses of a driven RNN. We show that it is possible for the echo index to change with inputs, highlighting a potential source of computational errors in RNNs due to characteristics of the inputs driving the dynamics.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

往事的回响：关于衰减记忆与回声状态的统一视角

Echoes of the past: A unified perspective on fading memory and echo states

Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore（数学科学系，物理与数学科学学院，南洋理工大学，新加坡）；Nanyang Technological University, Singapore（南洋理工大学，新加坡）

本文统一了RNN中衰减记忆、回声状态等概念，推导其关系并提供新证明，以深化对RNN时序处理能力的理解。

2025-12-12cs.LGmath.DSstat.ML

回声指数的转换及其对输入重复的依赖

Transitions in echo index and dependence on input repetitions

University of Exeter（埃克塞特大学）；University of Pisa（比萨大学）

本文研究了非自治动力系统回声指数对有限态遍历外部输入参数的依赖性，发现映射重复次数和强迫幅值决定回声指数变化，并将理论发现应用于RNN计算框架。

2023-09-09cs.AIcs.LGmath.DS

Using Echo State Networks to Approximate Value Functions for Control

University of Bath（巴斯大学）

An Echo State Network (ESN) is a type of single-layer recurrent neural network with randomly-chosen internal weights and a trainable output layer. We prove under mild conditions that a sufficiently large Echo State Network can approximate the value function of a broad class of stochastic and deterministic control problems. Such control problems are generally non-Markovian. We describe how the ESN can form the basis for novel and computationally efficient reinforcement learning algorithms in a non-Markovian framework. We demonstrate this theory with two examples. In the first, we use an ESN to …

2021-06-25cs.AImath.DS

A Step Towards Uncovering The Structure of Multistable Neural Networks

We study how the connectivity within a recurrent neural network determines and is determined by the multistable solutions of network activity. To gain analytic tractability we let neural activation be a non-smooth Heaviside step function. This nonlinearity partitions the phase space into regions with different, yet linear dynamics. In each region either a stable equilibrium state exists, or network activity flows to outside of the region. The stable states are identified by their semipositivity constraints on the synaptic weight matrix. The restrictions can be separated by their effects on the…

2023-03-07cond-mat.dis-nncs.NEmath.DS

Embedding and Approximation Theorems for Echo State Networks

Echo State Networks (ESNs) are a class of single layer recurrent neural networks that have enjoyed recent attention. In this paper we prove that a suitable ESN, trained on a series of measurements of an invertible dynamical system, induces a C1 map from the dynamical system's phase space to the ESN's reservoir space. We call this the Echo State Map. We then prove that the Echo State Map is generically an embedding with positive probability. Under additional mild assumptions, we further conjecture that the Echo State Map is almost surely an embedding. For sufficiently large, and specially struc…

2020-05-18math.DSnlin.CD

Echo State Networks trained by Tikhonov least squares are L2(μ) approximators of ergodic dynamical systems

University of Bath（巴斯大学）

Echo State Networks (ESNs) are a class of single-layer recurrent neural networks with randomly generated internal weights, and a single layer of tuneable outer weights, which are usually trained by regularised linear least squares regression. Remarkably, ESNs still enjoy the universal approximation property despite the training procedure being entirely linear. In this paper, we prove that an ESN trained on a sequence of observations from an ergodic dynamical system (with invariant measure $μ$) using Tikhonov least squares regression against a set of targets, will approximate the target functio…

2021-02-18cs.LGmath.DSstat.ML

更多相关论文收起更多论文

Multiplex visibility graphs to investigate recurrent neural networks dynamics

A recurrent neural network (RNN) is a universal approximator of dynamical systems, whose performance often depends on sensitive hyperparameters. Tuning of such hyperparameters may be difficult and, typically, based on a trial-and-error approach. In this work, we adopt a graph-based framework to interpret and characterize the internal RNN dynamics. Through this insight, we are able to design a principled unsupervised method to derive configurations with maximized performances, in terms of prediction error and memory capacity. In particular, we propose to model time series of neurons activations…

2017-01-20cs.NEmath.DS

用于可控动力系统建模的物理信息回声状态网络

Physics-Informed Echo State Networks for Modeling Controllable Dynamical Systems

Federal University of Santa Catarina（圣卡塔琳娜联邦大学）

本文将物理信息回声状态网络扩展至带外部输入的可控非线性系统，结合自适应平衡损失，在数据稀缺时显著降低过拟合和测试误差，并提升模型预测控制性能。

2025-02-04cs.LGcs.NEmath.DS

On the difficulty of learning chaotic dynamics with RNNs

Central Institute of Mental Health（中央心理健康研究所）；Heidelberg University（海德堡大学）

Recurrent neural networks (RNNs) are wide-spread machine learning tools for modeling sequential and time series data. They are notoriously hard to train because their loss gradients backpropagated in time tend to saturate or diverge during training. This is known as the exploding and vanishing gradient problem. Previous solutions to this issue either built on rather complicated, purpose-engineered architectures with gated memory buffers, or - more recently - imposed constraints that ensure convergence to a fixed point or restrict (the eigenspectrum of) the recurrence matrix. Such constraints, …

2022-10-06cs.LGmath.DSstat.ML

储层计算用于有限数据下的系统辨识与预测控制

Reservoir computing for system identification and predictive control with limited data

University of Washington（华盛顿大学）

本文评估递归神经网络变体在有限数据下学习系统动力学并作为模型预测控制替代模型的能力，发现回声状态网络在计算复杂度、预测时长和控制成本上均具优势。

2024-10-23cs.LGcs.SYeess.SY