Classical methods of solving spatiotemporal dynamical systems include statistical approaches such as autoregressive integrated moving average, which assume linear and stationary relationships between systems' previous outputs. Development and implementation of linear methods are relatively simple, but they often do not capture non-linear relationships in the data. Thus, artificial neural networks (ANNs) are receiving attention from researchers in analyzing and forecasting dynamical systems. Recurrent neural networks (RNN), derived from feed-forward ANNs, use internal memory to process variable-length sequences of inputs. This allows RNNs to applicable for finding solutions for a vast variety of problems in spatiotemporal dynamical systems. Thus, in this paper, we utilize RNNs to treat some specific issues associated with dynamical systems. Specifically, we analyze the performance of RNNs applied to three tasks: reconstruction of correct Lorenz solutions for a system with a formulation error, reconstruction of corrupted collective motion trajectories, and forecasting of streamflow time series possessing spikes, representing three fields, namely, ordinary differential equations, collective motion, and hydrological modeling, respectively. We train and test RNNs uniquely in each task to demonstrate the broad applicability of RNNs in reconstruction and forecasting the dynamics of dynamical systems.
In contrast to existing works on stochastic averaging on finite intervals, we establish an averaging principle on the whole real axis, i.e. the so-called second Bogolyubov theorem, for semilinear stochastic ordinary differential equations in Hilbert space with Poisson stable (in particular, periodic, quasi-periodic, almost periodic, almost automorphic etc) coefficients. Under some appropriate conditions we prove that there exists a unique recurrent solution to the original equation, which possesses the same recurrence property as the coefficients, in a small neighborhood of the stationary solu…
The design of recurrent neural networks (RNNs) to accurately process sequential inputs with long-time dependencies is very challenging on account of the exploding and vanishing gradient problem. To overcome this, we propose a novel RNN architecture which is based on a structure preserving discretization of a Hamiltonian system of second-order ordinary differential equations that models networks of oscillators. The resulting RNN is fast, invertible (in time), memory efficient and we derive rigorous bounds on the hidden state gradients to prove the mitigation of the exploding and vanishing gradi…
Continuous time recurrent neural networks (CTRNN) are systems of coupled ordinary differential equations that are simple enough to be insightful for describing learning and computation, from both biological and machine learning viewpoints. We describe a direct constructive method of realising finite state input-dependent computations on an arbitrary directed graph. The constructed system has an excitable network attractor whose dynamics we illustrate with a number of examples. The resulting CTRNN has intermittent dynamics: trajectories spend long periods of time close to steady-state, with rap…
We introduce a novel approach to obtaining mathematically rigorous results on the global dynamics of ordinary differential equations. Motivated by models of regulatory networks, we construct a state transition graph from a piecewise affine ordinary differential equation. We use efficient graph algorithms to compute an associated Morse graph that codifies the recurrent and gradient-like dynamics. We prove that for 2-dimensional systems, the Morse graph defines a Morse decomposition for the dynamics of any smooth differential equation that is sufficiently close to the original piecewise affine o…
We study decidability and complexity questions related to a continuous analogue of the Skolem-Pisot problem concerning the zeros and nonnegativity of a linear recurrent sequence. In particular, we show that the continuous version of the nonnegativity problem is NP-hard in general and we show that the presence of a zero is decidable for several subcases, including instances of depth two or less, although the decidability in general is left open. The problems may also be stated as reachability problems related to real zeros of exponential polynomials or solutions to initial value problems of lin…
Recurrent neural networks (RNN) as used in machine learning are commonly formulated in discrete time, i.e. as recursive maps. This brings a lot of advantages for training models on data, e.g. for the purpose of time series prediction or dynamical systems identification, as powerful and efficient inference algorithms exist for discrete time systems and numerical integration of differential equations is not necessary. On the other hand, mathematical analysis of dynamical systems inferred from data is often more convenient and enables additional insights if these are formulated in continuous time…
ETH Zürich（苏黎世联邦理工学院）；University of Pittsburgh（匹兹堡大学）；ICSI（国际计算机科学研究所）；UC Berkeley（加州大学伯克利分校）
We propose a novel method called Long Expressive Memory (LEM) for learning long-term sequential dependencies. LEM is gradient-based, it can efficiently process sequential tasks with very long-term dependencies, and it is sufficiently expressive to be able to learn complicated input-output maps. To derive LEM, we consider a system of multiscale ordinary differential equations, as well as a suitable time-discretization of this system. For LEM, we derive rigorous bounds to show the mitigation of the exploding and vanishing gradients problem, a well-known challenge for gradient-based recurrent seq…
We analyze the chain recurrent set of skew product semiflows obtained from nonautonomous differential equations -- ordinary differential equations or semilinear parabolic differential equations. For many gradient-like dynamical systems, Morse-Smale dynamical systems e.g., the chain recurrent set contains only isolated equilibria. The structure in the asymptotically autonomous setting is richer but still close to to the structure of a Morse-Smale dynamical system. The main tool used in this paper is a nonautonomous flavour of Conley index theory developed by the author. We will see that for a c…
Northeastern University（东北大学）；The United States Military Academy（美国军事学院）；University of Missouri（密苏里大学）；Florida International University（佛罗里达国际大学）
Modeling biological dynamical systems is challenging due to the interdependence of different system components, some of which are not fully understood. To fill existing gaps in our ability to mechanistically model physiological systems, we propose to combine neural networks with physics-based models. Specifically, we demonstrate how we can approximate missing ordinary differential equations (ODEs) coupled with known ODEs using Bayesian filtering techniques to train the model parameters and simultaneously estimate dynamic state variables. As a study case we leverage a well-understood model for …
Decomposition of state spaces into dynamically different components is helpful for the understanding of dynamical behaviors of complex systems. A Conley type decomposition theorem is proved for nonautonomous dynamical systems defined on a non-compact but separable state space. Namely, the state space can be decomposed into a chain recurrent part and a gradient-like part. This result applies to both nonautonomous ordinary differential equations on Euclidean space (which is only locally compact), and nonautonomous partial differential equations on infinite dimensional function space (which is no…
We present and review an algorithmic and theoretical framework for improving neural network architecture design via momentum. As case studies, we consider how momentum can improve the architecture design for recurrent neural networks (RNNs), neural ordinary differential equations (ODEs), and transformers. We show that integrating momentum into neural network architectures has several remarkable theoretical and empirical benefits, including 1) integrating momentum into RNNs and neural ODEs can overcome the vanishing gradient issues in training RNNs and neural ODEs, resulting in effective learni…
The conjoining of dynamical systems and deep learning has become a topic of great interest. In particular, neural differential equations (NDEs) demonstrate that neural networks and differential equation are two sides of the same coin. Traditional parameterised differential equations are a special case. Many popular neural network architectures, such as residual networks and recurrent networks, are discretisations. NDEs are suitable for tackling generative problems, dynamical systems, and time series (particularly in physics, finance, ...) and are thus of interest to both modern machine learnin…
