Training dynamic models, such as neural ODEs, on long trajectories is a hard problem that requires using various tricks, such as trajectory splitting, to make model training work in practice. These methods are often heuristics with poor theoretical justifications, and require iterative manual tuning. We propose a principled multiple shooting technique for neural ODEs that splits the trajectories into manageable short segments, which are optimised in parallel, while ensuring probabilistic control on continuity over consecutive segments. We derive variational inference for our shooting-based latent neural ODE models and propose amortized encodings of irregularly sampled trajectories with a transformer-based recognition network with temporal attention and relative positional encoding. We demonstrate efficient and stable training, and state-of-the-art performance on multiple large-scale benchmark datasets.
Aalto University（阿尔托大学）；University of Tübingen（蒂宾根大学）
Recent machine learning advances have proposed black-box estimation of unknown continuous-time system dynamics directly from data. However, earlier works are based on approximative ODE solutions or point estimates. We propose a novel Bayesian nonparametric model that uses Gaussian processes to infer posteriors of unknown ODE systems directly from data. We derive sparse variational inference with decoupled functional sampling to represent vector field posteriors. We also introduce a probabilistic shooting augmentation to enable efficient inference from arbitrarily long trajectories. The method …
The University of Tokyo（东京大学）；KAIST（韩国科学技术院）；RIKEN（理化学研究所）
We detail a novel class of implicit neural models. Leveraging time-parallel methods for differential equations, Multiple Shooting Layers (MSLs) seek solutions of initial value problems via parallelizable root-finding algorithms. MSLs broadly serve as drop-in replacements for neural ordinary differential equations (Neural ODEs) with improved efficiency in number of function evaluations (NFEs) and wall-clock inference time. We develop the algorithmic framework of MSLs, analyzing the different choices of solution methods from a theoretical and computational perspective. MSLs are showcased in long…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Memorial Sloan Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）
Smooth dynamics interrupted by discontinuities are known as hybrid systems and arise commonly in nature. Latent ODEs allow for powerful representation of irregularly sampled time series but are not designed to capture trajectories arising from hybrid systems. Here, we propose the Latent Segmented ODE (LatSegODE), which uses Latent ODEs to perform reconstruction and changepoint detection within hybrid trajectories featuring jump discontinuities and switching dynamical modes. Where it is possible to train a Latent ODE on the smooth dynamical flows between discontinuities, we apply the pruned exa…
Gran Sasso Science Institute (GSSI)（格兰萨索科学研究所）；University of Cambridge（剑桥大学）；Norwegian University of Science and Technology (NTNU)（挪威科技大学）；University of Edinburgh（爱丁堡大学）
Stanford University（斯坦福大学）；University of Toronto（多伦多大学）；MILA（蒙特利尔学习算法研究所）
Methods based on ordinary differential equations (ODEs) are widely used to build generative models of time-series. In addition to high computational overhead due to explicitly computing hidden states recurrence, existing ODE-based models fall short in learning sequence data with sharp transitions - common in many real-world systems - due to numerical challenges during optimization. In this work, we propose LS4, a generative model for sequences with latent variables evolving according to a state space ODE to increase modeling capacity. Inspired by recent deep state space models (S4), we achieve…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Time series with non-uniform intervals occur in many applications, and are difficult to model using standard recurrent neural networks (RNNs). We generalize RNNs to have continuous-time hidden dynamics defined by ordinary differential equations (ODEs), a model we call ODE-RNNs. Furthermore, we use ODE-RNNs to replace the recognition network of the recently-proposed Latent ODE model. Both ODE-RNNs and Latent ODEs can naturally handle arbitrary time gaps between observations, and can explicitly model the probability of observation times using Poisson processes. We show experimentally that these …
Machine Learning Methods for Studying Latent Neural Activity Dynamics
School of Software Engineering, Sun Yat-sen University（中山大学软件工程学院）；Department of Computer Science, Cornell University（康奈尔大学计算机科学系）；Department of Neurobiology and Behavior, Cornell University（康奈尔大学神经生物学与行为学系）；Department of Ecology and Evolutionary Biology, Cornell University（康奈尔大学生态学与进化生物学系）；School of Computer Science and Artificial Intelligence, Foshan University（佛山大学计算机科学与人工智能学院）
Despite substantial progress in deep learning approaches to time-series reconstruction, no existing methods are designed to uncover local activities with minute signal strength due to their negligible contribution to the optimization loss. Such local activities however can signify important abnormal events in physiological systems, such as an extra foci triggering an abnormal propagation of electrical waves in the heart. We discuss a novel technique for reconstructing such local activity that, while small in signal strength, is the cause of subsequent global activities that have larger signal …
Doctoral Program of Computer Science, University of Helsinki, Helsinki, Finland（赫尔辛基大学计算机科学博士项目）；Research Program in Systems Oncology, Faculty of Medicine, University of Helsinki, Helsinki, Finland（赫尔辛基大学医学系系统肿瘤研究计划）
We present Ordinary Differential Equation Variational Auto-Encoder (ODE$^2$VAE), a latent second order ODE model for high-dimensional sequential data. Leveraging the advances in deep generative models, ODE$^2$VAE can simultaneously learn the embedding of high dimensional trajectories and infer arbitrarily complex continuous-time latent dynamics. Our model explicitly decomposes the latent space into momentum and position components and solves a second order ODE system, which is in contrast to recurrent neural network (RNN) based time series models and recently proposed black-box ODE techniques.…
University of Tübingen（图宾根大学）；University of Southern Denmark（南丹麦大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）
We study time uncertainty-aware modeling of continuous-time dynamics of interacting objects. We introduce a new model that decomposes independent dynamics of single objects accurately from their interactions. By employing latent Gaussian process ordinary differential equations, our model infers both independent dynamics and their interactions with reliable uncertainty estimates. In our formulation, each object is represented as a graph node and interactions are modeled by accumulating the messages coming from neighboring objects. We show that efficient inference of such a complex network of va…
A key problem in computational biology is discovering the gene expression changes that regulate cell fate transitions, in which one cell type turns into another. However, each individual cell cannot be tracked longitudinally, and cells at the same point in real time may be at different stages of the transition process. This can be viewed as a problem of learning the behavior of a dynamical system from observations whose times are unknown. Additionally, a single progenitor cell type often bifurcates into multiple child cell types, further complicating the problem of modeling the dynamics. To ad…
School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）；University of California, Los Angeles（加州大学洛杉矶分校）；Akool Research（Akool研究院）
Black Sigatoka disease severely decreases global banana production, and climate change aggravates the problem by altering fungal species distributions. Due to the heavy financial burden of managing this infectious disease, farmers in developing countries face significant banana crop losses. Though scientists have produced mathematical models of infectious diseases, adapting these models to incorporate climate effects is difficult. We present MR. NODE (Multiple predictoR Neural ODE), a neural network that models the dynamics of black Sigatoka infection learnt directly from data via Neural Ordin…
University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）；Canadian Institute for Advanced Research（加拿大高等研究院）
The core objective of machine-assisted scientific discovery is to learn physical laws from experimental data without prior knowledge of the systems in question. In the area of quantum physics, making progress towards these goals is significantly more challenging due to the curse of dimensionality as well as the counter-intuitive nature of quantum mechanics. Here, we present the QNODE, a latent neural ODE trained on expectation values of closed and open quantum systems dynamics. It can learn to generate such measurement data and extrapolate outside of its training region that satisfies the von …
University of Cambridge（剑桥大学）；UCLA（加州大学洛杉矶分校）；Amsterdam UMC（阿姆斯特丹大学医学中心）；The Alan Turing Institute（艾伦·图灵研究所）
Modeling a system's temporal behaviour in reaction to external stimuli is a fundamental problem in many areas. Pure Machine Learning (ML) approaches often fail in the small sample regime and cannot provide actionable insights beyond predictions. A promising modification has been to incorporate expert domain knowledge into ML models. The application we consider is predicting the progression of disease under medications, where a plethora of domain knowledge is available from pharmacology. Pharmacological models describe the dynamics of carefully-chosen medically meaningful variables in terms of …
USACE Engineer Research Development Center（美国陆军工程兵团工程师研究与发展中心）；University of Genoa（热那亚大学）；University of Padua（帕多瓦大学）
Model reduction for fluid flow simulation continues to be of great interest across a number of scientific and engineering fields. In a previous work [arXiv:2104.13962], we explored the use of Neural Ordinary Differential Equations (NODE) as a non-intrusive method for propagating the latent-space dynamics in reduced order models. Here, we investigate employing deep autoencoders for discovering the reduced basis representation, the dynamics of which are then approximated by NODE. The ability of deep autoencoders to represent the latent-space is compared to the traditional proper orthogonal decom…
Dynamic causal modeling (DCM) is a Bayesian framework to infer directed connections between compartments, and has been used to describe the interactions between underlying neural populations based on functional neuroimaging data. DCM is typically analyzed with the expectation-maximization (EM) algorithm. However, because the inversion of a large-scale continuous system is difficult when noisy observations are present, DCM by EM is typically limited to a small number of compartments ($<10$). Another drawback with the current method is its complexity; when the forward model changes, the posterio…
Duke University（杜克大学）；University of Minnesota-Twin Cities（明尼苏达大学双城分校）
We propose Characteristic-Neural Ordinary Differential Equations (C-NODEs), a framework for extending Neural Ordinary Differential Equations (NODEs) beyond ODEs. While NODEs model the evolution of a latent variables as the solution to an ODE, C-NODE models the evolution of the latent variables as the solution of a family of first-order quasi-linear partial differential equations (PDEs) along curves on which the PDEs reduce to ODEs, referred to as characteristic curves. This in turn allows the application of the standard frameworks for solving ODEs, namely the adjoint method. Learning optimal c…
Tandon School of Engineering, New York University（纽约大学坦登工程学院）；Inria（法国国家信息与自动化研究所）；École normale supérieure（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）
Modeling dynamical systems plays a crucial role in capturing and understanding complex physical phenomena. When physical models are not sufficiently accurate or hardly describable by analytical formulas, one can use generic function approximators such as neural networks to capture the system dynamics directly from sensor measurements. As for now, current methods to learn the parameters of these neural networks are highly sensitive to the inherent instability of most dynamical systems of interest, which in turn prevents the study of very long sequences. In this work, we introduce a generic and …
Effective Latent Differential Equation Models via Attention and Multiple Shooting
Universidad de Buenos Aires（布宜诺斯艾利斯大学）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；University of Montreal（蒙特利尔大学）；IBM Research（IBM研究院）；CONICET（国家科学技术研究委员会）；CHU Sainte-Justine（圣贾斯汀大学医院中心）
Ansys France（法国安斯公司）；Mines Paris - PSL University（巴黎国立高等矿业学校 - PSL大学）；RWTH Aachen University（亚琛工业大学）
Identifying dynamical systems from experimental data is a notably difficult task. Prior knowledge generally helps, but the extent of this knowledge varies with the application, and customized models are often needed. Neural ordinary differential equations can be written as a flexible framework for system identification and can incorporate a broad spectrum of physical insight, giving physical interpretability to the resulting latent space. In the case of partial observations, however, the data points cannot directly be mapped to the latent state of the ODE. Hence, we propose to design recogniti…
Monash University（莫纳什大学）；La Trobe University（乐卓博大学）；Shanghai Jiao Tong University（上海交通大学）；East China Normal University（华东师范大学）；Griffith University（格里菲斯大学）
Multivariate time series forecasting has long received significant attention in real-world applications, such as energy consumption and traffic prediction. While recent methods demonstrate good forecasting abilities, they have three fundamental limitations. (i) Discrete neural architectures: Interlacing individually parameterized spatial and temporal blocks to encode rich underlying patterns leads to discontinuous latent state trajectories and higher forecasting numerical errors. (ii) High complexity: Discrete approaches complicate models with dedicated designs and redundant parameters, leadin…
Polytechnique Montréal（蒙特利尔理工学院）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；École de technologie supérieure（高等工程技术学院）；CIFAR（加拿大高等研究院）
Despite having been studied to a great extent, the task of conditional generation of sequences of frames, or videos, remains extremely challenging. It is a common belief that a key step towards solving this task resides in modelling accurately both spatial and temporal information in video signals. A promising direction to do so has been to learn latent variable models that predict the future in latent space and project back to pixels, as suggested in recent literature. Following this line of work and building on top of a family of models introduced in prior work, Neural ODE, we investigate an…
Learning Individual Dynamics from Sparse Cross-Sectional Snapshots
Statistics and Machine Learning, German Center for Neurodegenerative Diseases (DZNE)（统计与机器学习，德国神经退行性疾病中心（DZNE））；MediaTek Research（联发科技研究）；Department of Mechanical Engineering & AI Institute in Dynamic Systems, University of Washington, Seattle（机械工程与人工智能动态系统研究所，华盛顿大学，西雅图）；DZNE & University of Bonn, Bonn, Germany and University of Cambridge, Cambridge, United Kingdom（DZNE与波恩大学，波恩，德国和剑桥大学，剑桥，英国）
Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）
A wide range of applications require learning image generation models whose latent space effectively captures the high-level factors of variation present in the data distribution. The extent to which a model represents such variations through its latent space can be judged by its ability to interpolate between images smoothly. However, most generative models mapping a fixed prior to the generated images lead to interpolation trajectories lacking smoothness and containing images of reduced quality. In this work, we propose a novel generative model that learns a flexible non-parametric prior ove…
End-to-end learning of dynamical systems with black-box models, such as neural ordinary differential equations (ODEs), provides a flexible framework for learning dynamics from data without prescribing a mathematical model for the dynamics. Unfortunately, this flexibility comes at the cost of understanding the dynamical system, for which ODEs are used ubiquitously. Further, experimental data are collected under various conditions (inputs), such as treatments, or grouped in some way, such as part of sub-populations. Understanding the effects of these system inputs on system outputs is crucial to…
A statistical approach to latent dynamic modeling with differential equations
Institute of Medical Biometry and Statistics, Faculty of Medicine and Medical Center, University of Freiburg（医学生物计量学与统计学研究所，医学学院与医学中心，弗赖堡大学）；Freiburg Center for Data Analysis and Modeling, University of Freiburg（弗赖堡数据分析与建模中心，弗赖堡大学）；Centre for Integrative Biological Signaling Studies (CIBSS), University of Freiburg（整合生物信号研究中心（CIBSS），弗赖堡大学）；Department of Neuropediatrics and Muscle Disorders, Faculty of Medicine and Medical Center, University of Freiburg（神经儿科与肌肉疾病部门，医学学院与医学中心，弗赖堡大学）
ESAT-STADIUS；KU Leuven（鲁汶大学）；Department of Computer Science（计算机科学系）；University of Toronto（多伦多大学）
Neural ordinary differential equations (Neural ODEs) are an effective framework for learning dynamical systems from irregularly sampled time series data. These models provide a continuous-time latent representation of the underlying dynamical system where new observations at arbitrary time points can be used to update the latent representation of the dynamical system. Existing parameterizations for the dynamics functions of Neural ODEs limit the ability of the model to retain global information about the time series; specifically, a piece-wise integration of the latent process between observat…
Structured Kolmogorov-Arnold Neural ODEs for Interpretable Learning and Symbolic Discovery of Nonlinear Dynamics
Department of Industrial Systems Engineering and Management, National University of Singapore（工业系统工程与管理系，新加坡国立大学）；Department of Civil, Environmental and Geomatic Engineering, ETH Zürich（土木、环境与测绘工程系，苏黎世联邦理工学院）；Future Resilient Systems, Singapore-ETH Centre（未来韧性系统，新加坡-苏黎世中心）
HKUST(GZ)（香港科技大学（广州））；HKUST（香港科技大学）；Future Resilient Systems, Singapore-ETH Centre（新加坡-苏黎世联邦理工学院中心未来弹性系统）；National University of Singapore（新加坡国立大学）；Tongji University（同济大学）；Shanghai Qizhi Institute（上海期智研究院）
The order/dimension of models derived on the basis of data is commonly restricted by the number of observations, or in the context of monitored systems, sensing nodes. This is particularly true for structural systems (e.g., civil or mechanical structures), which are typically high-dimensional in nature. In the scope of physics-informed machine learning, this paper proposes a framework -- termed Neural Modal ODEs -- to integrate physics-based modeling with deep learning for modeling the dynamics of monitored and high-dimensional engineered systems. Neural Ordinary Differential Equations -- Neur…
Southwestern University of Finance and Economics（西南财经大学）；University of Maryland, College Park（马里兰大学帕克分校）；Pennsylvania State University（宾夕法尼亚州立大学）；University of Electronic Science and Technology of China（电子科技大学）
Stock selection attempts to rank a list of stocks for optimizing investment decision making, aiming at minimizing investment risks while maximizing profit returns. Recently, researchers have developed various (recurrent) neural network-based methods to tackle this problem. Without exceptions, they primarily leverage historical market volatility to enhance the selection performance. However, these approaches greatly rely on discrete sampled market observations, which either fail to consider the uncertainty of stock fluctuations or predict continuous stock dynamics in the future. Besides, some s…
Neural differential equations are a promising new member in the neural network family. They show the potential of differential equations for time series data analysis. In this paper, the strength of the ordinary differential equation (ODE) is explored with a new extension. The main goal of this work is to answer the following questions: (i)~can ODE be used to redefine the existing neural network model? (ii)~can Neural ODEs solve the irregular sampling rate challenge of existing neural network models for a continuous time series, i.e., length and dynamic nature, (iii)~how to reduce the training…
Neural ODE and SDE Models for Adaptation and Planning in Model-Based Reinforcement Learning
School of Computer Science, The University of Sheffield, UK（谢菲尔德大学计算机科学学院，英国）；Cancer Research UK National Biomarker Centre, The University of Manchester, UK（英国曼彻斯特大学癌症研究UK国家生物标志物中心）；School of Chemical, Materials and Biological Engineering, The University of Sheffield, UK（谢菲尔德大学化学、材料和生物工程学院，英国）；Machine Learning group, Centrum Wiskunde & Informatica, Amsterdam, Netherlands（荷兰阿姆斯特丹Centrum Wiskunde & Informatica机器学习组）；Institute for Ecological Economics, Vienna University of Economics and Business, Austria（奥地利维也纳经济与商业大学生态经济研究所）
