Reasoning over an instance composed of a set of vectors, like a point cloud, requires that one accounts for intra-set dependent features among elements. However, since such instances are unordered, the elements' features should remain unchanged when the input's order is permuted. This property, permutation equivariance, is a challenging constraint for most neural architectures. While recent work has proposed global pooling and attention-based solutions, these may be limited in the way that intradependencies are captured in practice. In this work we propose a more general formulation to achieve permutation equivariance through ordinary differential equations (ODE). Our proposed module, Exchangeable Neural ODE (ExNODE), can be seamlessly applied for both discriminative and generative tasks. We also extend set modeling in the temporal dimension and propose a VAE based model for temporal set modeling. Extensive experiments demonstrate the efficacy of our method over strong baselines.
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Modeling dependencies among features is fundamental for many machine learning tasks. Although there are often multiple related instances that may be leveraged to inform conditional dependencies, typical approaches only model conditional dependencies over individual instances. In this work, we propose a novel framework, partially observed exchangeable modeling (POEx) that takes in a set of related partially observed instances and infers the conditional distribution for the unobserved dimensions over multiple elements. Our approach jointly models the intra-instance (among features in a point) an…
Google Research（谷歌研究院）；University of Alberta（阿尔伯塔大学）
Recently there has been growing interest in modeling sets with exchangeability such as point clouds. A shortcoming of current approaches is that they restrict the cardinality of the sets considered or can only express limited forms of distribution over unobserved data. To overcome these limitations, we introduce Energy-Based Processes (EBPs), which extend energy based models to exchangeable data while allowing neural network parameterizations of the energy function. A key advantage of these models is the ability to express more flexible distributions over sets without restricting their cardina…
Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning
Korea University（高丽大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Texas A&M University-Commerce（德州农工大学 Commerce 分校）；Qualcomm AI Research（高通人工智能研究部）；Stanford（斯坦福大学）；KAIST（韩国科学技术院）；Qualcomm Vietnam Company Limited（高通越南有限公司）
Technion - Israel Institute of Technology（以色列理工学院）；Rambam Health Care Campus（拉姆巴姆医疗中心）
In several crucial applications, domain knowledge is encoded by a system of ordinary differential equations (ODE), often stemming from underlying physical and biological processes. A motivating example is intensive care unit patients: the dynamics of vital physiological functions, such as the cardiovascular system with its associated variables (heart rate, cardiac contractility and output and vascular resistance) can be approximately described by a known system of ODEs. Typically, some of the ODE variables are directly observed (heart rate and blood pressure for example) while some are unobser…
Effectively modeling phenomena present in highly nonlinear dynamical systems whilst also accurately quantifying uncertainty is a challenging task, which often requires problem-specific techniques. We present a novel, domain-agnostic approach to tackling this problem, using compositions of physics-informed random features, derived from ordinary differential equations. The architecture of our model leverages recent advances in approximate inference for deep Gaussian processes, such as layer-wise weight-space approximations which allow us to incorporate random Fourier features, and stochastic var…
GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series
KU LEUVEN（鲁汶大学）
对真实世界的多维时间序列进行建模在这些序列被零星地观察时（即，采样在时间上和跨维度上都是不规则的）-例如在临床患者数据的情况下。为了解决这些挑战，我们提出（1）基于最近的神经常微分方程（Chen et al.，2018），以及（2）处理零星观测的贝叶斯更新网络。我们把这两个想法在我们的GRU-ODE-Bayes方法。然后，我们证明了所提出的方法编码的潜在过程的连续性之前，它可以准确地表示由多维随机微分方程驱动的复杂过程的福克-普朗克动力学。此外，实证评估表明，我们的方法优于最先进的合成数据和现实世界的数据与医疗保健和气候预测的应用。更重要的是，之前的连续性被证明是非常适合于低数量的样本设置。
Shanghai Jiao Tong University（上海交通大学）；University of Birmingham（伯明翰大学）
Predicting the behaviors of pedestrian crowds is of critical importance for a variety of real-world problems. Data driven modeling, which aims to learn the mathematical models from observed data, is a promising tool to construct models that can make accurate predictions of such systems. In this work, we present a data-driven modeling approach based on the ODE-Net framework, for constructing continuous-time models of crowd dynamics. We discuss some challenging issues in applying the ODE-Net method to such problems, which are primarily associated with the dimensionality of the underlying crowd s…
NAVER R&D Center（NAVER研发中心）；NAVER CLOVA；NAVER AI LAB（NAVER人工智能实验室）
Survival analysis appears in various fields such as medicine, economics, engineering, and business. Recent studies showed that the Ordinary Differential Equation (ODE) modeling framework unifies many existing survival models while the framework is flexible and widely applicable. However, naively applying the ODE framework to survival analysis problems may model fiercely changing density function which may worsen the model's performance. Though we can apply L1 or L2 regularizers to the ODE model, their effect on the ODE modeling framework is barely known. In this paper, we propose hazard gradie…
Neural Ordinary Differential Equations (ODEs) was recently introduced as a new family of neural network models, which relies on black-box ODE solvers for inference and training. Some ODE solvers called adaptive can adapt their evaluation strategy depending on the complexity of the problem at hand, opening great perspectives in machine learning. However, this paper describes a simple set of experiments to show why adaptive solvers cannot be seamlessly leveraged as a black-box for dynamical systems modelling. By taking the Lorenz'63 system as a showcase, we show that a naive application of the F…
A statistical approach to latent dynamic modeling with differential equations
Institute of Medical Biometry and Statistics, Faculty of Medicine and Medical Center, University of Freiburg（医学生物计量学与统计学研究所，医学学院与医学中心，弗赖堡大学）；Freiburg Center for Data Analysis and Modeling, University of Freiburg（弗赖堡数据分析与建模中心，弗赖堡大学）；Centre for Integrative Biological Signaling Studies (CIBSS), University of Freiburg（整合生物信号研究中心（CIBSS），弗赖堡大学）；Department of Neuropediatrics and Muscle Disorders, Faculty of Medicine and Medical Center, University of Freiburg（神经儿科与肌肉疾病部门，医学学院与医学中心，弗赖堡大学）
We present two elegant solutions for modeling continuous-time dynamics, in a novel model-based reinforcement learning (RL) framework for semi-Markov decision processes (SMDPs), using neural ordinary differential equations (ODEs). Our models accurately characterize continuous-time dynamics and enable us to develop high-performing policies using a small amount of data. We also develop a model-based approach for optimizing time schedules to reduce interaction rates with the environment while maintaining the near-optimal performance, which is not possible for model-free methods. We experimentally …
Technical University of Munich（慕尼黑工业大学）；AWS AI Labs（AWS AI 实验室）
Neural ordinary differential equations describe how values change in time. This is the reason why they gained importance in modeling sequential data, especially when the observations are made at irregular intervals. In this paper we propose an alternative by directly modeling the solution curves - the flow of an ODE - with a neural network. This immediately eliminates the need for expensive numerical solvers while still maintaining the modeling capability of neural ODEs. We propose several flow architectures suitable for different applications by establishing precise conditions on when a funct…
Doctoral Program of Computer Science, University of Helsinki, Helsinki, Finland（赫尔辛基大学计算机科学博士项目）；Research Program in Systems Oncology, Faculty of Medicine, University of Helsinki, Helsinki, Finland（赫尔辛基大学医学系系统肿瘤研究计划）
Virginia Polytechnic Institute and State University（弗吉尼亚理工学院暨州立大学）
Finite element analysis (FEA) has been widely used to generate simulations of complex and nonlinear systems. Despite its strength and accuracy, the limitations of FEA can be summarized into two aspects: a) running high-fidelity FEA often requires significant computational cost and consumes a large amount of time; b) FEA is a deterministic method that is insufficient for uncertainty quantification (UQ) when modeling complex systems with various types of uncertainties. In this paper, a physics-informed data-driven surrogate model, named Neural Process Aided Ordinary Differential Equation (NP-ODE…
Advanced Global Wildfire Activity Modeling with Hierarchical Graph ODE
University of Science and Technology of China（中国科学技术大学）；Tsinghua University（清华大学）；Southwest Jiaotong University（西南交通大学）；Beijing Jiaotong University（北京交通大学）；Tencent（腾讯）；Nanyang Technological University（南洋理工大学）
University of Rochester（罗切斯特大学）；Stony Brook University（石溪大学）
The marriage of recurrent neural networks and neural ordinary differential networks (ODE-RNN) is effective in modeling irregularly-observed sequences. While ODE produces the smooth hidden states between observation intervals, the RNN will trigger a hidden state jump when a new observation arrives, thus cause the interpolation discontinuity problem. To address this issue, we propose the cubic spline smoothing compensation, which is a stand-alone module upon either the output or the hidden state of ODE-RNN and can be trained end-to-end. We derive its analytical solution and provide its theoretic…
University of California, Berkeley（加州大学伯克利分校）；Massachusetts Institute of Technology（麻省理工学院）；Johns Hopkins University（约翰斯·霍普金斯大学）；Microsoft Research（微软研究院）；Carnegie Mellon University（卡内基梅隆大学）；Duke University（杜克大学）
Korea University（高丽大学）；KAIST（韩国科学技术院）；The University of Tokyo（东京大学）
By interpreting the forward dynamics of the latent representation of neural networks as an ordinary differential equation, Neural Ordinary Differential Equation (Neural ODE) emerged as an effective framework for modeling a system dynamics in the continuous time domain. However, real-world systems often involves external interventions that cause changes in the system dynamics such as a moving ball coming in contact with another ball, or such as a patient being administered with particular drug. Neural ODE and a number of its recent variants, however, are not suitable for modeling such intervent…
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；Argonne National Laboratory（阿贡国家实验室）
In data-driven modeling of spatiotemporal phenomena careful consideration often needs to be made in capturing the dynamics of the high wavenumbers. This problem becomes especially challenging when the system of interest exhibits shocks or chaotic dynamics. We present a data-driven modeling method that accurately captures shocks and chaotic dynamics by proposing a novel architecture, stabilized neural ordinary differential equation (ODE). In our proposed architecture, we learn the right-hand-side (RHS) of an ODE by adding the outputs of two NN together where one learns a linear term and the oth…
Hybrid$^2$ Neural ODE Causal Modeling and an Application to Glycemic Response
Institute for Computational and Mathematical Engineering, Stanford University（计算与数学工程研究所，斯坦福大学）；Broad Institute of MIT and Harvard（哈佛大学与麻省理工学院Broad研究所）；Department of Pediatrics, Stanford University（斯坦福大学儿科系）；Department of Management Science and Engineering, Stanford University（斯坦福大学管理科学与工程系）；Department of Statistics and Department of Computer Science, Stanford University（斯坦福大学统计系与计算机科学系）
From Fourier to Neural ODEs: Flow Matching for Modeling Complex Systems
College of Science, National University of Defense Technology（国防科技大学理学院）；Research Institute of Intelligent Complex Systems, Fudan University（复旦大学智能复杂系统研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Temporally Detailed Hypergraph Neural ODEs for Disease Progression Modeling
Department of Computer and Information Science and Engineering, University of Florida（佛罗里达大学计算机与信息科学与工程系）；Regenstrief Institute（Regenstrief研究所）；Department of Biostatistics and Health Data Science, Indiana University（印第安纳大学生物统计学与健康数据科学系）；Department of Pharmacy Practice, Purdue University（普渡大学药学实践系）
Data-driven ODE modeling of the high-frequency complex dynamics via a low-frequency dynamics model
Faculty of Commerce and Management, Hitotsubashi University（一桥大学经营学部）；The Graduate School of Environment, Life, Natural Science and Technology, Okayama University（冈山大学环境生命自然科学技术研究院）；Graduate School of Business Administration, Hitotsubashi University（一桥大学经营管理研究科）
We propose to formulate MRI image reconstruction as an optimization problem and model the optimization trajectory as a dynamic process using ordinary differential equations (ODEs). We model the dynamics in ODE with a neural network and solve the desired ODE with the off-the-shelf (fixed) solver to obtain reconstructed images. We extend this model and incorporate the knowledge of off-the-shelf ODE solvers into the network design (learned solvers). We investigate several models based on three ODE solvers and compare models with fixed solvers and learned solvers. Our models achieve better reconst…
University of Macau（澳门大学）；Hanyang University（汉阳大学）；Zhejiang University（浙江大学）；Wuhan University（武汉大学）；Tsinghua University（清华大学）；South China University of Technology（华南理工大学）；University of Liverpool（利物浦大学）
Proper orthogonal decomposition (POD) allows reduced-order modeling of complex dynamical systems at a substantial level, while maintaining a high degree of accuracy in modeling the underlying dynamical systems. Advances in machine learning algorithms enable learning POD-based dynamics from data and making accurate and fast predictions of dynamical systems. In this paper, we leverage the recently proposed heavy-ball neural ODEs (HBNODEs) [Xia et al. NeurIPS, 2021] for learning data-driven reduced-order models (ROMs) in the POD context, in particular, for learning dynamics of time-varying coeffi…
Continuous deep learning architectures enable learning of flexible probabilistic models for predictive modeling as neural ordinary differential equations (ODEs), and for generative modeling as continuous normalizing flows. In this work, we design a framework to decipher the internal dynamics of these continuous depth models by pruning their network architectures. Our empirical results suggest that pruning improves generalization for neural ODEs in generative modeling. We empirically show that the improvement is because pruning helps avoid mode-collapse and flatten the loss surface. Moreover, p…
HKUST(GZ)（香港科技大学（广州））；HKUST（香港科技大学）；Future Resilient Systems, Singapore-ETH Centre（新加坡-苏黎世联邦理工学院中心未来弹性系统）；National University of Singapore（新加坡国立大学）；Tongji University（同济大学）；Shanghai Qizhi Institute（上海期智研究院）
The order/dimension of models derived on the basis of data is commonly restricted by the number of observations, or in the context of monitored systems, sensing nodes. This is particularly true for structural systems (e.g., civil or mechanical structures), which are typically high-dimensional in nature. In the scope of physics-informed machine learning, this paper proposes a framework -- termed Neural Modal ODEs -- to integrate physics-based modeling with deep learning for modeling the dynamics of monitored and high-dimensional engineered systems. Neural Ordinary Differential Equations -- Neur…
Neural Network Emulator for Atmospheric Chemical ODE
School of Engineering Science, Lappeenranta-Lahti University of Technology LUT（拉普兰塔-拉赫蒂理工大学工程科学学院）；Atmospheric Modelling Centre Lahti, Lahti University Campus（拉赫蒂大学校区拉赫蒂大气建模中心）；Institute for Atmospheric and Earth System Research (INAR), The University of Helsinki（赫尔辛基大学大气与地球系统研究所（INAR））
USACE Engineer Research Development Center（美国陆军工程兵团工程师研究与发展中心）；University of Genoa（热那亚大学）；University of Padua（帕多瓦大学）
Model reduction for fluid flow simulation continues to be of great interest across a number of scientific and engineering fields. In a previous work [arXiv:2104.13962], we explored the use of Neural Ordinary Differential Equations (NODE) as a non-intrusive method for propagating the latent-space dynamics in reduced order models. Here, we investigate employing deep autoencoders for discovering the reduced basis representation, the dynamics of which are then approximated by NODE. The ability of deep autoencoders to represent the latent-space is compared to the traditional proper orthogonal decom…
MODE: Efficient Time Series Prediction with Mamba Enhanced by Low-Rank Neural ODEs
School of Computing and Data Science, The University of Hong Kong（计算与数据科学学院，香港大学）；Department of Computing and Data Science, Nanyang Technological University（计算与数据科学系，南洋理工大学）；School of Information Engineering, Beijing Institute of Graphic Communication（信息工程学院，北京印刷学院）；Department of Computing and Data Science, The University of Hong Kong（计算与数据科学系，香港大学）；Yale university（耶鲁大学）；University of Cambridge（剑桥大学）；Nanyang technological university（南洋理工大学）；University of Hong Kong（香港大学）
Dissipative partial differential equations that exhibit chaotic dynamics tend to evolve to attractors that exist on finite-dimensional manifolds. We present a data-driven reduced order modeling method that capitalizes on this fact by finding the coordinates of this manifold and finding an ordinary differential equation (ODE) describing the dynamics in this coordinate system. The manifold coordinates are discovered using an undercomplete autoencoder -- a neural network (NN) that reduces then expands dimension. Then the ODE, in these coordinates, is approximated by a NN using the neural ODE fram…
A convolutional autoencoder and neural ODE surrogate modeling framework applied to transient counterflow flames
Department of Aerospace Engineering, KAIST（韩国科学技术院航空航天工程系）；Institute for Reactive Flows (IRST), University of Stuttgart（斯图加特大学反应流研究所）；Clean Energy Research Platform (CERP), KAUST（阿联酋科威特大学清洁能源研究平台）
