National Research University Higher School of Economics（俄罗斯国家研究型高等经济大学）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Samsung AI Center Moscow（三星莫斯科人工智能中心）
Viktor Oganesyan, Alexandra Volokhova, Dmitry Vetrov
Stochastic regularization of neural networks (e.g. dropout) is a wide-spread technique in deep learning that allows for better generalization. Despite its success, continuous-time models, such as neural ordinary differential equation (ODE), usually rely on a completely deterministic feed-forward operation. This work provides an empirical study of stochastically regularized neural ODE on several image-classification tasks (CIFAR-10, CIFAR-100, TinyImageNet). Building upon the formalism of stochastic differential equations (SDEs), we demonstrate that neural SDE is able to outperform its deterministic counterpart. Further, we show that data augmentation during the training improves the performance of both deterministic and stochastic versions of the same model. However, the improvements obtained by the data augmentation completely eliminate the empirical gains of the stochastic regularization, making the difference in the performance of neural ODE and neural SDE negligible.
We show that Neural Ordinary Differential Equations (ODEs) learn representations that preserve the topology of the input space and prove that this implies the existence of functions Neural ODEs cannot represent. To address these limitations, we introduce Augmented Neural ODEs which, in addition to being more expressive models, are empirically more stable, generalize better and have a lower computational cost than Neural ODEs.
National University of Ireland Galway（爱尔兰国立大学戈尔韦分校）
Mathematical modeling with Ordinary Differential Equations (ODEs) has proven to be extremely successful in a variety of fields, including biology. However, these models are completely deterministic given a certain set of initial conditions. We convert mathematical ODE models of three benchmark biological systems to Dynamic Bayesian Networks (DBNs). The DBN model can handle model uncertainty and data uncertainty in a principled manner. They can be used for temporal data mining for noisy and missing variables. We apply Particle Filtering algorithm to infer the model variables by re-estimating th…
Neural ordinary differential equations (ODEs) have been attracting increasing attention in various research domains recently. There have been some works studying optimization issues and approximation capabilities of neural ODEs, but their robustness is still yet unclear. In this work, we fill this important gap by exploring robustness properties of neural ODEs both empirically and theoretically. We first present an empirical study on the robustness of the neural ODE-based networks (ODENets) by exposing them to inputs with various types of perturbations and subsequently investigating the change…
We consider the neural ODE and optimal control perspective of supervised learning, with $\ell^1$-control penalties, where rather than only minimizing a final cost (the \emph{empirical risk}) for the state, we integrate this cost over the entire time horizon. We prove that any optimal control (for this cost) vanishes beyond some positive stopping time. When seen in the discrete-time context, this result entails an \emph{ordered} sparsity pattern for the parameters of the associated residual neural network: ordered in the sense that these parameters are all $0$ beyond a certain layer. Furthermor…
Tsinghua University（清华大学）；Peng Cheng Laboratory（鹏城实验室）；Renmin University of China（中国人民大学）
Score-based generative models have excellent performance in terms of generation quality and likelihood. They model the data distribution by matching a parameterized score network with first-order data score functions. The score network can be used to define an ODE ("score-based diffusion ODE") for exact likelihood evaluation. However, the relationship between the likelihood of the ODE and the score matching objective is unclear. In this work, we prove that matching the first-order score is not sufficient to maximize the likelihood of the ODE, by showing a gap between the maximum likelihood and…
Black Sigatoka disease severely decreases global banana production, and climate change aggravates the problem by altering fungal species distributions. Due to the heavy financial burden of managing this infectious disease, farmers in developing countries face significant banana crop losses. Though scientists have produced mathematical models of infectious diseases, adapting these models to incorporate climate effects is difficult. We present MR. NODE (Multiple predictoR Neural ODE), a neural network that models the dynamics of black Sigatoka infection learnt directly from data via Neural Ordin…
Multilayer Perceptrons struggle to learn certain simple arithmetic tasks. Specialist neural modules for arithmetic can outperform classical architectures with gains in extrapolation, interpretability and convergence speeds, but are highly sensitive to the training range. In this paper, we show that Neural Multiplication Units (NMUs) are unable to reliably learn tasks as simple as multiplying two inputs when given different training ranges. Causes of failure are linked to inductive and input biases which encourage convergence to solutions in undesirable optima. A solution, the stochastic NMU (s…
KAIST（韩国科学技术院）；Center for Superintelligence, SNU（首尔大学超级智能中心）；New York University（纽约大学）；Facebook AI Research（Facebook人工智能研究院）
Although stochastic gradient descent (SGD) is a driving force behind the recent success of deep learning, our understanding of its dynamics in a high-dimensional parameter space is limited. In recent years, some researchers have used the stochasticity of minibatch gradients, or the signal-to-noise ratio, to better characterize the learning dynamics of SGD. Inspired from these work, we here analyze SGD from a geometrical perspective by inspecting the stochasticity of the norms and directions of minibatch gradients. We propose a model of the directional concentration for minibatch gradients thro…
Interpretable Diagnostics and Adaptive Data Assimilation for Neural ODEs via Discrete Empirical Interpolation
School of Mechanical Engineering, Purdue University（普渡大学机械工程学院）；Mathematics and Computer Science Division, Argonne National Laboratory（阿贡国家实验室数学与计算机科学部）
University of Oxford（牛津大学）；University of Amsterdam（阿姆斯特丹大学）
We investigate the benefit of treating all the parameters in a Bayesian neural network stochastically and find compelling theoretical and empirical evidence that this standard construction may be unnecessary. To this end, we prove that expressive predictive distributions require only small amounts of stochasticity. In particular, partially stochastic networks with only $n$ stochastic biases are universal probabilistic predictors for $n$-dimensional predictive problems. In empirical investigations, we find no systematic benefit of full stochasticity across four different inference modalities an…
Artificial and Natural Intelligence Toulouse Institute, Universite de Toulouse（图卢兹大学人工与自然智能研究所）；Centre de Recherche Cerveau & Cognition CNRS, Université de Toulouse（图卢兹大学国家科学研究中心大脑与认知研究中心）；Data Science Initiative, Brown University（布朗大学数据科学计划）
Despite their elegant formulation and lightweight memory cost, neural ordinary differential equations (NODEs) suffer from known representational limitations. In particular, the single flow learned by NODEs cannot express all homeomorphisms from a given data space to itself, and their static weight parameterization restricts the type of functions they can learn compared to discrete architectures with layer-dependent weights. Here, we describe a new module called neurally controlled ODE (N-CODE) designed to improve the expressivity of NODEs. The parameters of N-CODE modules are dynamic variables…
Technical University of Munich（慕尼黑工业大学）；AWS AI Labs（AWS AI 实验室）
Neural ordinary differential equations describe how values change in time. This is the reason why they gained importance in modeling sequential data, especially when the observations are made at irregular intervals. In this paper we propose an alternative by directly modeling the solution curves - the flow of an ODE - with a neural network. This immediately eliminates the need for expensive numerical solvers while still maintaining the modeling capability of neural ODEs. We propose several flow architectures suitable for different applications by establishing precise conditions on when a funct…
Nonparametric Filtering, Estimation and Classification using Neural Jump ODEs
Department of Mathematics ETH Zurich（数学系，苏黎世联邦理工学院）；Department of Statistics UC Berkeley（统计系，伯克利大学）；Department of Mathematics Albert-Ludwigs-Universität Freiburg（数学系，弗赖堡阿尔伯特-路易斯大学）
California Institute of Technology（加州理工学院）；Argo AI（阿尔戈人工智能公司）
We propose a method for training ordinary differential equations by using a control-theoretic Lyapunov condition for stability. Our approach, called LyaNet, is based on a novel Lyapunov loss formulation that encourages the inference dynamics to converge quickly to the correct prediction. Theoretically, we show that minimizing Lyapunov loss guarantees exponential convergence to the correct solution and enables a novel robustness guarantee. We also provide practical algorithms, including one that avoids the cost of backpropagating through a solver or using the adjoint method. Relative to standar…
We revisit the theoretical properties of Hamiltonian stochastic differential equations (SDES) for Bayesian posterior sampling, and we study the two types of errors that arise from numerical SDE simulation: the discretization error and the error due to noisy gradient estimates in the context of data subsampling. Our main result is a novel analysis for the effect of mini-batches through the lens of differential operator splitting, revising previous literature results. The stochastic component of a Hamiltonian SDE is decoupled from the gradient noise, for which we make no normality assumptions. T…
A minimax optimal control approach for robust neural ODEs
Technical University Munich (TUM)（慕尼黑工业大学（TUM））；Munich Data Science Institute (MDSI)（慕尼黑数据科学研究所（MDSI））；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））
On the Closed-Form of Flow Matching: Generalization Does Not Arise from Target Stochasticity
Université Jean Monnet Saint-Étienne, CNRS, Institut d’Optique Graduate School, Inria, Laboratoire Hubert Curien UMR 5516；ENS de Lyon, CNRS, Université Claude Bernard Lyon 1, Inria, LIP UMR 5668；Inria, ENS de Lyon, CNRS, Université Claude Bernard Lyon 1, LIP UMR 5668；Institut Universitaire de France
Continuous deep learning architectures enable learning of flexible probabilistic models for predictive modeling as neural ordinary differential equations (ODEs), and for generative modeling as continuous normalizing flows. In this work, we design a framework to decipher the internal dynamics of these continuous depth models by pruning their network architectures. Our empirical results suggest that pruning improves generalization for neural ODEs in generative modeling. We empirically show that the improvement is because pruning helps avoid mode-collapse and flatten the loss surface. Moreover, p…
We present and review an algorithmic and theoretical framework for improving neural network architecture design via momentum. As case studies, we consider how momentum can improve the architecture design for recurrent neural networks (RNNs), neural ordinary differential equations (ODEs), and transformers. We show that integrating momentum into neural network architectures has several remarkable theoretical and empirical benefits, including 1) integrating momentum into RNNs and neural ODEs can overcome the vanishing gradient issues in training RNNs and neural ODEs, resulting in effective learni…
Duke University（杜克大学）；University of Minnesota-Twin Cities（明尼苏达大学双城分校）
We propose Characteristic-Neural Ordinary Differential Equations (C-NODEs), a framework for extending Neural Ordinary Differential Equations (NODEs) beyond ODEs. While NODEs model the evolution of a latent variables as the solution to an ODE, C-NODE models the evolution of the latent variables as the solution of a family of first-order quasi-linear partial differential equations (PDEs) along curves on which the PDEs reduce to ODEs, referred to as characteristic curves. This in turn allows the application of the standard frameworks for solving ODEs, namely the adjoint method. Learning optimal c…
In recent years, the connections between deep residual networks and first-order Ordinary Differential Equations (ODEs) have been disclosed. In this work, we further bridge the deep neural architecture design with the second-order ODEs and propose a novel reversible neural network, termed as m-RevNet, that is characterized by inserting momentum update to residual blocks. The reversible property allows us to perform backward pass without access to activation values of the forward pass, greatly relieving the storage burden during training. Furthermore, the theoretical foundation based on second-o…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Nevada（内华达大学）
In this work, multiplicative stochasticity is applied to the learning rate of stochastic optimization algorithms, giving rise to stochastic learning-rate schemes. In-expectation theoretical convergence results of Stochastic Gradient Descent equipped with this novel stochastic learning rate scheme under the stochastic setting, as well as convergence results under the online optimization settings are provided. Empirical results consider the case of an adaptively uniformly distributed multiplicative stochasticity and include not only Stochastic Gradient Descent, but also other popular algorithms …
Neural ODE and SDE Models for Adaptation and Planning in Model-Based Reinforcement Learning
School of Computer Science, The University of Sheffield, UK（谢菲尔德大学计算机科学学院，英国）；Cancer Research UK National Biomarker Centre, The University of Manchester, UK（英国曼彻斯特大学癌症研究UK国家生物标志物中心）；School of Chemical, Materials and Biological Engineering, The University of Sheffield, UK（谢菲尔德大学化学、材料和生物工程学院，英国）；Machine Learning group, Centrum Wiskunde & Informatica, Amsterdam, Netherlands（荷兰阿姆斯特丹Centrum Wiskunde & Informatica机器学习组）；Institute for Ecological Economics, Vienna University of Economics and Business, Austria（奥地利维也纳经济与商业大学生态经济研究所）
Detecting Stochasticity in Discrete Signals via Nonparametric Excursion Theorem
Dept. of Mechanical Engineering, Michigan State University（密歇根州立大学机械工程系）；Dept. of Computational Mathematics, Science and Engineering, Michigan State University（密歇根州立大学计算数学、科学与工程系）
In reinforcement learning, continuous time is often discretized by a time scale $δ$, to which the resulting performance is known to be highly sensitive. In this work, we seek to find a $δ$-invariant algorithm for policy gradient (PG) methods, which performs well regardless of the value of $δ$. We first identify the underlying reasons that cause PG methods to fail as $δ\to 0$, proving that the variance of the PG estimator can diverge to infinity in stochastic environments under a certain assumption of stochasticity. While durative actions or action repetition can be employed to have $δ$-invaria…
Activation Saturation and Floquet Spectrum Collapse in Neural ODEs
School of Pedagogical & Technological Education (ASPETE)（教育与技术教育学院（ASPETE））；Department of Mathematics, Chair of Dynamics, Control, Machine Learning and Numerics, Friedrich-Alexander-Universität Erlangen–Nürnberg (FAU)（弗里德里希-亚历山大-埃尔朗根-纽伦堡大学（FAU）数学系动力学、控制、机器学习与数值计算教席）
Partially Stochastic Infinitely Deep Bayesian Neural Networks
Oxford-Man Institute of Quantitative Finance, University of Oxford（牛津大学牛津-曼量化金融研究所）；Mathematical Institute, University of Oxford（牛津大学数学研究所）；Imperial College London（伦敦帝国学院）
University of California, Berkeley（加州大学伯克利分校）；The Ohio State University（俄亥俄州立大学）
Stochastic differential equations (SDEs) are used to describe a wide variety of complex stochastic dynamical systems. Learning the hidden physics within SDEs is crucial for unraveling fundamental understanding of these systems' stochastic and nonlinear behavior. We propose a flexible and scalable framework for training artificial neural networks to learn constitutive equations that represent hidden physics within SDEs. The proposed stochastic physics-informed neural ordinary differential equation framework (SPINODE) propagates stochasticity through the known structure of the SDE (i.e., the kno…
Towards Complex Dynamic Physics System Simulation with Graph Neural ODEs
Department of Chemical and Biological Engineering, Faculty of Engineering, Monash University（莫纳什大学化学与生物工程系）；Department of Data Science and Artificial Intelligence, Faculty of IT, Monash University（莫纳什大学数据科学与人工智能系）；School of Information and Communication Technology, Griffith University（格里菲斯大学信息与通信技术学院）；Department of Computer Science, University of Illinois at Chicago（伊利诺伊大学芝加哥分校计算机科学系）
Recently, dropout has seen increasing use in deep learning. For deep convolutional neural networks, dropout is known to work well in fully-connected layers. However, its effect in pooling layers is still not clear. This paper demonstrates that max-pooling dropout is equivalent to randomly picking activation based on a multinomial distribution at training time. In light of this insight, we advocate employing our proposed probabilistic weighted pooling, instead of commonly used max-pooling, to act as model averaging at test time. Empirical evidence validates the superiority of probabilistic weig…
Worcester Polytechnic Institute（伍斯特理工学院）；University of Bologna（博洛尼亚大学）
In this paper we use neural networks to learn governing equations from data. Specifically we reconstruct the right-hand side of a system of ODEs $\dot{x}(t) = f(t, x(t))$ directly from observed uniformly time-sampled data using a neural network. In contrast with other neural network based approaches to this problem, we add a Lipschitz regularization term to our loss function. In the synthetic examples we observed empirically that this regularization results in a smoother approximating function and better generalization properties when compared with non-regularized models, both on trajectory an…
