Training deep neural networks (DNNs) can be difficult due to the occurrence of vanishing/exploding gradients during weight optimization. To avoid this problem, we propose a class of DNNs stemming from the time discretization of Hamiltonian systems. The time-invariant version of the corresponding Hamiltonian models enjoys marginal stability, a property that, as shown in previous works and for specific DNNs architectures, can mitigate convergence to zero or divergence of gradients. In the present paper, we formally study this feature by deriving and analysing the backward gradient dynamics in continuous time. The proposed Hamiltonian framework, besides encompassing existing networks inspired by marginally stable ODEs, allows one to derive new and more expressive architectures. The good performance of the novel DNNs is demonstrated on benchmark classification problems, including digit recognition using the MNIST dataset.
Shanghai Jiao Tong University（上海交通大学）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
The past few years have witnessed an increased interest in learning Hamiltonian dynamics in deep learning frameworks. As an inductive bias based on physical laws, Hamiltonian dynamics endow neural networks with accurate long-term prediction, interpretability, and data-efficient learning. However, Hamiltonian dynamics also bring energy conservation or dissipation assumptions on the input data and additional computational overhead. In this paper, we systematically survey recently proposed Hamiltonian neural network models, with a special emphasis on methodologies. In general, we discuss the majo…
Graph neural networks (GNNs) have achieved success in various inference tasks on graph-structured data. However, common challenges faced by many GNNs in the literature include the problem of graph node embedding under various geometries and the over-smoothing problem. To address these issues, we propose a novel graph information propagation strategy called Hamiltonian Dynamic GNN (HDG) that uses a Hamiltonian mechanics approach to learn node embeddings in a graph. The Hamiltonian energy function in HDG is learnable and can adapt to the underlying geometry of any given graph dataset. We demonst…
Ecole Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
Deep neural networks can be fragile and sensitive to small input perturbations that might cause a significant change in the output. In this paper, we employ contraction theory to improve the robustness of neural ODEs (NODEs). A dynamical system is contractive if all solutions with different initial conditions converge to each other exponentially fast. As a consequence, perturbations in initial conditions become less and less relevant over time. Since in NODEs the input data corresponds to the initial condition of dynamical systems, we show contractivity can mitigate the effect of input perturb…
Symplectic Neural Networks for Learning Non-Separable Hamiltonians
Czech Technical University in Prague（捷克布拉格理工大学）；University of California at San Diego（加州大学圣地亚哥分校）；Athena Research and Innovation Center（雅典研究与创新中心）；HSBC Holdings（汇丰控股）
Learning partially observed systems with neural Hamiltonian ordinary differential equations
Department of Mathematics and Cybernetics, SINTEF Digital（数学与自动化系，SINTEF数字研究院）；Department of Physics, University of Oslo（物理系，奥斯陆大学）；Department of Data Science, Norwegian University of Life Science（数据科学系，挪威生命科学大学）
Port-Hamiltonian Neural Networks with Output Error Noise Models
Eindhoven University of Technology（埃因霍温理工大学）；Eindhoven Artificial Intelligence Systems Institute（埃因霍温人工智能系统研究所）；HUN-REN Institute for Computer Science and Control（匈牙利科学院计算机科学与控制研究所（HUN-REN））
A Hamiltonian driven Geometric Construction of Neural Networks via the Lognormal family, Application to Financial Fraud Detection and to Network Security
University of Yaounde1（雅温迪大学）；University of Maroua（马拉乌大学）；Higher national school of Polytechnic（国家高等理工学院）
Normalizing Flows (NF) are Generative models which transform a simple prior distribution into the desired target. They however require the design of an invertible mapping whose Jacobian determinant has to be computable. Recently introduced, Neural Hamiltonian Flows (NHF) are Hamiltonian dynamics-based flows, which are continuous, volume-preserving and invertible and thus make for natural candidates for robust NF architectures. In particular, their similarity to classical Mechanics could lead to easier interpretability of the learned mapping. In this paper, we show that the current NHF architec…
In this work, we propose a meta-learner based on ODE neural networks that learns gradients. This approach makes the optimizer is more flexible inducing an automatic inductive bias to the given task. Using the simplest Hamiltonian Neural Network we demonstrate that our method outperforms a meta-learner based on LSTM for an artificial task and the MNIST dataset with ReLU activations in the optimizee. Furthermore, it also surpasses the classic optimization methods for the artificial task and achieves comparable results for MNIST.
Many physical phenomena are described by Hamiltonian mechanics using an energy function (the Hamiltonian). Recently, the Hamiltonian neural network, which approximates the Hamiltonian as a neural network, and its extensions have attracted much attention. This is a very powerful method, but its use in theoretical studies remains limited. In this study, by combining the statistical learning theory and Kolmogorov-Arnold-Moser (KAM) theory, we provide a theoretical analysis of the behavior of Hamiltonian neural networks when the learning error is not completely zero. A Hamiltonian neural network w…
Understanding natural symmetries is key to making sense of our complex and ever-changing world. Recent work has shown that neural networks can learn such symmetries directly from data using Hamiltonian Neural Networks (HNNs). But HNNs struggle when trained on datasets where energy is not conserved. In this paper, we ask whether it is possible to identify and decompose conservative and dissipative dynamics simultaneously. We propose Dissipative Hamiltonian Neural Networks (D-HNNs), which parameterize both a Hamiltonian and a Rayleigh dissipation function. Taken together, they represent an impli…
Hybrid machine learning based on Hamiltonian formulations has recently been successfully demonstrated for simple mechanical systems, both energy conserving and not energy conserving. We introduce a pseudo-Hamiltonian formulation that is a generalization of the Hamiltonian formulation via the port-Hamiltonian formulation, and show that pseudo-Hamiltonian neural network models can be used to learn external forces acting on a system. We argue that this property is particularly useful when the external forces are state dependent, in which case it is the pseudo-Hamiltonian structure that facilitate…
There has been a wave of interest in applying machine learning to study dynamical systems. We present a Hamiltonian neural network that solves the differential equations that govern dynamical systems. This is an equation-driven machine learning method where the optimization process of the network depends solely on the predicted functions without using any ground truth data. The model learns solutions that satisfy, up to an arbitrarily small error, Hamilton's equations and, therefore, conserve the Hamiltonian invariants. The choice of an appropriate activation function drastically improves the …
Learning Stochastic Hamiltonian Systems via Stochastic Generating Function Neural Network
School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）；Department of Mathematics and Statistics, Auburn University（阿伯茨罕大学数学与统计学系）
Well-designed molecular representations (fingerprints) are vital to combine medical chemistry and deep learning. Whereas incorporating 3D geometry of molecules (i.e. conformations) in their representations seems beneficial, current 3D algorithms are still in infancy. In this paper, we propose a novel molecular representation algorithm which preserves 3D conformations of molecules with a Molecular Hamiltonian Network (HamNet). In HamNet, implicit positions and momentums of atoms in a molecule interact in the Hamiltonian Engine following the discretized Hamiltonian equations. These implicit coor…
Fudan University（复旦大学）；Shanghai Qizhi Institution（上海期智研究院）；Nankai University（南开大学）；University of Arkansas（阿肯色大学）
The effective spin Hamiltonian method is widely adopted to simulate and understand the behavior of magnetism. However, the magnetic interactions of some systems, such as itinerant magnets, are too complex to be described by any explicit function, which prevents an accurate description of magnetism in such systems. Here, we put forward a machine learning (ML) approach, applying an artificial neural network (ANN) and a local spin descriptor to develop effective spin potentials for any form of interaction. The constructed Hamiltonians include an explicit Heisenberg part and an implicit non-linear…
Arizona State University（亚利桑那州立大学）；CCDC Army Research Laboratory（美国陆军作战能力发展司令部陆军研究实验室）
The rapid growth of research in exploiting machine learning to predict chaotic systems has revived a recent interest in Hamiltonian Neural Networks (HNNs) with physical constraints defined by the Hamilton's equations of motion, which represent a major class of physics-enhanced neural networks. We introduce a class of HNNs capable of adaptable prediction of nonlinear physical systems: by training the neural network based on time series from a small number of bifurcation-parameter values of the target Hamiltonian system, the HNN can predict the dynamical states at other parameter values, where t…
Hamiltonian Monte Carlo (HMC) is a Markov chain Monte Carlo (MCMC) approach that exhibits favourable exploration properties in high-dimensional models such as neural networks. Unfortunately, HMC has limited use in large-data regimes and little work has explored suitable approaches that aim to preserve the entire Hamiltonian. In our work, we introduce a new symmetric integration scheme for split HMC that does not rely on stochastic gradients. We show that our new formulation is more efficient than previous approaches and is easy to implement with a single GPU. As a result, we are able to perfor…
University of Oxford（牛津大学）；Harvard University（哈佛大学）
Accurately learning the temporal behavior of dynamical systems requires models with well-chosen learning biases. Recent innovations embed the Hamiltonian and Lagrangian formalisms into neural networks and demonstrate a significant improvement over other approaches in predicting trajectories of physical systems. These methods generally tackle autonomous systems that depend implicitly on time or systems for which a control signal is known apriori. Despite this success, many real world dynamical systems are non-autonomous, driven by time-dependent forces and experience energy dissipation. In this…
We propose an effective and lightweight learning algorithm, Symplectic Taylor Neural Networks (Taylor-nets), to conduct continuous, long-term predictions of a complex Hamiltonian dynamic system based on sparse, short-term observations. At the heart of our algorithm is a novel neural network architecture consisting of two sub-networks. Both are embedded with terms in the form of Taylor series expansion designed with symmetric structure. The key mechanism underpinning our infrastructure is the strong expressiveness and special symmetric property of the Taylor series expansion, which naturally ac…
University of Michigan（密歇根大学）；University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；University of South Carolina（南卡罗来纳大学）
A new approach is introduced to classify faults in rotating machinery based on the total energy signature estimated from sensor measurements. The overall goal is to go beyond using black-box models and incorporate additional physical constraints that govern the behavior of mechanical systems. Observational data is used to train Hamiltonian neural networks that describe the conserved energy of the system for normal and various abnormal regimes. The estimated total energy function, in the form of the weights of the Hamiltonian neural network, serves as the new feature vector to discriminate betw…
Stanford University（斯坦福大学）；Google AI Perception（谷歌人工智能感知部）；Google Brain（谷歌大脑）
We present a general-purpose method to train Markov chain Monte Carlo kernels, parameterized by deep neural networks, that converge and mix quickly to their target distribution. Our method generalizes Hamiltonian Monte Carlo and is trained to maximize expected squared jumped distance, a proxy for mixing speed. We demonstrate large empirical gains on a collection of simple but challenging distributions, for instance achieving a 106x improvement in effective sample size in one case, and mixing when standard HMC makes no measurable progress in a second. Finally, we show quantitative and qualitati…
Phase space integrity in neural network models of Hamiltonian dynamics: A Lagrangian descriptor approach
Department of Mathematics, Khalifa University of Science and Technology（哈利法科学技术大学数学系）；Public Health and Epidemeology Department, Khalifa University of Science and Technology（哈利法科学技术大学公共卫生与流行病学系）
New York University（纽约大学）；Facebook AI Research（脸书人工智能研究院）；Tianjin University（天津大学）
We propose Symplectic Recurrent Neural Networks (SRNNs) as learning algorithms that capture the dynamics of physical systems from observed trajectories. An SRNN models the Hamiltonian function of the system by a neural network and furthermore leverages symplectic integration, multiple-step training and initial state optimization to address the challenging numerical issues associated with Hamiltonian systems. We show that SRNNs succeed reliably on complex and noisy Hamiltonian systems. We also show how to augment the SRNN integration scheme in order to handle stiff dynamical systems such as bou…
Learning Hamiltonian neural Koopman operator and simultaneously sustaining and discovering conservation law
Research Institute of Intelligent Complex Systems, Fudan University（复旦大学智能复杂系统研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；State Key Laboratory of Medical Neurobiology, Fudan University（复旦大学医学神经生物学国家重点实验室）
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
Large-scale cyber-physical systems require that control policies are distributed, that is, that they only rely on local real-time measurements and communication with neighboring agents. Optimal Distributed Control (ODC) problems are, however, highly intractable even in seemingly simple cases. Recent work has thus proposed training Neural Network (NN) distributed controllers. A main challenge of NN controllers is that they are not dependable during and after training, that is, the closed-loop system may be unstable, and the training may fail due to vanishing and exploding gradients. In this pap…
Universal Approximation Property of Hamiltonian Deep Neural Networks
Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Sapienza University of Rome（罗马大学）；Istituto di Analisi dei Sistemi e Informatica, Italian National Research Council（意大利国家研究委员会系统分析与信息研究所）
