We propose heavy ball neural ordinary differential equations (HBNODEs), leveraging the continuous limit of the classical momentum accelerated gradient descent, to improve neural ODEs (NODEs) training and inference. HBNODEs have two properties that imply practical advantages over NODEs: (i) The adjoint state of an HBNODE also satisfies an HBNODE, accelerating both forward and backward ODE solvers, thus significantly reducing the number of function evaluations (NFEs) and improving the utility of the trained models. (ii) The spectrum of HBNODEs is well structured, enabling effective learning of long-term dependencies from complex sequential data. We verify the advantages of HBNODEs over NODEs on benchmark tasks, including image classification, learning complex dynamics, and sequential modeling. Our method requires remarkably fewer forward and backward NFEs, is more accurate, and learns long-term dependencies more effectively than the other ODE-based neural network models. Code is available at \url{https://github.com/hedixia/HeavyBallNODE}.
Accelerated Gossip via Stochastic Heavy Ball Method
School of Mathematics, KAUST, Saudi Arabia（卡士塔大学数学学院，沙特阿拉伯）；The University of Edinburgh（爱丁堡大学）；The University of Edinburgh, United Kingdom（英国爱丁堡大学）；Edinburgh, Scotland, UK（苏格兰爱丁堡，英国）；MIPT, Russia（莫斯科国立信息安全大学，俄罗斯）
On the fast convergence of minibatch heavy ball momentum
Operations Research and Industrial Engineering（运筹学与工业工程）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Mathematics（数学）；New York University（纽约大学）；Computer Science and Engineering（计算机科学与工程）
Learning POD of Complex Dynamics Using Heavy-ball Neural ODEs
University of Utah（犹他大学）
适当的正交分解（POD）允许在一个实质性的水平上的复杂的动力系统的降阶建模，同时保持在建模的基础动力系统的高度准确性。机器学习算法的进步使得能够从数据中学习基于POD的动态，并对动态系统进行准确快速的预测。在本文中，我们利用最近提出的重球神经ODE（HBNODE）[Xia et al. NeurIPS，2021]在POD上下文中学习数据驱动的降阶降阶模型（ROM），特别是学习时变系数的动态特性，这些时变系数是通过对求解全阶模型生成的训练快照进行POD分析而生成的。HBNODE在学习基于POD的ROM方面具有几个实际优势，并具有理论保证，包括1）HBNODE可以从顺序观察中有效地学习长期依赖关系，2）HBNODE在训练和测试方面都具有计算效率。我们比较HBNODE与其他流行的ROM上的几个复杂的动力系统，包括冯卡门街流，Kurganov-Petrova-Popov方程，和一维欧拉方程的流体建模。
Heavy-Ball Momentum Accelerated Actor-Critic With Function Approximation
Guangdong-Hong Kong-Macao Joint Laboratory for Emotional Intelligence and Pervasive Computing, Shenzhen MSU-BIT University（深圳北理莫斯科大学粤港澳情感智能与普适计算联合实验室）；University of Science and Technology Beijing（北京科技大学）；Beijing Institute of Technology（北京理工大学）
