Inspired by the ubiquitous use of differential equations to model continuous dynamics across diverse scientific and engineering domains, we propose a novel and intuitive approach to continuous sequence modeling. Our method interprets time-series data as \textit{discrete samples from an underlying continuous dynamical system}, and models its time evolution using Neural Stochastic Differential Equation (Neural SDE), where both the flow (drift) and diffusion terms are parameterized by neural networks. We derive a principled maximum likelihood objective and a \textit{simulation-free} scheme for efficient training of our Neural SDE model. We demonstrate the versatility of our approach through experiments on sequence modeling tasks across both embodied and generative AI. Notably, to the best of our knowledge, this is the first work to show that SDE-based continuous-time modeling also excels in such complex scenarios, and we hope that our work opens up new avenues for research of SDE models in high-dimensional and temporally intricate domains.
Disentangling Continuous-Time Latent Dynamics: Identifiability of Latent SDEs via Diffusion Shifts
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);The University of Melbourne(墨尔本大学);Peking University(北京大学);Carnegie Mellon University(卡内基梅隆大学)
AReS and MaRS - Adversarial and MMD-Minimizing Regression for SDEs
University of Oxford(牛津大学);ETH Zürich(苏黎世联邦理工学院);Max Planck ETH Center for Learning Systems(马克斯·普朗克-苏黎世联邦理工学习系统中心);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所)
Anomaly Detection for Sparse and Irregular Multivariate Time Series with Latent SDEs
Josef Ressel Centre for Intelligent and Secure Industrial Automation, University of Applied Sciences, Salzburg, Austria(约瑟夫·雷斯尔智能与安全工业自动化中心,应用科学大学,萨尔茨堡,奥地利);University of Salzburg, Austria(萨尔茨堡大学,奥地利)
Efficient Training of Neural SDEs Using Stochastic Optimal Control
Ghent University(根特大学);Dept. of Theor. Comp. Science(理论计算机科学系);Technical University of Berlin(柏林技术大学);Inst. of Mathematics(数学研究所);Centre for Sys. Modelling and Quant. Biomed(系统建模与定量生物医学中心);Dept. of Computing(计算系)
Variational Inference for SDEs Driven by Fractional Noise
Ghent University(根特大学);Flanders Make(法兰德斯制造研究院);Technical University of Berlin(柏林工业大学);University of Birmingham(伯明翰大学);University of Potsdam(波茨坦大学);Imperial College London(帝国理工学院)