Modeling time series with long- or short-memory characteristics is a fundamental challenge in many scientific and engineering domains. While fractional Brownian motion has been widely used as a noise source to capture such memory effects, its incompatibility with Itô calculus limits its applicability in neural stochastic differential equation~(SDE) frameworks. In this paper, we propose a novel class of noise, termed Neural Network-kernel ARMA-type noise~(NA-noise), which is an Itô-process-based alternative capable of capturing both long- and short-memory behaviors. The kernel function defining the noise structure is parameterized via neural networks and decomposed into a product form to preserve the Markov property. Based on this noise process, we develop NANSDE-Net, a generative model that extends Neural SDEs by incorporating NA-noise. We prove the theoretical existence and uniqueness of the solution under mild conditions and derive an efficient backpropagation scheme for training. Empirical results on both synthetic and real-world datasets demonstrate that NANSDE-Net matches or outperforms existing models, including fractional SDE-Net, in reproducing long- and short-memory features of the data, while maintaining computational tractability within the Itô calculus framework.
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
LTCI, Télécom ParisTech, Université Paris-Saclay(巴黎萨克雷大学巴黎高等电信学院LTCI实验室);Institute of Physics, École Polytechnique Fédérale de Lausanne(洛桑联邦理工学院物理研究所);Rutgers Business School(罗格斯商学院)
Variational Inference for SDEs Driven by Fractional Noise
Ghent University(根特大学);Flanders Make(法兰德斯制造研究院);Technical University of Berlin(柏林工业大学);University of Birmingham(伯明翰大学);University of Potsdam(波茨坦大学);Imperial College London(帝国理工学院)
On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
Télécom Paris, Institut Polytechnique de Paris(巴黎理工学院巴黎电信学院);University of Oxford(牛津大学);Rutgers Business School(罗格斯商学院);Facebook AI Research(脸书人工智能研究院)
Score-Based Generative Modeling through Stochastic Differential Equations
Stanford University(斯坦福大学);Google Brain(谷歌大脑)
从数据中创建噪声很容易;从噪声中创建数据是生成建模。我们提出了一个随机微分方程(RSDs),通过缓慢注入噪声,将复杂的数据分布平滑地转换为已知的先验分布,以及相应的逆时RSDs,通过缓慢去除噪声,将先验分布转换回数据分布。重要的是,逆时梯度仅取决于扰动数据分布的时间相关梯度场(\aka,分数)。通过利用基于分数的生成建模的进步,我们可以用神经网络准确地估计这些分数,并使用数值解算器来生成样本。我们表明,这个框架封装了以前的方法在基于分数的生成建模和扩散概率建模,允许新的采样程序和新的建模能力。特别是,我们引入了一个预测-校正框架来校正离散逆时模型演化中的错误。我们还推导出一个等价的神经ODE,它从与神经元相同的分布中采样,但另外还可以进行精确的似然计算,并提高采样效率。此外,我们提供了一种新的方法来解决逆问题的分数为基础的模型,类条件生成,图像修补和彩色化的实验证明。结合多个架构改进,我们在CIFAR-10上实现了破纪录的无条件图像生成性能,Inception评分为9.89,FID为2.20,竞争可能性为2.99 bits/dim,并首次从基于得分的生成模型中展示了1024 x 1024图像的高保真生成。
Trajectory Flow Matching with Applications to Clinical Time Series Modeling
McGill University(麦吉尔大学);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);Yale School of Medicine(耶鲁医学院);School of Clinical Medicine, University of Cambridge(剑桥大学临床医学院);Université de Montréal(蒙特利尔大学);CIFAR(加拿大高级研究院)
Hypergraph Neural Stochastic Diffusion: An SDE Framework for Uncertainty Estimation
School of Mathematics and Statistics, Shandong University, Weihai(山东大学(威海)数学与统计学院);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);University of Chinese Academy of Sciences(中国科学院大学)
Efficient CNN-LSTM based Parameter Estimation of Levy Driven Stochastic Differential Equations
School of Information Science(信息科学学院);Kyushu University(九州大学);Graduate School of Bioresource and Bioenvironmental Sciences(生物资源与生物环境科学研究院);Graduate School of Science and Technology(科学技术研究院);Joint Graduate School of Mathematics for Innovation(创新数学联合研究院)
Embedding stochastic dynamics in latent manifolds of recurrent neural networks
Mathematical Institute, University of Oxford(牛津大学数学研究所);Centre for Eudaimonia and Human Flourishing, University of Oxford(牛津大学幸福与人类繁荣中心);Complexity Science Hub, Vienna(维也纳复杂科学中心)
Hierarchical Stochastic Differential Equation Models for Latent Manifold Learning in Neural Time Series
Department of Biomedical Engineering(生物医学工程系);University of Houston(休斯顿大学);Department of Electrical and Computer Engineering(电气与计算机工程系);Isfahan University of Technology(伊斯法罕技术大学)
National University of Singapore(新加坡国立大学);Imperial College London(帝国理工学院伦敦分校);Department of Mathematics, Centre for Quantitative Finance, Risk Management Institute, National University of Singapore(新加坡国立大学数学系、量化金融中心、风险管理研究所)
Score-fPINN: Fractional Score-Based Physics-Informed Neural Networks for High-Dimensional Fokker-Planck-Levy Equations
National University of Singapore(新加坡国立大学);Worcester Polytechnic Institute(伍斯特理工学院);Brown University(布朗大学);Pacific Northwest National Laboratory(太平洋西北国家实验室)
ProGen: Revisiting Probabilistic Spatial-Temporal Time Series Forecasting from a Continuous Generative Perspective Using Stochastic Differential Equations
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))