It is critical yet challenging for deep learning models to properly characterize uncertainty that is pervasive in real-world environments. Although a lot of efforts have been made, such as heteroscedastic neural networks (HNNs), little work has demonstrated satisfactory practicability due to the different levels of compromise on learning efficiency, quality of uncertainty estimates, and predictive performance. Moreover, existing HNNs typically fail to construct an explicit interaction between the prediction and its associated uncertainty. This paper aims to remedy these issues by developing SDE-HNN, a new heteroscedastic neural network equipped with stochastic differential equations (SDE) to characterize the interaction between the predictive mean and variance of HNNs for accurate and reliable regression. Theoretically, we show the existence and uniqueness of the solution to the devised neural SDE. Moreover, based on the bias-variance trade-off for the optimization in SDE-HNN, we design an enhanced numerical SDE solver to improve the learning stability. Finally, to more systematically evaluate the predictive uncertainty, we present two new diagnostic uncertainty metrics. Experiments on the challenging datasets show that our method significantly outperforms the state-of-the-art baselines in terms of both predictive performance and uncertainty quantification, delivering well-calibrated and sharp prediction intervals.
University of California, San Diego（加利福尼亚大学圣迭戈分校）；Northeastern University（东北大学）
We introduce DeepGLEAM, a hybrid model for COVID-19 forecasting. DeepGLEAM combines a mechanistic stochastic simulation model GLEAM with deep learning. It uses deep learning to learn the correction terms from GLEAM, which leads to improved performance. We further integrate various uncertainty quantification methods to generate confidence intervals. We demonstrate DeepGLEAM on real-world COVID-19 mortality forecasting tasks.
Accurate and trustworthy epidemic forecasting is an important problem that has impact on public health planning and disease mitigation. Most existing epidemic forecasting models disregard uncertainty quantification, resulting in mis-calibrated predictions. Recent works in deep neural models for uncertainty-aware time-series forecasting also have several limitations; e.g. it is difficult to specify meaningful priors in Bayesian NNs, while methods like deep ensembling are computationally expensive in practice. In this paper, we fill this important gap. We model the forecasting task as a probabil…
Time series forecasting is widely used in the fields of equipment life cycle forecasting, weather forecasting, traffic flow forecasting, and other fields. Recently, some scholars have tried to apply Transformer to time series forecasting because of its powerful parallel training ability. However, the existing Transformer methods do not pay enough attention to the small time segments that play a decisive role in prediction, making it insensitive to small changes that affect the trend of time series, and it is difficult to effectively learn continuous time-dependent features. To solve this probl…
Data Science Institute, Imperial College London（伦敦帝国理工学院数据科学研究所）；Aquatic Informatics（Aquatic Informatics 公司）
Starting from the Kaya identity, we used a Neural ODE model to predict the evolution of several indicators related to carbon emissions, on a country-level: population, GDP per capita, energy intensity of GDP, carbon intensity of energy. We compared the model with a baseline statistical model - VAR - and obtained good performances. We conclude that this machine-learning approach can be used to produce a wide range of results and give relevant insight to policymakers
StoxLSTM: A Stochastic Extended Long Short-Term Memory Network for Time Series Forecasting
School of Information and Electronics（信息与电子学院）；Beijing Institute of Technology（北京理工大学）；Laboratory of Electromagnetic Space Cognition and Intelligent Control（电磁空间认知与智能控制实验室）；National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing（空间智能信息处理国家级重点实验室）
Stochastic Diffusion: A Diffusion Probabilistic Model for Stochastic Time Series Forecasting
School of Computing and Information Systems, The University of Melbourne（计算与信息系统学院，墨尔本大学）；Department of Surgery (Otolaryngology), The University of Melbourne（外科部门（耳鼻喉科），墨尔本大学）；School of Mathematics and Statistics, The University of Melbourne（数学与统计学学院，墨尔本大学）
WDformer: A Wavelet-based Differential Transformer Model for Time Series Forecasting
School of Computer Science and Technology, Zhejiang Normal University（浙江师范大学计算机科学与技术学院）；Zhejiang Key Laboratory of Intelligent Education Technology and Application, Zhejiang Normal University（浙江智能教育技术与应用重点实验室）
针对时间序列多域信息利用不足及传统注意力易引入噪声的问题，提出 WDformer，融合小波多分辨率分析、反转维度注意力与差分注意力，并在多个真实数据集上取得 SOTA 表现。
Time series forecasting based on deep architectures has been gaining popularity in recent years due to their ability to model complex non-linear temporal dynamics. The recurrent neural network is one such model capable of handling variable-length input and output. In this paper, we leverage recent advances in deep generative models and the concept of state space models to propose a stochastic adaptation of the recurrent neural network for multistep-ahead time series forecasting, which is trained with stochastic gradient variational Bayes. In our model design, the transition function of the rec…
ProGen: Revisiting Probabilistic Spatial-Temporal Time Series Forecasting from a Continuous Generative Perspective Using Stochastic Differential Equations
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Traffic forecasting is one of the most popular spatio-temporal tasks in the field of machine learning. A prevalent approach in the field is to combine graph convolutional networks and recurrent neural networks for the spatio-temporal processing. There has been fierce competition and many novel methods have been proposed. In this paper, we present the method of spatio-temporal graph neural controlled differential equation (STG-NCDE). Neural controlled differential equations (NCDEs) are a breakthrough concept for processing sequential data. We extend the concept and design two NCDEs: one for the…
Unveiling Delay Effects in Traffic Forecasting: A Perspective from Spatial-Temporal Delay Differential Equations
Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心）；Peking University（北京大学）；Institute of Zoology, Chinese Academy of Sciences（中国科学院动物研究所）；Terminus Group（特斯联科技集团）；University of Chinese Academy of Sciences（中国科学院大学）
Deep learning inspired by differential equations is a recent research trend and has marked the state of the art performance for many machine learning tasks. Among them, time-series modeling with neural controlled differential equations (NCDEs) is considered as a breakthrough. In many cases, NCDE-based models not only provide better accuracy than recurrent neural networks (RNNs) but also make it possible to process irregular time-series. In this work, we enhance NCDEs by redesigning their core part, i.e., generating a continuous path from a discrete time-series input. NCDEs typically use interp…
Xaurora: Generative Weather Forecasting with Denoising Stochastic Interpolants from a Foundation Model Prior
Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；University of Amsterdam（阿姆斯特丹大学）；Delft University of Technology（代尔夫特理工大学）；Centrum Wiskunde & Informatica（数学与计算机科学中心）
Huazhong University of Science and Technology（华中科技大学）
Despite the superiority of convolutional neural networks demonstrated in time series modeling and forecasting, it has not been fully explored on the design of the neural network architecture and the tuning of the hyper-parameters. Inspired by the incremental construction strategy for building a random multilayer perceptron, we propose a novel Error-feedback Stochastic Modeling (ESM) strategy to construct a random Convolutional Neural Network (ESM-CNN) for time series forecasting task, which builds the network architecture adaptively. The ESM strategy suggests that random filters and neurons of…
Neural networks inspired by differential equations have proliferated for the past several years. Neural ordinary differential equations (NODEs) and neural controlled differential equations (NCDEs) are two representative examples of them. In theory, NCDEs provide better representation learning capability for time-series data than NODEs. In particular, it is known that NCDEs are suitable for processing irregular time-series data. Whereas NODEs have been successfully extended after adopting attention, however, it had not been studied yet how to integrate attention into NCDEs. To this end, we pres…
Evaluation of a Foundational Model and Stochastic Models for Forecasting Sporadic or Spiky Production Outages of High-Performance Machine Learning Services
Differential Convolutional Fuzzy Time Series Forecasting
University of Electronic Science and Technology of China（电子科技大学）；Huawei Cloud（华为云）；China Mobile Information Technology Center（中国移动信息技术中心）；Vanderbilt University（范德堡大学）
Huazhong University of Science and Technology（华中科技大学）；Illinois Institute of Technology（伊利诺伊理工学院）；China Securities Co., Ltd（中信建投证券股份有限公司）
In this article, we employ a collection of stochastic differential equations with drift and diffusion coefficients approximated by neural networks to predict the trend of chaotic time series which has big jump properties. Our contributions are, first, we propose a model called Lévy induced stochastic differential equation network, which explores compounded stochastic differential equations with $α$-stable Lévy motion to model complex time series data and solve the problem through neural network approximation. Second, we theoretically prove that the numerical solution through our algorithm conv…
Synergistic Neural Forecasting of Air Pollution with Stochastic Sampling
Mohamed bin Zayed University of Artificial Intelligence（莫德尔·本·扎耶德人工智能大学）；Weizmann Institute of Science（魏兹曼科学研究所）；Linköping University（林雪平大学）；Australian National University（澳大利亚国立大学）
A broad class of stochastic volatility models are defined by systems of stochastic differential equations. While these models have seen widespread success in domains such as finance and statistical climatology, they typically lack an ability to condition on historical data to produce a true posterior distribution. To address this fundamental limitation, we show how to re-cast a class of stochastic volatility models as a hierarchical Gaussian process (GP) model with specialized covariance functions. This GP model retains the inductive biases of the stochastic volatility model while providing th…
Spatio-temporal stochastic graph-based learning for infectious disease forecasting
School of Public Health and Social Work, Queensland University of Technology（昆士兰理工大学公共卫生与社会科学学院）；QUT Centre for Data Science, Queensland University of Technology（昆士兰理工大学数据科学中心）
Differential-Integral Neural Operator for Long-Term Turbulence Forecasting
Tsinghua University（清华大学）；University of Science and Technology of China（中国科学技术大学）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）；Tencent（腾讯）
Stochastic Volatility Modelling with LSTM Networks: A Hybrid Approach for S&P 500 Index Volatility Forecasting
University of Warsaw（华沙大学）；Faculty of Economic Sciences（经济学院）；Department of Quantitative Finance and Machine Learning（金融与机器学习系）；Quantitative Finance Research Group（金融研究组）
Kate Gleason College of Engineering, Rochester Institute of Technology, Rochester, USA；Computer Science \& Engineering, Lehigh University, Bethlehem, USA
Recent years have seen a surge in interest in building deep learning-based fully data-driven models for weather prediction. Such deep learning models if trained on observations can mitigate certain biases in current state-of-the-art weather models, some of which stem from inaccurate representation of subgrid-scale processes. However, these data-driven models, being over-parameterized, require a lot of training data which may not be available from reanalysis (observational data) products. Moreover, an accurate, noise-free, initial condition to start forecasting with a data-driven weather model …
Deep learning has an increasing impact to assist research, allowing, for example, the discovery of novel materials. Until now, however, these artificial intelligence techniques have fallen short of discovering the full differential equation of an experimental physical system. Here we show that a dynamical neural network, trained on a minimal amount of data, can predict the behavior of spintronic devices with high accuracy and an extremely efficient simulation time, compared to the micromagnetic simulations that are usually employed to model them. For this purpose, we re-frame the formalism of …
Neural differential equations predict the derivative of a stochastic process. This allows irregular forecasting with arbitrary time-steps. However, the expressive temporal flexibility often comes with a high sensitivity to noise. In addition, current methods model measurements and control together, limiting generalization to different control policies. These properties severely limit applicability to medical treatment problems, which require reliable forecasting given high noise, limited data and changing treatment policies. We introduce the Neural Eigen-SDE algorithm (NESDE), which relies on …
Probabilistic Forecasting for Dynamical Systems with Missing or Imperfect Data
Institute of Applied Mathematics（应用数学研究所）；University of British Columbia（不列颠哥伦比亚大学）；Recherche en prévision numérique atmosphérique（大气数值预报研究中心）；Environnement et Changement climatique Canada（加拿大环境与气候变化部）
Pacific Northwest National Laboratory（太平洋西北国家实验室）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Real-time state estimation and forecasting is critical for efficient operation of power grids. In this paper, a physics-informed Gaussian process regression (PhI-GPR) method is presented and used for probabilistic forecasting and estimating the phase angle, angular speed, and wind mechanical power of a three-generator power grid system using sparse measurements. In standard data-driven Gaussian process regression (GPR), parameterized models for the prior statistics are fit by maximizing the marginal likelihood of observed data, whereas in PhI-GPR, we compute the prior statistics by solving sto…
