Autoregressive models (ARMs) currently hold state-of-the-art performance in likelihood-based modeling of image and audio data. Generally, neural network based ARMs are designed to allow fast inference, but sampling from these models is impractically slow. In this paper, we introduce the predictive sampling algorithm: a procedure that exploits the fast inference property of ARMs in order to speed up sampling, while keeping the model intact. We propose two variations of predictive sampling, namely sampling with ARM fixed-point iteration and learned forecasting modules. Their effectiveness is demonstrated in two settings: i) explicit likelihood modeling on binary MNIST, SVHN and CIFAR10, and ii) discrete latent modeling in an autoencoder trained on SVHN, CIFAR10 and Imagenet32. Empirically, we show considerable improvements over baselines in number of ARM inference calls and sampling speed.
Although Transformer has made breakthrough success in widespread domains especially in Natural Language Processing (NLP), applying it to time series forecasting is still a great challenge. In time series forecasting, the autoregressive decoding of canonical Transformer models could introduce huge accumulative errors inevitably. Besides, utilizing Transformer to deal with spatial-temporal dependencies in the problem still faces tough difficulties.~To tackle these limitations, this work is the first attempt to propose a Non-Autoregressive Transformer architecture for time series forecasting, aim…
Department of Computer Science, ETH Zürich（苏黎世联邦理工学院计算机科学系）；Department of Computer Science, ETH Zürich, Switzerland（苏黎世联邦理工学院计算机科学系）
We develop an effective generation of adversarial attacks on neural models that output a sequence of probability distributions rather than a sequence of single values. This setting includes the recently proposed deep probabilistic autoregressive forecasting models that estimate the probability distribution of a time series given its past and achieve state-of-the-art results in a diverse set of application domains. The key technical challenge we address is effectively differentiating through the Monte-Carlo estimation of statistics of the joint distribution of the output sequence. Additionally,…
University of Tübingen（图宾根大学）；Amazon Research（亚马逊研究院）；Technical University of Munich（慕尼黑工业大学）
Despite the increasing relevance of forecasting methods, causal implications of these algorithms remain largely unexplored. This is concerning considering that, even under simplifying assumptions such as causal sufficiency, the statistical risk of a model can differ significantly from its \textit{causal risk}. Here, we study the problem of \textit{causal generalization} -- generalizing from the observational to interventional distributions -- in forecasting. Our goal is to find answers to the question: How does the efficacy of an autoregressive (VAR) model in predicting statistical association…
Probabilistic forecasting, i.e. estimating the probability distribution of a time series' future given its past, is a key enabler for optimizing business processes. In retail businesses, for example, forecasting demand is crucial for having the right inventory available at the right time at the right place. In this paper we propose DeepAR, a methodology for producing accurate probabilistic forecasts, based on training an auto regressive recurrent network model on a large number of related time series. We demonstrate how by applying deep learning techniques to forecasting, one can overcome many…
We introduce Hyper-Conditioned Neural Autoregressive Flow (HCNAF); a powerful universal distribution approximator designed to model arbitrarily complex conditional probability density functions. HCNAF consists of a neural-net based conditional autoregressive flow (AF) and a hyper-network that can take large conditions in non-autoregressive fashion and outputs the network parameters of the AF. Like other flow models, HCNAF performs exact likelihood inference. We conduct a number of density estimation tasks on toy experiments and MNIST to demonstrate the effectiveness and attributes of HCNAF, in…
Toronto Metropolitan University（多伦多都会大学）；University of Toronto（多伦多大学）
Despite the high performance of neural network-based time series forecasting methods, the inherent challenge in explaining their predictions has limited their applicability in certain application areas. Due to the difficulty in identifying causal relationships between the input and output of such black-box methods, they rarely have been adopted in domains such as legal and medical fields in which the reliability and interpretability of the results can be essential. In this paper, we propose \model, a novel deep learning-based probabilistic time series forecasting architecture that is intrinsic…
Time series prediction has been a long-standing research topic and an essential application in many domains. Modern time series collected from sensor networks (e.g., energy consumption and traffic flow) are often large-scale and incomplete with considerable corruption and missing values, making it difficult to perform accurate predictions. In this paper, we propose a low-rank autoregressive tensor completion (LATC) framework to model multivariate time series data. The key of LATC is to transform the original multivariate time series matrix (e.g., sensor$\times$time point) to a third-order tens…
Universidad Carlos III de Madrid（马德里卡洛斯三世大学）；Health Research Institute Gregorio Marañón（格雷戈里奥·马拉尼翁健康研究所）；CIBERSAM, ISCIII（西班牙生物医学研究网络中心（卡洛斯三世健康研究所））
Time series forecasting is an important problem across many domains, playing a crucial role in multiple real-world applications. In this paper, we propose a forecasting architecture that combines deep autoregressive models with a Spectral Attention (SA) module, which merges global and local frequency domain information in the model's embedded space. By characterizing in the spectral domain the embedding of the time series as occurrences of a random process, our method can identify global trends and seasonality patterns. Two spectral attention models, global and local to the time series, integr…
KAIROS: Unified Training for Universal Non-Autoregressive Time Series Forecasting
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Department of Mathematical Sciences, Durham University（杜伦大学数学科学系）；University of Chinese Academy of Sciences（中国科学院大学）
In this work, we propose \texttt{TimeGrad}, an autoregressive model for multivariate probabilistic time series forecasting which samples from the data distribution at each time step by estimating its gradient. To this end, we use diffusion probabilistic models, a class of latent variable models closely connected to score matching and energy-based methods. Our model learns gradients by optimizing a variational bound on the data likelihood and at inference time converts white noise into a sample of the distribution of interest through a Markov chain using Langevin sampling. We demonstrate experi…
Autoregressive with Slack Time Series Model for Forecasting a Partially-Observed Dynamical Time Series
The Institute of Statistical Mathematics（统计数理研究所）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Kyoto University（京都大学）；Hitotsubashi University（一桥大学）
As a result of the greater availability of big data, as well as the decreasing costs and increasing power of modern computing, the use of artificial neural networks for financial time series forecasting is once again a major topic of discussion and research in the financial world. Despite this academic focus, there are still contrasting opinions and bodies of literature on which artificial neural networks perform the best and whether or not they outperform the forecasting capabilities of conventional time series models. This paper uses a long-short term memory network to perform financial time…
ARMA Block: A CNN-Based Autoregressive and Moving Average Module for Long-Term Time Series Forecasting
AI Lab, Grepp（Grepp人工智能实验室）；Department of Computer Science and Engineering, Hankuk University of Foreign Studies（韩国外国语大学计算机科学与工程系）；Department of Radiation Physics, The University of Texas MD Anderson Cancer Center（德克萨斯MD安德森癌症中心辐射物理系）
Hainan University（海南大学）；University of Chinese Academy of Science（中国科学院大学）
Traffic flow forecasting is hot spot research of intelligent traffic system construction. The existing traffic flow prediction methods have problems such as poor stability, high data requirements, or poor adaptability. In this paper, we define the traffic data time singularity ratio in the dropout module and propose a combination prediction method based on the improved long short-term memory neural network and time series autoregressive integrated moving average model (SDLSTM-ARIMA), which is derived from the Recurrent Neural Networks (RNN) model. It compares the traffic data time singularity …
LMU Munich（慕尼黑大学）；RWTH Aachen（亚琛工业大学）；ETH Zurich（苏黎世联邦理工学院）；University of Goettingen（哥廷根大学）；University of Zurich（苏黎世大学）
Probabilistic forecasting of time series is an important matter in many applications and research fields. In order to draw conclusions from a probabilistic forecast, we must ensure that the model class used to approximate the true forecasting distribution is expressive enough. Yet, characteristics of the model itself, such as its uncertainty or its feature-outcome relationship are not of lesser importance. This paper proposes Autoregressive Transformation Models (ATMs), a model class inspired by various research directions to unite expressive distributional forecasts using a semi-parametric di…
This paper studies some temporal dependence properties and addresses the issue of parametric estimation for a class of state-dependent autoregressive models for nonlinear time series in which we assume a stochastic autoregressive coefficient depending on the first lagged value of the process itself. We call such a model state-dependent first-order autoregressive process, (SDAR). We introduce some assumptions under which this class of models is strictly stationary and uniformly ergodic and we establish consistency and asymptotic normality of the quasi-maximum likelihood estimator of the paramet…
In this paper, we propose multi-variable LSTM capable of accurate forecasting and variable importance interpretation for time series with exogenous variables. Current attention mechanism in recurrent neural networks mostly focuses on the temporal aspect of data and falls short of characterizing variable importance. To this end, the multi-variable LSTM equipped with tensorized hidden states is developed to learn hidden states for individual variables, which give rise to our mixture temporal and variable attention. Based on such attention mechanism, we infer and quantify variable importance. Ext…
Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Renmin University of China（中国人民大学）
Transformer-based methods have shown great potential in long-term time series forecasting. However, most of these methods adopt the standard point-wise self-attention mechanism, which not only becomes intractable for long-term forecasting since its complexity increases quadratically with the length of time series, but also cannot explicitly capture the predictive dependencies from contexts since the corresponding key and value are transformed from the same point. This paper proposes a predictive Transformer-based model called {\em Preformer}. Preformer introduces a novel efficient {\em Multi-S…
We propose three methods for forecasting a time series modeled using a functional coefficient autoregressive model (FCAR) fit via spline-backfitted local linear (SBLL) smoothing. The three methods are a "naive" plug-in method, a bootstrap method, and a multistage method. We present asymptotic results of the SBLL estimation method for FCAR models and show the estimators are oracally efficient. The three forecasting methods are compared through simulation. We find that the naive method performs just as well as the multistage method and even outperforms it in some situations. We apply the naive a…
Time series forecasting is often fundamental to scientific and engineering problems and enables decision making. With ever increasing data set sizes, a trivial solution to scale up predictions is to assume independence between interacting time series. However, modeling statistical dependencies can improve accuracy and enable analysis of interaction effects. Deep learning methods are well suited for this problem, but multivariate models often assume a simple parametric distribution and do not scale to high dimensions. In this work we model the multivariate temporal dynamics of time series via a…
INESC TEC（INESC技术研究所）；University of Porto（波尔图大学）；Dalhousie University（达尔豪斯大学）
Time series forecasting is one of the most active research topics. Machine learning methods have been increasingly adopted to solve these predictive tasks. However, in a recent work, these were shown to systematically present a lower predictive performance relative to simple statistical methods. In this work, we counter these results. We show that these are only valid under an extremely low sample size. Using a learning curve method, our results suggest that machine learning methods improve their relative predictive performance as the sample size grows. The code to reproduce the experiments is…
The purpose of this paper is to propose a time-varying vector autoregressive model (TV-VAR) for forecasting multivariate time series. The model is casted into a state-space form that allows flexible description and analysis. The volatility covariance matrix of the time series is modelled via inverted Wishart and singular multivariate beta distributions allowing a fully conjugate Bayesian inference. Model performance and model comparison is done via the likelihood function, sequential Bayes factors, the mean of squared standardized forecast errors, the mean of absolute forecast errors (known al…
Lung tumor tracking for radiotherapy requires real-time, multiple-step ahead forecasting of a quasi-periodic time series recording instantaneous tumor locations. We introduce a location-mixture autoregressive (LMAR) process that admits multimodal conditional distributions, fast approximate inference using the EM algorithm and accurate multiple-step ahead predictive distributions. LMAR outperforms several commonly used methods in terms of out-of-sample prediction accuracy using clinical data from lung tumor patients. With its superior predictive performance and real-time computation, the LMAR m…
This study was conducted to find an appropriate statistical model to forecast the volatilities of PSEi using the model Generalized Autoregressive Conditional Heteroskedasticity (GARCH). Using the R software, the log returns of PSEi is modeled using various ARIMA models and with the presence of heteroskedasticity, the log returns was modeled using GARCH. Based on the analysis, GARCH models are the most appropriate to use for the log returns of PSEi. Among the selected GARCH models, GARCH (1,2) has the lowest AIC value and also has the highest LL value implying that GARCH (1,2) is the best model…
A Hybridizable Neural Time Integrator for Stable Autoregressive Forecasting
Mechanical Engineering and Applied Mechanics, University of Pennsylvania（宾夕法尼亚大学机械工程与应用力学系）；Department of Mathematics, Tufts University（塔夫茨大学数学系）；Department of Mathematics and Cybernetics, SINTEF Digital（SINTEF数字技术部门数学与控制系）；Department of Mathematical Sciences, Norwegian University of Science and Technology（挪威科技大学数学科学系）；Department of Physics, University of Oslo（奥斯陆大学物理系）；Sandia National Laboratories（桑地亚国家实验室）
Time series models aim for accurate predictions of the future given the past, where the forecasts are used for important downstream tasks like business decision making. In practice, deep learning based time series models come in many forms, but at a high level learn some continuous representation of the past and use it to output point or probabilistic forecasts. In this paper, we introduce a novel autoregressive architecture, VQ-AR, which instead learns a \emph{discrete} set of representations that are used to predict the future. Extensive empirical comparison with other competitive deep learn…
Nantes University（南特大学）；LS2N / UMR CNRS 6004（LS2N / 法国国家科研中心UMR 6004）
Time series subject to change in regime have attracted much interest in domains such as econometry, finance or meteorology. For discrete-valued regimes, some models such as the popular Hidden Markov Chain (HMC) describe time series whose state process is unknown at all time-steps. Sometimes, time series are firstly labelled thanks to some annotation function. Thus, another category of models handles the case with regimes observed at all time-steps. We present a novel model which addresses the intermediate case: (i) state processes associated to such time series are modelled by Partially Hidden…
AR-KAN: Autoregressive-Weight-Enhanced Kolmogorov-Arnold Network for Time Series Forecasting
School of Information Science and Engineering, Southeast University（东南大学信息科学与工程学院）；School of Economics and Management, Southeast University（东南大学经济管理学院）
We develop a Bayesian median autoregressive (BayesMAR) model for time series forecasting. The proposed method utilizes time-varying quantile regression at the median, favorably inheriting the robustness of median regression in contrast to the widely used mean-based methods. Motivated by a working Laplace likelihood approach in Bayesian quantile regression, BayesMAR adopts a parametric model bearing the same structure as autoregressive models by altering the Gaussian error to Laplace, leading to a simple, robust, and interpretable modeling strategy for time series forecasting. We estimate model…
A Predictive Approach To Enhance Time-Series Forecasting
Department of Electrical and Computer Engineering（电气与计算机工程系）；University of California Santa Cruz（加州大学圣克鲁兹分校）；Friedrich Miescher Institute for Biomedical Research（弗里德里希·迈斯切尔生物医学研究中心）；Institut de Neurosciences de la Timone Aix Marseille Univ CNRS（里沃利神经科学研究所）；School of Biomedical Engineering The University of Sydney（悉尼大学生物医学工程学院）
Knowledge of the current state of economies, how they respond to COVID-19 mitigations and indicators, and what the future might hold for them is important. We use recently-developed generalised network autoregressive (GNAR) models, using trade-determined networks, to model and forecast the Purchasing Managers' Indices for a number of countries. We use networks that link countries where the links themselves, or their weights, are determined by the degree of export trade between the countries. We extend these models to include node-specific time series exogenous variables (GNARX models), using t…
The forecasting of multi-variate time processes through graph-based techniques has recently been addressed under the graph signal processing framework. However, problems in the representation and the processing arise when each time series carries a vector of quantities rather than a scalar one. To tackle this issue, we devise a new framework and propose new methodologies based on the graph vector autoregressive model. More explicitly, we leverage product graphs to model the high-dimensional graph data and develop multi-dimensional graph-based vector autoregressive models to forecast future tre…
This paper introduces a new parsimonious structure for mixture of autoregressive models. the weighting coefficients are determined through latent random variables, following a hidden Markov model. We propose a dynamic programming algorithm for the application of forecasting. We also derive the limiting behavior of unconditional first moment of the process and an appropriate upper bound for the limiting value of the variance. This can be considered as long run behavior of the process. Finally we show convergence and stability of the second moment. Further, we illustrate the efficacy of the prop…
We present a new method for forecasting systems of multiple interrelated time series. The method learns the forecast models together with discovering leading indicators from within the system that serve as good predictors improving the forecast accuracy and a cluster structure of the predictive tasks around these. The method is based on the classical linear vector autoregressive model (VAR) and links the discovery of the leading indicators to inferring sparse graphs of Granger causality. We formulate a new constrained optimisation problem to promote the desired sparse structures across the mod…
Linear autoregressive models serve as basic representations of discrete time stochastic processes. Different attempts have been made to provide non-linear versions of the basic autoregressive process, including different versions based on kernel methods. Motivated by the powerful framework of Hilbert space embeddings of distributions, in this paper we apply this methodology for the kernel embedding of an autoregressive process of order $p$. By doing so, we provide a non-linear version of an autoregressive process, that shows increased performance over the linear model in highly complex time se…
A new forecasting method based on the concept of the profile predictive the likelihood function is proposed for discrete-valued processes. In particular, generalized autoregressive and moving average (GARMA) models for Poisson distributed data are explored in details. Highest density regions are used to construct forecasting regions. The proposed forecast estimates and regions are coherent. Large sample results are derived for the forecasting distribution. Numerical studies using simulations and a real data set are used to establish the performance of the proposed forecasting method. Robustnes…
