There has been an increasing interest from the scientific community in using likelihood-free inference (LFI) to determine which parameters of a given simulator model could best describe a set of experimental data. Despite exciting recent results and a wide range of possible applications, an important bottleneck of LFI when applied to time series data is the necessity of defining a set of summary features, often hand-tailored based on domain knowledge. In this work, we present a data-driven strategy for automatically learning summary features from univariate time series and apply it to signals generated from autoregressive-moving-average (ARMA) models and the Van der Pol Oscillator. Our results indicate that learning summary features from data can compete and even outperform LFI methods based on hand-crafted values such as autocorrelation coefficients even in the linear case.
This study addresses the problem of learning an extended summary causal graph on time series. The algorithms we propose fit within the well-known constraint-based framework for causal discovery and make use of information-theoretic measures to determine (in)dependencies between time series. We first introduce generalizations of the causation entropy measure to any lagged or instantaneous relations, prior to using this measure to construct extended summary causal graphs by adapting two well-known algorithms, namely PC and FCI. The behavior of our methods is illustrated through several experimen…
Cyber-physical systems often consist of entities that interact with each other over time. Meanwhile, as part of the continued digitization of industrial processes, various sensor technologies are deployed that enable us to record time-varying attributes (a.k.a., time series) of such entities, thus producing correlated time series. To enable accurate forecasting on such correlated time series, this paper proposes two models that combine convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The first model employs a CNN on each individual time series, combines the convoluted…
Accelerometry data has been widely used to measure activity and the circadian rhythm of individuals across the health sciences, in particular with people with advanced dementia. Modern accelerometers can record continuous observations on a single individual for several days at a sampling frequency of the order of one hertz. Such rich and lengthy data sets provide new opportunities for statistical insight, but also pose challenges in selecting from a wide range of possible summary statistics, and how the calculation of such statistics should be optimally tuned and implemented. In this paper, we…
University of Oxford（牛津大学）；Improbable（Improbable公司）；Durham University（杜伦大学）
Simulation models of complex dynamics in the natural and social sciences commonly lack a tractable likelihood function, rendering traditional likelihood-based statistical inference impossible. Recent advances in machine learning have introduced novel algorithms for estimating otherwise intractable likelihood functions using a likelihood ratio trick based on binary classifiers. Consequently, efficient likelihood approximations can be obtained whenever good probabilistic classifiers can be constructed. We propose a kernel classifier for sequential data using path signatures based on the recently…
Department of Medicine, University of Chicago（芝加哥大学医学院）
Recognizing subtle historical patterns is central to modeling and forecasting problems in time series analysis. Here we introduce and develop a new approach to quantify deviations in the underlying hidden generators of observed data streams, resulting in a new efficiently computable universal metric for time series. The proposed metric is in the sense that we can compare and contrast data streams regardless of where and how they are generated and without any feature engineering step. The approach proposed in this paper is conceptually distinct from our previous work on data smashing, and vastl…
A new forecasting method based on the concept of the profile predictive the likelihood function is proposed for discrete-valued processes. In particular, generalized autoregressive and moving average (GARMA) models for Poisson distributed data are explored in details. Highest density regions are used to construct forecasting regions. The proposed forecast estimates and regions are coherent. Large sample results are derived for the forecasting distribution. Numerical studies using simulations and a real data set are used to establish the performance of the proposed forecasting method. Robustnes…
Empirical likelihood method has been applied to short-memory time series models by Monti (1997) through the Whittle's estimation method. Yau (2012) extended this idea to long-memory time series models. Asymptotic distributions of the empirical likelihood ratio statistic for short and long-memory time series have been derived to construct confidence regions for the corresponding model parameters. However, computing profile empirical likelihood function involving constrained maximization does not always have a solution which leads to several drawbacks. In this paper, we propose an adjusted empir…
This paper presents a general theoretical framework of penalized quasi-maximum likelihood (PQML) estimation in stationary multiple time series models when the number of parameters possibly diverges. We show the oracle property of the PQML estimator under high-level, but tractable, assumptions, comprising the first half of the paper. Utilizing these results, we propose in the latter half of the paper a method of sparse estimation in high-dimensional vector autoregressive (VAR) models. Finally, the usability of the sparse high-dimensional VAR model is confirmed with a simulation study and an emp…
Neural composite likelihood estimation: simulation based inference for time series
University of Bristol（布里斯托大学）；School of Mathematics, University of Bristol（布里斯托大学数学学院）；School of Biological Sciences, University of Bristol（布里斯托大学生物科学学院）
Empirical likelihood method has been applied to dependent observations by Monti (1997) through the Whittle's estimation method. Similar asymptotic distribution of the empirical likelihood ratio statistic for stationary time series has been derived to construct the confidence regions for the parameters. However, required numerical problem of computing profile empirical likelihood function which involves constrained maximization has no solution sometimes, which leads to the drawbacks of using the original version of the empirical likelihood ratio. In this paper, we propose an adjusted empirical …
Identifiability of total effects from abstractions of time series causal graphs
Sorbonne Université（索邦大学）；INSERM（法国国家健康与医学研究院）；Institut Pierre Louis d’Epidémiologie et de Santé Publique（皮埃尔·路易流行病学与公共卫生研究所）；EasyVista（易视塔公司）；Univ Grenoble Alpes（格勒诺布尔大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；LIG（信息与信号处理实验室）；INRIA（法国国家信息与自动化研究所）；Université of Rennes 2（雷恩第二大学）
Estimators derived from score functions that are not the likelihood are in wide use in practical and modern applications. Their regularization is often carried by pseudo-posterior estimation, equivalently by adding penalty to the score function. We argue that this approach is suboptimal, and propose a two-staged alternative involving estimation of a new score function which better approximates the true likelihood for the purpose of regularization. Our approach typically identifies with maximum a-posteriori estimation if the original score function is in fact the likelihood. We apply our theory…
The goodness of the long-term prediction in the state-space model was evaluated using the squared long-term prediction error. In order to estimate the model parameters suitable for long-term prediction, we devised a modified log-likelihood corresponding to the long-term prediction error variance. Trend models and seasonally adjusted models with and without AR component are examined as examples.
The problem of dealing with misreported data is very common in a wide range of contexts for different reasons. The current situation caused by the Covid-19 worldwide pandemic is a clear example, where the data provided by official sources were not always reliable due to data collection issues and to the high proportion of asymptomatic cases. In this work, we explore the performance of Bayesian Synthetic Likelihood to estimate the parameters of a model capable of dealing with misreported information and to reconstruct the most likely evolution of the phenomenon. The performance of the proposed …
The upper bounds on the coverage probabilities of the confidence regions based on blockwise empirical likelihood [Kitamura (1997)] and nonstandard expansive empirical likelihood [Nordman et al. (2013)] methods for time series data are investigated via studying the probability for the violation of the convex hull constraint. The large sample bounds are derived on the basis of the pivotal limit of the blockwise empirical log-likelihood ratio obtained under the fixed-b asymptotics, which has been recently shown to provide a more accurate approximation to the finite sample distribution than the co…
Functional time series whose sample elements are recorded sequentially over time are frequently encountered with increasing technology. Recent studies have shown that analyzing and forecasting of functional time series can be performed easily using functional principal component analysis and existing univariate/multivariate time series models. However, the forecasting performance of such functional time series models may be affected by the presence of outlying observations which are very common in many scientific fields. Outliers may distort the functional time series model structure, and thus…
Mixed Time Series Quasi-Likelihood Models for Uncovering Covid-19 Viral Load and Mortality Dynamics
Loyola University Chicago（芝加哥洛约拉大学）；Southern Methodist University（南方卫理公会大学）；Queen Mary University of London（伦敦大学玛丽女王学院）；University College Dublin（都柏林大学学院）
We propose a new class of univariate nonstationary time series models, using the framework of modulated time series, which is appropriate for the analysis of rapidly-evolving time series as well as time series observations with missing data. We extend our techniques to a class of bivariate time series that are isotropic. Exact inference is often not computationally viable for time series analysis, and so we propose an estimation method based on the Whittle-likelihood, a commonly adopted pseudo-likelihood. Our inference procedure is shown to be consistent under standard assumptions, as well as …
We consider the problem of fast time-series data clustering. Building on previous work modeling the correlation-based Hamiltonian of spin variables we present an updated fast non-expensive Agglomerative Likelihood Clustering algorithm (ALC). The method replaces the optimized genetic algorithm based approach (f-SPC) with an agglomerative recursive merging framework inspired by previous work in Econophysics and Community Detection. The method is tested on noisy synthetic correlated time-series data-sets with built-in cluster structure to demonstrate that the algorithm produces meaningful non-tri…
The state-space model and the Kalman filter provide us with unified and computationaly efficient procedure for computing the log-likelihood of the diverse type of time series models. This paper presents an algorithm for computing the gradient and the Hessian matrix of the log-likelihood by extending the Kalman filter without resorting to the numerical difference. Different from the previous paper(Kitagawa 2020), it is assumed that the observation noise variance R=1. It is known that for univariate time series, by maximizing the log-likelihood of this restricted model, we can obtain the same es…
This paper is devoted to the off-line multiple change-point detection in a semiparametric framework. The time series is supposed to belong to a large class of models including AR($\infty$), ARCH($\infty$), TARCH($\infty$),... models where the coefficients change at each instant of breaks. The different unknown parameters (number of changes, change dates and parameters of successive models) are estimated using a penalized contrast built on conditional quasi-likelihood. Under Lipshitzian conditions on the model, the consistency of the estimator is proved when the moment order $r$ of the process …
In this paper, we use convolutional neural networks to address the problem of model identification for autoregressive moving average time series models. We compare the performance of several neural network architectures, trained on simulated time series, with likelihood based methods, in particular the Akaike and Bayesian information criteria. We find that our neural networks can significantly outperform these likelihood based methods in terms of accuracy and, by orders of magnitude, in terms of speed.
Bayesian neural networks (BNNs) can account for both aleatoric and epistemic uncertainty. However, in BNNs the priors are often specified over the weights which rarely reflects true prior knowledge in large and complex neural network architectures. We present a simple approach to incorporate prior knowledge in BNNs based on external summary information about the predicted classification probabilities for a given dataset. The available summary information is incorporated as augmented data and modeled with a Dirichlet process, and we derive the corresponding \emph{Summary Evidence Lower BOund}. …
In time series analysis there is an apparent dichotomy between time and frequency domain methods. The aim of this paper is to draw connections between frequency and time domain methods. Our focus will be on reconciling the Gaussian likelihood and the Whittle likelihood. We derive an exact, interpretable, bound between the Gaussian and Whittle likelihood of a second order stationary time series. The derivation is based on obtaining the transformation which is biorthogonal to the discrete Fourier transform of the time series. Such a transformation yields a new decomposition for the inverse of a …
Standard blockwise empirical likelihood (BEL) for stationary, weakly dependent time series requires specifying a fixed block length as a tuning parameter for setting confidence regions. This aspect can be difficult and impacts coverage accuracy. As an alternative, this paper proposes a new version of BEL based on a simple, though nonstandard, data-blocking rule which uses a data block of every possible length. Consequently, the method does not involve the usual block selection issues and is also anticipated to exhibit better coverage performance. Its nonstandard blocking scheme, however, induc…
The aim of this paper is to compare numerically the performance of two estimators based on Hyvärinen's local homogeneous scoring rule with that of the full and the pairwise maximum likelihood estimators. In particular, two different model settings, for which both full and pairwise maximum likelihood estimators can be obtained, have been considered: the first order autoregressive model (AR(1)) and the moving average model (MA(1)). Simulation studies highlight very different behaviours for the Hyvärinen scoring rule estimators relative to the pairwise likelihood estimators in these two settings.
Information from frequency bands in biomedical time series provides useful summaries of the observed signal. Many existing methods consider summaries of the time series obtained over a few well-known, pre-defined frequency bands of interest. However, these methods do not provide data-driven methods for identifying frequency bands that optimally summarize frequency-domain information in the time series. A new method to identify partition points in the frequency space of a multivariate locally stationary time series is proposed. These partition points signify changes across frequencies in the ti…
This paper considers a general class of parameter-driven models for time series of counts. A comprehensive simulation study is conducted to evaluate the accuracy and efficiency of three estimators: the maximum likelihood estimators of the generalized linear model, 2-state finite mixture model, and 2-state hidden Markov model. Standard errors for these estimators are derived. Our results show that except in extreme cases, the maximum likelihood estimator of the generalized linear model is an efficient, consistent and robust estimator with a well-behaved estimated standard error. The maximum lik…
Graphical interaction models have become an important tool for analysing multivariate time series. In these models, the interrelationships among the components of a time series are described by undirected graphs in which the vertices depict the components while the edges indictate possible dependencies between the components. Current methods for the identification of the graphical structure are based on nonparametric spectral stimation, which prevents application of common model selection strategies. In this paper, we present a parametric approach for graphical interaction modelling of multiva…
We consider the problem of parametric statistical inference when likelihood computations are prohibitively expensive but sampling from the model is possible. Several so-called likelihood-free methods have been developed to perform inference in the absence of a likelihood function. The popular synthetic likelihood approach infers the parameters by modelling summary statistics of the data by a Gaussian probability distribution. In another popular approach called approximate Bayesian computation, the inference is performed by identifying parameter values for which the summary statistics of the si…
Likelihood-free methods are an essential tool for performing inference for implicit models which can be simulated from, but for which the corresponding likelihood is intractable. However, common likelihood-free methods do not scale well to a large number of model parameters. A promising approach to high-dimensional likelihood-free inference involves estimating low-dimensional marginal posteriors by conditioning only on summary statistics believed to be informative for the low-dimensional component, and then combining the low-dimensional approximations in some way. In this paper, we demonstrate…
Parametric statistical models that are implicitly defined in terms of a stochastic data generating process are used in a wide range of scientific disciplines because they enable accurate modeling. However, learning the parameters from observed data is generally very difficult because their likelihood function is typically intractable. Likelihood-free Bayesian inference methods have been proposed which include the frameworks of approximate Bayesian computation (ABC), synthetic likelihood, and its recent generalization that performs likelihood-free inference by ratio estimation (LFIRE). A major …
Likelihood-free methods are an established approach for performing approximate Bayesian inference for models with intractable likelihood functions. However, they can be computationally demanding. Bayesian synthetic likelihood (BSL) is a popular such method that approximates the likelihood function of the summary statistic with a known, tractable distribution -- typically Gaussian -- and then performs statistical inference using standard likelihood-based techniques. However, as the number of summary statistics grows, the number of model simulations required to accurately estimate the covariance…
We introduce a framework using Generative Adversarial Networks (GANs) for likelihood--free inference (LFI) and Approximate Bayesian Computation (ABC) where we replace the black-box simulator model with an approximator network and generate a rich set of summary features in a data driven fashion. On benchmark data sets, our approach improves on others with respect to scalability, ability to handle high dimensional data and complex probability distributions.
Many scientifically well-motivated statistical models in natural, engineering and environmental sciences are specified through a generative process, but in some cases it may not be possible to write down a likelihood for these models analytically. Approximate Bayesian computation (ABC) methods, which allow Bayesian inference in these situations, are typically computationally intensive. Recently, computationally attractive empirical likelihood based ABC methods have been suggested in the literature. These methods heavily rely on the availability of a set of suitable analytically tractable estim…
Uppsala University（乌普萨拉大学）；University of California-Santa Barbara（加州大学圣塔芭芭拉分校）；Umeå University（于默奥大学）
State-of-the-art neural network-based methods for learning summary statistics have delivered promising results for simulation-based likelihood-free parameter inference. Existing approaches require density estimation as a post-processing step building upon deterministic neural networks, and do not take network prediction uncertainty into account. This work proposes a robust integrated approach that learns summary statistics using Bayesian neural networks, and directly estimates the posterior density using categorical distributions. An adaptive sampling scheme selects simulation locations to eff…
Informatics Institute（信息学研究所）；University of Amsterdam（阿姆斯特丹大学）；Donald Bren School of Information and Computer Sciences（唐纳德·布伦信息与计算机科学学院）；University of California, Irvine（加州大学欧文分校）；Canadian Institute for Advanced Research（加拿大高等研究院）
We describe an embarrassingly parallel, anytime Monte Carlo method for likelihood-free models. The algorithm starts with the view that the stochasticity of the pseudo-samples generated by the simulator can be controlled externally by a vector of random numbers u, in such a way that the outcome, knowing u, is deterministic. For each instantiation of u we run an optimization procedure to minimize the distance between summary statistics of the simulator and the data. After reweighing these samples using the prior and the Jacobian (accounting for the change of volume in transforming from the space…
Simulation-based Bayesian inference with ameliorative learned summary statistics -- Part I
Department of Electrical & Computer Engineering, College of Engineering, Physics, and Computing, The Catholic University of America（电气与计算机工程系，工程、物理与计算学院，天主教美国大学）
Performing exact posterior inference in complex generative models is often difficult or impossible due to an expensive to evaluate or intractable likelihood function. Approximate Bayesian computation (ABC) is an inference framework that constructs an approximation to the true likelihood based on the similarity between the observed and simulated data as measured by a predefined set of summary statistics. Although the choice of appropriate problem-specific summary statistics crucially influences the quality of the likelihood approximation and hence also the quality of the posterior sample in ABC…
École polytechnique fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
Fitting network models to neural activity is an important tool in neuroscience. A popular approach is to model a brain area with a probabilistic recurrent spiking network whose parameters maximize the likelihood of the recorded activity. Although this is widely used, we show that the resulting model does not produce realistic neural activity. To correct for this, we suggest to augment the log-likelihood with terms that measure the dissimilarity between simulated and recorded activity. This dissimilarity is defined via summary statistics commonly used in neuroscience and the optimization is eff…
Approximate Bayesian Computation (ABC) are likelihood-free Monte Carlo methods. ABC methods use a comparison between simulated data, using different parameters drew from a prior distribution, and observed data. This comparison process is based on computing a distance between the summary statistics from the simulated data and the observed data. For complex models, it is usually difficult to define a methodology for choosing or constructing the summary statistics. Recently, a nonparametric ABC has been proposed, that uses a dissimilarity measure between discrete distributions based on empirical …
Many models of interest in the natural and social sciences have no closed-form likelihood function, which means that they cannot be treated using the usual techniques of statistical inference. In the case where such models can be efficiently simulated, Bayesian inference is still possible thanks to the Approximate Bayesian Computation (ABC) algorithm. Although many refinements have been suggested, ABC inference is still far from routine. ABC is often excruciatingly slow due to very low acceptance rates. In addition, ABC requires introducing a vector of "summary statistics", the choice of which…
University of California, Berkeley（加州大学伯克利分校）；University of Warwick（华威大学）；Swarthmore College（斯沃斯莫尔学院）
An explosion of high-throughput DNA sequencing in the past decade has led to a surge of interest in population-scale inference with whole-genome data. Recent work in population genetics has centered on designing inference methods for relatively simple model classes, and few scalable general-purpose inference techniques exist for more realistic, complex models. To achieve this, two inferential challenges need to be addressed: (1) population data are exchangeable, calling for methods that efficiently exploit the symmetries of the data, and (2) computing likelihoods is intractable as it requires …
We consider the Bayesian calibration of models describing the phenomenon of block copolymer (BCP) self-assembly using image data produced by microscopy or X-ray scattering techniques. To account for the random long-range disorder in BCP equilibrium structures, we introduce auxiliary variables to represent this aleatory uncertainty. These variables, however, result in an integrated likelihood for high-dimensional image data that is generally intractable to evaluate. We tackle this challenging Bayesian inference problem using a likelihood-free approach based on measure transport together with th…
