This is a revised version of a chapter written for the Handbook of Computational Statistics, edited by J. Gentle, W. Hardle and Y. Mori in 2003, in preparation for the second edition
In this chapter, we will first present the most standard computational challenges met in Bayesian Statistics, focussing primarily on mixture estimation and on model choice issues, and then relate these problems with computational solutions. Of course, this chapter is only a terse introduction to the problems and solutions related to Bayesian computations. For more complete references, see Robert and Casella (2004, 2009), or Marin and Robert (2007), among others. We also restrain from providing an introduction to Bayesian Statistics per se and for comprehensive coverage, address the reader to Robert (2007), (again) among others.
This survey covers state-of-the-art Bayesian techniques for the estimation of mixtures. It complements the earlier Marin, Mengersen and Robert (2005) by studying new types of distributions, the multinomial, latent class and t distributions. It also exhibits closed form solutions for Bayesian inference in some discrete setups. Lastly, it sheds a new light on the computation of Bayes factors via the approximation of Chib (1995).
This chapter provides a overview of Bayesian inference, mostly emphasising that it is a universal method for summarising uncertainty and making estimates and predictions using probability statements conditional on observed data and an assumed model (Gelman 2008). The Bayesian perspective is thus applicable to all aspects of statistical inference, while being open to the incorporation of information items resulting from earlier experiments and from expert opinions. We provide here the basic elements of Bayesian analysis when considered for standard models, refering to Marin and Robert (2007) an…
This chapter surveys advances in the field of Bayesian computation over the past twenty years, with missing data. It also contains some novel computational entries on the double-exponential model that may be of interest per se.
The INLA package provides a tool for computationally efficient Bayesian modeling and inference for various widely used models, more formally the class of latent Gaussian models. It is a non-sampling based framework which provides approximate results for Bayesian inference, using sparse matrices. The swift uptake of this framework for Bayesian modeling is rooted in the computational efficiency of the approach and catalyzed by the demand presented by the big data era. In this paper, we present new developments within the INLA package with the aim to provide a computationally efficient mechanism …
Many psychological theories can be operationalized as linear inequality constraints on the parameters of multinomial distributions (e.g., discrete choice analysis). These constraints can be described in two equivalent ways: Either as the solution set to a system of linear inequalities or as the convex hull of a set of extremal points (vertices). For both representations, we describe a general Gibbs sampler for drawing posterior samples in order to carry out Bayesian analyses. We also summarize alternative sampling methods for estimating Bayes factors for these model representations using the e…
Clinical outcome prediction from high-dimensional data is problematic in the common setting where there is only a relatively small number of samples. The imbalance causes data overfitting, and outcome prediction becomes computationally expensive or even impossible. We propose a Bayesian outcome prediction method that can be applied to data of arbitrary dimension d, from 2 outcome classes, and reduces overfitting without any approximations at parameter level. This is achieved by avoiding numerical integration or approximation, and solving the Bayesian integrals analytically. We thereby reduce t…
Gaussian time-series models are often specified through their spectral density. Such models present several computational challenges, in particular because of the non-sparse nature of the covariance matrix. We derive a fast approximation of the likelihood for such models. We propose to sample from the approximate posterior (that is, the prior times the approximate likelihood), and then to recover the exact posterior through importance sampling. We show that the variance of the importance sampling weights vanishes as the sample size goes to infinity. We explain why the approximate posterior may…
There is an increasing amount of literature focused on Bayesian computational methods to address problems with intractable likelihood. One approach is a set of algorithms known as Approximate Bayesian Computational (ABC) methods. One of the problems of these algorithms is that the performance depends on the tuning of some parameters, such as the summary statistics, distance and tolerance level. To bypass this problem, Mengersen, Pudlo and Robert (2013) introduced an alternative method based on empirical likelihood, which can be easily implemented when a set of constraints, related to the momen…
The Bayesian statistical paradigm uses the language of probability to express uncertainty about the phenomena that generate observed data. Probability distributions thus characterize Bayesian analysis, with the rules of probability used to transform prior probability distributions for all unknowns - parameters, latent variables, models - into posterior distributions, subsequent to the observation of data. Conducting Bayesian analysis requires the evaluation of integrals in which these probability distributions appear. Bayesian computation is all about evaluating such integrals in the typical c…
We combine conditioning techniques with sparse grid quadrature rules to develop a computationally efficient method to approximate marginal, but not necessarily univariate, posterior quantities, yielding approximate Bayesian inference via Sparse grid Quadrature Evaluation (BISQuE) for hierarchical models. BISQuE reformulates posterior quantities as weighted integrals of conditional quantities, such as densities and expectations. Sparse grid quadrature rules allow computationally efficient approximation of high dimensional integrals, which appear in hierarchical models with many hyperparameters.…
Performing optimal Bayesian design for discriminating between competing models is computationally intensive as it involves estimating posterior model probabilities for thousands of simulated datasets. This issue is compounded further when the likelihood functions for the rival models are computationally expensive. A new approach using supervised classification methods is developed to perform Bayesian optimal model discrimination design. This approach requires considerably fewer simulations from the candidate models than previous approaches using approximate Bayesian computation. Further, it is…
This paper focuses on utilizing two different Bayesian methods to deal with a variety of toy problems which occur in data analysis. In particular we implement the Variational Bayesian and Nested Sampling methods to tackle the problems of polynomial selection and Gaussian Mixture Models, comparing the algorithms in terms of processing speed and accuracy. In the problems tackled here it is the Variational Bayesian algorithms which are the faster though both results give similar results.
In this chapter we review some of the most recent computational advances in the rapidly expanding field of statistical social network analysis using the R open-source software. In particular we will focus on Bayesian estimation for two important families of models: exponential random graph models (ERGMs) and latent space models (LSMs).
In this note, we shortly survey some recent approaches on the approximation of the Bayes factor used in Bayesian hypothesis testing and in Bayesian model choice. In particular, we reassess importance sampling, harmonic mean sampling, and nested sampling from a unified perspective.
This article is a discussion of Zanella and Roberts' paper: Multilevel linear models, gibbs samplers and multigrid decompositions. We consider several extensions in which the multigrid decomposition would bring us interesting insights, including vector hierarchical models, linear mixed effects models and partial centering parametrizations.
This chapter surveys the most standard Monte Carlo methods available for simulating from a posterior distribution associated with a mixture and conducts some experiments about the robustness of the Gibbs sampler in high dimensional Gaussian settings. This is a chapter prepared for the forthcoming 'Handbook of Mixture Analysis'.
Also known as likelihood-free methods, approximate Bayesian computational (ABC) methods have appeared in the past ten years as the most satisfactory approach to untractable likelihood problems, first in genetics then in a broader spectrum of applications. However, these methods suffer to some degree from calibration difficulties that make them rather volatile in their implementation and thus render them suspicious to the users of more traditional Monte Carlo methods. In this survey, we study the various improvements and extensions made to the original ABC algorithm over the recent years.
Bayesian statistics has gained great momentum since the computational developments of the 1990s. Gradually, advances in Bayesian methodology and software have made Bayesian techniques much more accessible to applied statisticians and, in turn, have potentially transformed Bayesian education at the undergraduate level. This article provides an overview on the various options for implementing Bayesian computational methods motivated to achieve particular learning outcomes. The advantages and disadvantages of each computational method are described based on the authors' experience in using these …
A brief comment on A variational Bayesian approach for inverse problems with skew-t error distributions (Guha et al., Journal of Computational Physics 301 (2015) 377-393) is given in this letter.
Gaussian processes allow for flexible specification of prior assumptions of unknown dynamics in state space models. We present a procedure for efficient Bayesian learning in Gaussian process state space models, where the representation is formed by projecting the problem onto a set of approximate eigenfunctions derived from the prior covariance structure. Learning under this family of models can be conducted using a carefully crafted particle MCMC algorithm. This scheme is computationally efficient and yet allows for a fully Bayesian treatment of the problem. Compared to conventional system id…
Approximate Bayesian computation (ABC) methods perform inference on model-specific parameters of mechanistically motivated parametric statistical models when evaluating likelihoods is difficult. Central to the success of ABC methods is computationally inexpensive simulation of data sets from the parametric model of interest. However, when simulating data sets from a model is so computationally expensive that the posterior distribution of parameters cannot be adequately sampled by ABC, inference is not straightforward. We present approximate approximate Bayesian computation" (AABC), a class of …
Bayesian methodologies prioritising accurate associations above sparsity in Gaussian graphical model (GGM) estimation remain relatively scarce in scientific literature. It is well accepted that the $\ell_2$ penalty enjoys a smaller computational footprint in GGM estimation, whilst the $\ell_1$ penalty encourages sparsity in the estimand. The Bayesian adaptive graphical lasso prior is used as a departure point in the formulation of a computationally efficient graphical ridge-type prior for events where accurate associations are prioritised over sparse representations. A novel block Gibbs sample…
The generation of decision-theoretic Bayesian optimal designs is complicated by the significant computational challenge of minimising an analytically intractable expected loss function over a, potentially, high-dimensional design space. A new general approach for approximately finding Bayesian optimal designs is proposed which uses computationally efficient normal-based approximations to posterior summaries to aid in approximating the expected loss. This new approach is demonstrated on illustrative, yet challenging, examples including hierarchical models for blocked experiments, and experiment…
Powerful ideas recently appeared in the literature are adjusted and combined to design improved samplers for Bayesian exponential random graph models. Different forms of adaptive Metropolis-Hastings proposals (vertical, horizontal and rectangular) are tested and combined with the Delayed rejection (DR) strategy with the aim of reducing the variance of the resulting Markov chain Monte Carlo estimators for a given computational time. In the examples treated in this paper the best combination, namely horizontal adaptation with delayed rejection, leads to a variance reduction that varies between 9…
Assessing variability according to distinct factors in data is a fundamental technique of statistics. The method commonly regarded to as analysis of variance (ANOVA) is, however, typically confined to the case where all levels of a factor are present in the data (i.e. the population of factor levels has been exhausted). Random and mixed effects models are used for more elaborate cases, but require distinct nomenclature, concepts and theory, as well as distinct inferential procedures. Following a hierarchical Bayesian approach, a comprehensive ANOVA framework is shown, which unifies the above s…
It has become commonplace to use complex computer models to predict outcomes in regions where data does not exist. Typically these models need to be calibrated and validated using some experimental data, which often consists of multiple correlated outcomes. In addition, some of the model parameters may be categorical in nature, such as a pointer variable to alternate models (or submodels) for some of the physics of the system. Here we present a general approach for calibration in such situations where an emulator of the computationally demanding models and a discrepancy term from the model to …
Bayesian inference for models with intractable likelihood functions represents a challenging suite of problems in modern statistics. In this work we analyse the Conway-Maxwell-Poisson (COM-Poisson) distribution, a two parameter generalisation of the Poisson distribution. COM-Poisson regression modelling allows the flexibility to model dispersed count data as part of a generalised linear model (GLM) with a COM-Poisson response, where exogenous covariates control the mean and dispersion level of the response. The major difficulty with COM-Poisson regression is that the likelihood function contai…
We study the computational complexity of Markov chain Monte Carlo (MCMC) methods for high-dimensional Bayesian linear regression under sparsity constraints. We first show that a Bayesian approach can achieve variable-selection consistency under relatively mild conditions on the design matrix. We then demonstrate that the statistical criterion of posterior concentration need not imply the computational desideratum of rapid mixing of the MCMC algorithm. By introducing a truncated sparsity prior for variable selection, we provide a set of conditions that guarantee both variable-selection consiste…
Likelihood-free Bayesian inference algorithms are popular methods for calibrating the parameters of complex, stochastic models, required when the likelihood of the observed data is intractable. These algorithms characteristically rely heavily on repeated model simulations. However, whenever the computational cost of simulation is even moderately expensive, the significant burden incurred by likelihood-free algorithms leaves them unviable in many practical applications. The multifidelity approach has been introduced (originally in the context of approximate Bayesian computation) to reduce the s…
Learning the structure of Bayesian networks from data is known to be a computationally challenging, NP-hard problem. The literature has long investigated how to perform structure learning from data containing large numbers of variables, following a general interest in high-dimensional applications ("small n, large p") in systems biology and genetics. More recently, data sets with large numbers of observations (the so-called "big data") have become increasingly common; and these data sets are not necessarily high-dimensional, sometimes having only a few tens of variables depending on the applic…
We present a Bayesian nonparametric system reliability model which scales well and provides a great deal of flexibility in modeling. The Bayesian approach naturally handles the disparate amounts of component and subsystem data that may exist. However, traditional Bayesian reliability models are quite computationally complex, relying on MCMC techniques. Our approach utilizes the conjugate properties of the beta-Stacy process, which is the fundamental building block of our model. These individual models are linked together using a method of moments estimation approach. This model is computationa…
Statistical estimates from survey samples have traditionally been obtained via design-based estimators. In many cases, these estimators tend to work well for quantities such as population totals or means, but can fall short as sample sizes become small. In today's "information age," there is a strong demand for more granular estimates. To meet this demand, using a Bayesian pseudo-likelihood, we propose a computationally efficient unit-level modeling approach for non-Gaussian data collected under informative sampling designs. Specifically, we focus on binary and multinomial data. Our approach i…
Computer experiments can emulate the physical systems, help computational investigations, and yield analytic solutions. They have been widely employed with many engineering applications (e.g., aerospace, automotive, energy systems. Conventional Bayesian optimization did not incorporate the nested structures in computer experiments. This paper proposes a novel nested Bayesian optimization for complex computer experiments with multi-step or hierarchical characteristics. We prove the theoretical properties of nested outputs given two cases: Gaussian or non-Gaussian. The closed forms of nested exp…
This paper develops a matrix-variate adaptive Markov chain Monte Carlo (MCMC) methodology for Bayesian Cointegrated Vector Auto Regressions (CVAR). We replace the popular approach to sampling Bayesian CVAR models, involving griddy Gibbs, with an automated efficient alternative, based on the Adaptive Metropolis algorithm of Roberts and Rosenthal, (2009). Developing the adaptive MCMC framework for Bayesian CVAR models allows for efficient estimation of posterior parameters in significantly higher dimensional CVAR series than previously possible with existing griddy Gibbs samplers. For a n-dimens…
In this paper we review recently developed methods for nonparametric Bayesian inference for one-dimensional diffusion models. We discuss different possible prior distributions, computational issues, and asymptotic results.
Implementing Bayesian inference is often computationally challenging in applications involving complex models, and sometimes calculating the likelihood itself is difficult. Synthetic likelihood is one approach for carrying out inference when the likelihood is intractable, but it is straightforward to simulate from the model. The method constructs an approximate likelihood by taking a vector summary statistic as being multivariate normal, with the unknown mean and covariance matrix estimated by simulation for any given parameter value. Our article makes three contributions. The first shows that…
Uncertainty-Aware Surrogate-based Amortized Bayesian Inference for Computationally Expensive Models
Department of Stochastic Simulation and Safety Research for Hydrosystems（随机模拟与安全研究部门）；Cluster of Excellence SimTech（卓越中心SimTech）；University of Stuttgart（斯图加特大学）；Department of Statistics（统计学系）；TU Dortmund University（多特蒙德技术大学）
This paper addresses the Bayesian calibration of dynamic models with parametric and structural uncertainties, in particular where the uncertain parameters are unknown/poorly known spatio-temporally varying subsystem models. Independent stationary Gaussian processes with uncertain hyper-parameters describe uncertainties of the model structure and parameters while Karhunnen-Loeve expansion is adopted to spectrally represent these Gaussian processes. The Karhunnen-Loeve expansion of a prior Gaussian process is projected on a generalized Polynomial Chaos basis, whereas intrusive Galerkin projectio…
Statistical modeling of point patterns is an important and common problem in several areas. The Poisson process is the most common process used for this purpose, in particular, its generalization that considers the intensity function to be stochastic. This is called a Cox process and different choices to model the dynamics of the intensity gives rise to a wide range of models. We present a new class of unidimensional Cox process models in which the intensity function assumes parametric functional forms that switch among them according to a continuous-time Markov chain. A novel methodology is p…
Probabilistic record linkage (PRL) is the process of determining which records in two databases correspond to the same underlying entity in the absence of a unique identifier. Bayesian solutions to this problem provide a powerful mechanism for propagating uncertainty due to uncertain links between records (via the posterior distribution). However, computational considerations severely limit the practical applicability of existing Bayesian approaches. We propose a new computational approach, providing both a fast algorithm for deriving point estimates of the linkage structure that properly acco…
This paper considers the computational power of constant size, dynamic Bayesian networks. Although discrete dynamic Bayesian networks are no more powerful than hidden Markov models, dynamic Bayesian networks with continuous random variables and discrete children of continuous parents are capable of performing Turing-complete computation. With modified versions of existing algorithms for belief propagation, such a simulation can be carried out in real time. This result suggests that dynamic Bayesian networks may be more powerful than previously considered. Relationships to causal models and rec…
We present a continuation method that entails generating a sequence of transition probability density functions from the prior to the posterior in the context of Bayesian inference for parameter estimation problems. The characterization of transition distributions, by tempering the likelihood function, results in a homogeneous nonlinear partial integro-differential equation whose existence and uniqueness of solutions are addressed. The posterior probability distribution comes as the interpretation of the final state of the path of transition distributions. A computationally stable scaling doma…
In the 1960s, Shiryaev developed a Bayesian theory of change-point detection in the i.i.d. case, which was generalized in the beginning of the 2000s by Tartakovsky and Veeravalli for general stochastic models assuming a certain stability of the log-likelihood ratio process. Hidden Markov models represent a wide class of stochastic processes that are very useful in a variety of applications. In this paper, we investigate the performance of the Bayesian Shiryaev change-point detection rule for hidden Markov models. We propose a set of regularity conditions under which the Shiryaev procedure is f…
Approximate Bayesian computation (ABC), also known as likelihood-free methods, have become a favourite tool for the analysis of complex stochastic models, primarily in population genetics but also in financial analyses. We advocated in Grelaud et al. (2009) the use of ABC for Bayesian model choice in the specific case of Gibbs random fields (GRF), relying on a sufficiency property mainly enjoyed by GRFs to show that the approach was legitimate. Despite having previously suggested the use of ABC for model choice in a wider range of models in the DIY ABC software (Cornuet et al., 2008), we prese…
