Poisson log-linear models are ubiquitous in many applications, and one of the most popular approaches for parametric count regression. In the Bayesian context, however, there are no sufficient specific computational tools for efficient sampling from the posterior distribution of parameters, and standard algorithms, such as random walk Metropolis-Hastings or Hamiltonian Monte Carlo algorithms, are typically used. Herein, we developed an efficient Metropolis-Hastings algorithm and importance sampler to simulate from the posterior distribution of the parameters of Poisson log-linear models under conditional Gaussian priors with superior performance with respect to the state-of-the-art alternatives. The key for both algorithms is the introduction of a proposal density based on a Gaussian approximation of the posterior distribution of parameters. Specifically, our result leverages the negative binomial approximation of the Poisson likelihood and the successful Pólya-gamma data augmentation scheme. Via simulation, we obtained that the time per independent sample of the proposed samplers is competitive with that obtained using the successful Hamiltonian Monte Carlo sampling, with the Metropolis-Hastings showing superior performance in all scenarios considered.
We recently proposed a general algorithm for approximating nonstandard Bayesian posterior distributions by minimization of their Kullback-Leibler divergence with respect to a more convenient approximating distribution. In this note we offer details on how to efficiently implement this algorithm in practice. We also suggest default choices for the form of the posterior approximation, the number of iterations, the step size, and other user choices. By using these defaults it becomes possible to construct good posterior approximations for hierarchical models completely automatically.
The logistic specification has been used extensively in non-Bayesian statistics to model the dependence of discrete outcomes on the values of specified covariates. Because the likelihood function is globally weakly concave estimation by maximum likelihood is generally straightforward even in commonly arising applications with scores or hundreds of parameters. In contrast Bayesian inference has proven awkward, requiring normal approximations to the likelihood or specialized adaptations of existing Markov chain Monte Carlo and data augmentation methods. This paper approaches Bayesian inference i…
Let $X$ have a Generalized Poisson distribution with mean $kb$, where $b$ is a known constant in the unit interval and $k$ is a discrete, non-negative parameter. We show that if an uninformative uniform prior for $k$ is assumed, then the posterior distribution of $k$ can be approximated using the gamma distribution when $b$ is small.
We explore the use of higher-order tail area approximations for Bayesian simulation. These approximations give rise to an alternative simulation scheme to MCMC for Bayesian computation of marginal posterior distributions for a scalar parameter of interest, in the presence of nuisance parameters. Its advantage over MCMC methods is that samples are drawn independently with lower computational time and the implementation requires only standard maximum likelihood routines. The method is illustrated by a genetic linkage model, a normal regression with censored data and a logistic regression model.
Expected-posterior priors (EPP) have been proved to be extremely useful for testing hypothesis on the regression coefficients of normal linear models. One of the advantages of using EPPs is that impropriety of baseline priors causes no indeterminacy. However, in regression problems, they based on one or more \textit{training samples}, that could influence the resulting posterior distribution. The power-expected-posterior priors are minimally-informative priors that diminishing the effect of training samples on the EPP approach, by combining ideas from the power-prior and unit-information-prior…
Bayesian inference for models with intractable likelihood functions represents a challenging suite of problems in modern statistics. In this work we analyse the Conway-Maxwell-Poisson (COM-Poisson) distribution, a two parameter generalisation of the Poisson distribution. COM-Poisson regression modelling allows the flexibility to model dispersed count data as part of a generalised linear model (GLM) with a COM-Poisson response, where exogenous covariates control the mean and dispersion level of the response. The major difficulty with COM-Poisson regression is that the likelihood function contai…
Erasmus University Rotterdam（鹿特丹伊拉斯姆斯大学）；Stanford University（斯坦福大学）
We propose a general algorithm for approximating nonstandard Bayesian posterior distributions. The algorithm minimizes the Kullback-Leibler divergence of an approximating distribution to the intractable posterior distribution. Our method can be used to approximate any posterior distribution, provided that it is given in closed form up to the proportionality constant. The approximation can be any distribution in the exponential family or any mixture of such distributions, which means that it can be made arbitrarily precise. Several examples illustrate the speed and accuracy of our approximation…
In this article, we investigate posterior convergence of nonparametric binary and Poisson regression under possible model misspecification, assuming general stochastic process prior with appropriate properties. Our model setup and objective for binary regression is similar to that of Ghosal and Roy (2006) where the authors have used the approach of entropy bound and exponentially consistent tests with the sieve method to achieve consistency with respect to their Gaussian process prior. In contrast, for both binary and Poisson regression, using general stochastic process prior, our approach inv…
We consider Bayesian inference in inverse regression problems where the objective is to infer about unobserved covariates from observed responses and covariates. We establish posterior consistency of such unobserved covariates in Bayesian inverse regression problemsunder appropriate priors in a leave-one-out cross-validation setup. We relate this to posterior consistency of inverse reference distributions (Bhattacharya (2013)) for assessing model adequacy. We illustrate our theory and methods with various examples of Bayesian inverse regression, along with adequate simulation experiments.
Quantile regression is a powerful data analysis tool that accommodates heterogeneous covariate-response relationships. We find that by coupling the asymmetric Laplace working likelihood with appropriate shrinkage priors, we can deliver posterior inference that automatically adapts to possible sparsity in quantile regression analysis. After a suitable adjustment on the posterior variance, the posterior inference provides asymptotically valid inference under heterogeneity. Furthermore, the proposed approach leads to oracle asymptotic efficiency for the active (nonzero) quantile regression coeffi…
The iterated posterior linearization filter (IPLF) is an algorithm for Bayesian state estimation that performs the measurement update using iterative statistical regression. The main result behind IPLF is that the posterior approximation is more accurate when the statistical regression of measurement function is done in the posterior instead of the prior as is done in non-iterative Kalman filter extensions. In IPLF, each iteration in principle gives a better posterior estimate to obtain a better statistical regression and more accurate posterior estimate in the next iteration. However, IPLF ma…
The paper addresses joint sparsity selection in the regression coefficient matrix and the error precision (inverse covariance) matrix for high-dimensional multivariate regression models in the Bayesian paradigm. The selected sparsity patterns are crucial to help understand the network of relationships between the predictor and response variables, as well as the conditional relationships among the latter. While Bayesian methods have the advantage of providing natural uncertainty quantification through posterior inclusion probabilities and credible intervals, current Bayesian approaches either r…
Regression models with fat-tailed error terms are an increasingly popular choice to obtain more robust inference to the presence of outlying observations. This article focuses on Bayesian inference for the Student-$t$ linear regression model under objective priors that are based on the Jeffreys rule. Posterior propriety results presented in Fonseca et al. (2008) are revisited and corrected. In particular, it is shown that the standard Jeffreys-rule prior precludes the existence of a proper posterior distribution.
Although linear regression models are fundamental tools in statistical science, the estimation results can be sensitive to outliers. While several robust methods have been proposed in frequentist frameworks, statistical inference is not necessarily straightforward. We here propose a Bayesian approach to robust inference on linear regression models using synthetic posterior distributions based on $γ$-divergence, which enables us to naturally assess the uncertainty of the estimation through the posterior distribution. We also consider the use of shrinkage priors for the regression coefficients t…
We study posterior rates of contraction in Gaussian process regression with unbounded covariate domain. Our argument relies on developing a Gaussian approximation to the posterior of the leading coefficients of a Karhunen--Loéve expansion of the Gaussian process. The salient feature of our result is deriving such an approximation in the $L^2$ Wasserstein distance and relating the speed of the approximation to the posterior contraction rate using a coupling argument. Specific illustrations are provided for the Gaussian or squared-exponential covariance kernel.
Being in the era of big data, modeling and prediction of count data have become significantly important in many fields including health, finance, social, etc. Although linear Poisson regression has been widely used to model count and rate data, it might not be always suitable as it cannot capture some inherent variability within complex data. In this study, we introduce a probabilistically driven nonlinear Poisson regression model with Bayesian artificial neural networks (ANN) to model count or rate data. This new nonlinear Poisson regression model developed with Bayesian ANN provides higher p…
Regression models for dichotomous data are ubiquitous in statistics. Besides being useful for inference on binary responses, these methods serve also as building blocks in more complex formulations, such as density regression, nonparametric classification and graphical models. Within the Bayesian framework, inference proceeds by updating the priors for the coefficients, typically set to be Gaussians, with the likelihood induced by probit or logit regressions for the responses. In this updating, the apparent absence of a tractable posterior has motivated a variety of computational methods, incl…
In this work we consider Bayesian inference problems with intractable likelihood functions. We present a method to compute an approximate of the posterior with a limited number of model simulations. The method features an inverse Gaussian Process regression (IGPR), i.e., one from the output of a simulation model to the input of it. Within the method, we provide an adaptive algorithm with a tempering procedure to construct the approximations of the marginal posterior distributions. With examples we demonstrate that IGPR has a competitive performance compared to some commonly used algorithms, es…
We consider the nonparametric regression problem with multiple predictors and an additive error, where the regression function is assumed to be coordinatewise nondecreasing. We propose a Bayesian approach to make an inference on the multivariate monotone regression function, obtain the posterior contraction rate, and construct a universally consistent Bayesian testing procedure for multivariate monotonicity. To facilitate posterior analysis, we set aside the shape restrictions temporarily, and endow a prior on blockwise constant regression functions with heights independently normally distribu…
We prove the weak consistency of the posterior distribution and that of the Bayes estimator for a two-phase piecewise linear regression mdoel where the break-point is unknown. The non-differentiability of the likelihood of the model with regard to the break- point parameter induces technical difficulties that we overcome by creating a regularised version of the problem at hand. We first recover the strong consistency of the quantities of interest for the regularised version, using results about the MLE, and we then prove that the regularised version and the original version of the problem shar…
The asymmetric Laplace density (ALD) is used as a working likelihood for Bayesian quantile regression. Sriram et al.(2013) derived posterior consistency for Bayesian linear quantile regression based on the misspecified ALD. While their paper also argued for $\sqrt{n}-$consistency, Sriram and Ramamoorthi (2017) highlighted that the argument was only valid for a rate less than $\sqrt{n}$. So, the question of $\sqrt{n}-$rate has remained unaddressed, but is necessary to carry out meaningful Bayesian inference based on the ALD. In this paper, we derive results to obtain posterior consistency rates…
In this article, we investigate posterior convergence in nonparametric regression models where the unknown regression function is modeled by some appropriate stochastic process. In this regard, we consider two setups. The first setup is based on Gaussian processes, where the covariates are either random or non-random and the noise may be either normally or double-exponentially distributed. In the second setup, we assume that the underlying regression function is modeled by some reasonably smooth, but unspecified stochastic process satisfying reasonable conditions. The distribution of the noise…
This work is motivated by the analysis of ecological interaction networks. Poisson stochastic blockmodels are widely used in this field to decipher the structure that underlies a weighted network, while accounting for covariate effects. Efficient algorithms based on variational approximations exist for frequentist inference, but without statistical guaranties as for the resulting estimates. In absence of variational Bayes estimates, we show that a good proxy of the posterior distribution can be straightforwardly derived from the frequentist variational estimation procedure, using a Laplace app…
In the Bayesian approach, the a priori knowledge about the input of a mathematical model is described via a probability measure. The joint distribution of the unknown input and the data is then conditioned, using Bayes' formula, giving rise to the posterior distribution on the unknown input. In this setting we prove posterior consistency for nonlinear inverse problems: a sequence of data is considered, with diminishing fluctuations around a single truth and it is then of interest to show that the resulting sequence of posterior measures arising from this sequence of data concentrates around th…
In logistic regression, separation occurs when a linear combination of the predictors can perfectly classify part or all of the observations in the sample, and as a result, finite maximum likelihood estimates of the regression coefficients do not exist. Gelman et al. (2008) recommended independent Cauchy distributions as default priors for the regression coefficients in logistic regression, even in the case of separation, and reported posterior modes in their analyses. As the mean does not exist for the Cauchy prior, a natural question is whether the posterior means of the regression coefficie…
Gaussian process (GP) regression is a powerful interpolation technique due to its flexibility in capturing non-linearity. In this paper, we provide a general framework for understanding the frequentist coverage of point-wise and simultaneous Bayesian credible sets in GP regression. As an intermediate result, we develop a Bernstein von-Mises type result under supremum norm in random design GP regression. Identifying both the mean and covariance function of the posterior distribution of the Gaussian process as regularized $M$-estimators, we show that the sampling distribution of the posterior me…
We study the behavior of the posterior distribution in high-dimensional Bayesian Gaussian linear regression models having $p\gg n$, with $p$ the number of predictors and $n$ the sample size. Our focus is on obtaining quantitative finite sample bounds ensuring sufficient posterior probability assigned in neighborhoods of the true regression coefficient vector, $β^0$, with high probability. We assume that $β^0$ is approximately $S$-sparse and obtain universal bounds, which provide insight into the role of the prior in controlling concentration of the posterior. Based on these finite sample bound…
Supremum norm loss is intuitively more meaningful to quantify function estimation error in statistics. In the context of multivariate nonparametric regression with unknown error, we propose a Bayesian procedure based on spike-and-slab prior and wavelet projections to estimate the regression function and all its mixed partial derivatives. We show that their posterior distributions contract to the truth optimally and adaptively under supremum-norm loss. The master theorem through tests with exponential errors used in Bayesian nonparametrics was not adequate to deal with this problem, and we deve…
In the setting of nonparametric multivariate regression with unknown error variance, we study asymptotic properties of a Bayesian method for estimating a regression function f and its mixed partial derivatives. We use a random series of tensor product of B-splines with normal basis coefficients as a prior for f, and the error variance is either estimated using the empirical Bayes approach or is endowed with a suitable prior in a hierarchical Bayes approach. We establish pointwise, L2 and supremum norm posterior contraction rates for f and its mixed partial derivatives, and show that they coinc…
During the past decade, shrinkage priors have received much attention in Bayesian analysis of high-dimensional data. This paper establishes the posterior consistency for high-dimensional linear regression with a class of shrinkage priors, which has a heavy and flat tail and allocates a sufficiently large probability mass in a very small neighborhood of zero. While enjoying its efficiency in posterior simulations, the shrinkage prior can lead to a nearly optimal posterior contraction rate and variable selection consistency as the spike-and-slab prior. Our numerical results show that under the p…
We provide a complete framework for performing infinite-dimensional Bayesian inference and uncertainty quantification for image reconstruction with Poisson data. In particular, we address the following issues to make the Bayesian framework applicable in practice. We first introduce a positivity-preserving reparametrization, and we prove that under the reparametrization and a hybrid prior, the posterior distribution is well-posed in the infinite dimensional setting. Second we provide a dimension-independent MCMC algorithm, based on the preconditioned Crank-Nicolson Langevin method, in which we …
This article proposes a Bayesian approach to regression with a scalar response against vector and tensor covariates. Tensor covariates are commonly vectorized prior to analysis, failing to exploit the structure of the tensor, and resulting in poor estimation and predictive performance. We develop a novel class of multiway shrinkage priors for the coefficients in tensor regression models. Properties are described, including posterior consistency under mild conditions, and an efficient Markov chain Monte Carlo algorithm is developed for posterior computation. Simulation studies illustrate substa…
Bayesian posterior distributions arising in modern applications, including inverse problems in partial differential equation models in tomography and subsurface flow, are often computationally intractable due to the large computational cost of evaluating the data likelihood. To alleviate this problem, we consider using Gaussian process regression to build a surrogate model for the likelihood, resulting in an approximate posterior distribution that is amenable to computations in practice. This work serves as an introduction to Gaussian process regression, in particular in the context of buildin…
Since their inception in the 1980's, regression trees have been one of the more widely used non-parametric prediction methods. Tree-structured methods yield a histogram reconstruction of the regression surface, where the bins correspond to terminal nodes of recursive partitioning. Trees are powerful, yet susceptible to over-fitting. Strategies against overfitting have traditionally relied on pruning greedily grown trees. The Bayesian framework offers an alternative remedy against overfitting through priors. Roughly speaking, a good prior charges smaller trees where overfitting does not occur. …
University of Utah（犹他大学）；School of Computing（计算学院）；Scientific Computing and Imaging Institute（科学计算与成像研究所）
Gaussian process regression networks (GPRN) are powerful Bayesian models for multi-output regression, but their inference is intractable. To address this issue, existing methods use a fully factorized structure (or a mixture of such structures) over all the outputs and latent functions for posterior approximation, which, however, can miss the strong posterior dependencies among the latent variables and hurt the inference quality. In addition, the updates of the variational parameters are inefficient and can be prohibitively expensive for a large number of outputs. To overcome these limitations…
In application areas where data generation is expensive, Gaussian processes are a preferred supervised learning model due to their high data-efficiency. Particularly in model-based control, Gaussian processes allow the derivation of performance guarantees using probabilistic model error bounds. To make these approaches applicable in practice, two open challenges must be solved i) Existing error bounds rely on prior knowledge, which might not be available for many real-world tasks. (ii) The relationship between training data and the posterior variance, which mainly drives the error bound, is no…
We study the posterior contraction rates of a Bayesian method with Gaussian process priors in nonparametric regression and its plug-in property for differential operators. For a general class of kernels, we establish convergence rates of the posterior measure of the regression function and its derivatives, which are both minimax optimal up to a logarithmic factor for functions in certain classes. Our calculation shows that the rate-optimal estimation of the regression function and its derivatives share the same choice of hyperparameter, indicating that the Bayes procedure remarkably adapts to …
Nested error regression models are useful tools for analysis of grouped data, especially in the case of small area estimation. This paper suggests a nested error regression model using uncertain random effects in which the random effect in each area is expressed as a mixture of a normal distribution and a positive mass at $0$. For estimation of the model parameters and prediction of the random effects, an objective Bayesian inference is proposed by setting non-informative prior distributions on the model parameters. Under mild sufficient conditions, it is shown that the posterior distribution …
Shape restrictions such as monotonicity on functions often arise naturally in statistical modeling. We consider a Bayesian approach to the problem of estimation of a monotone regression function and testing for monotonicity. We construct a prior distribution using piecewise constant functions. For estimation, a prior imposing monotonicity of the heights of these steps is sensible, but the resulting posterior is harder to analyze theoretically. We consider a ``projection-posterior'' approach, where a conjugate normal prior is used, but the monotonicity constraint is imposed on posterior samples…
Caltech（加州理工学院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Graph-based semi-supervised regression (SSR) is the problem of estimating the value of a function on a weighted graph from its values (labels) on a small subset of the vertices. This paper is concerned with the consistency of SSR in the context of classification, in the setting where the labels have small noise and the underlying graph weighting is consistent with well-clustered nodes. We present a Bayesian formulation of SSR in which the weighted graph defines a Gaussian prior, using a graph Laplacian, and the labeled data defines a likelihood. We analyze the rate of contraction of the poster…
We study posterior contraction rates for a class of deep Gaussian process priors applied to the nonparametric regression problem under a general composition assumption on the regression function. It is shown that the contraction rates can achieve the minimax convergence rate (up to $\log n$ factors), while being adaptive to the underlying structure and smoothness of the target function. The proposed framework extends the Bayesian nonparametric theory for Gaussian process priors.
This paper extends the idea of decoupling shrinkage and sparsity for continuous priors to Bayesian Quantile Regression (BQR). The procedure follows two steps: In the first step, we shrink the quantile regression posterior through state of the art continuous priors and in the second step, we sparsify the posterior through an efficient variant of the adaptive lasso, the signal adaptive variable selection (SAVS) algorithm. We propose a new variant of the SAVS which automates the choice of penalisation through quantile specific loss-functions that are valid in high dimensions. We show in large sca…
The power-expected-posterior (PEP) prior is an objective prior for Gaussian linear models, which leads to consistent model selection inference, under the M-closed scenario, and tends to favor parsimonious models. Recently, two new forms of the PEP prior were proposed which generalize its applicability to a wider range of models. The properties of these two PEP variants within the context of the normal linear model are examined thoroughly, focusing on the prior dispersion and on the consistency of the induced model selection procedure. Results show that both PEP variants have larger variances t…
