In the Bayesian approach, the a priori knowledge about the input of a mathematical model is described via a probability measure. The joint distribution of the unknown input and the data is then conditioned, using Bayes' formula, giving rise to the posterior distribution on the unknown input. In this setting we prove posterior consistency for nonlinear inverse problems: a sequence of data is considered, with diminishing fluctuations around a single truth and it is then of interest to show that the resulting sequence of posterior measures arising from this sequence of data concentrates around the truth used to generate the data. Posterior consistency justifies the use of the Bayesian approach very much in the same way as error bounds and convergence results for regularisation techniques do. As a guiding example, we consider the inverse problem of reconstructing the diffusion coefficient from noisy observations of the solution to an elliptic PDE in divergence form. This problem is approached by splitting the forward operator into the underlying continuum model and a simpler observation operator based on the output of the model.
In general, these splittings allow us to conclude posterior consistency provided a deterministic stability result for the underlying inverse problem and a posterior consistency result for the Bayesian regression problem with the push-forward prior.
Moreover, we prove posterior consistency for the Bayesian regression problem based on the regularity, the tail behaviour and the small ball probabilities of the prior.
We consider Bayesian inference in inverse regression problems where the objective is to infer about unobserved covariates from observed responses and covariates. We establish posterior consistency of such unobserved covariates in Bayesian inverse regression problemsunder appropriate priors in a leave-one-out cross-validation setup. We relate this to posterior consistency of inverse reference distributions (Bhattacharya (2013)) for assessing model adequacy. We illustrate our theory and methods with various examples of Bayesian inverse regression, along with adequate simulation experiments.
In this paper we study posterior consistency for different topologies on the parameters for hidden Markov models with finite state space. We first obtain weak and strong posterior consistency for the marginal density function of finitely many consecutive observations. We deduce posterior consistency for the different components of the parameter. We also obtain posterior consistency for marginal smoothing distributions in the discrete case. We finally apply our results to independent emission probabilities, translated emission probabilities and discrete HMMs, under various types of priors.
We study posterior rates of contraction in Gaussian process regression with unbounded covariate domain. Our argument relies on developing a Gaussian approximation to the posterior of the leading coefficients of a Karhunen--Loéve expansion of the Gaussian process. The salient feature of our result is deriving such an approximation in the $L^2$ Wasserstein distance and relating the speed of the approximation to the posterior contraction rate using a coupling argument. Specific illustrations are provided for the Gaussian or squared-exponential covariance kernel.
In this article, we investigate posterior convergence of nonparametric binary and Poisson regression under possible model misspecification, assuming general stochastic process prior with appropriate properties. Our model setup and objective for binary regression is similar to that of Ghosal and Roy (2006) where the authors have used the approach of entropy bound and exponentially consistent tests with the sieve method to achieve consistency with respect to their Gaussian process prior. In contrast, for both binary and Poisson regression, using general stochastic process prior, our approach inv…
We consider the nonparametric regression problem with multiple predictors and an additive error, where the regression function is assumed to be coordinatewise nondecreasing. We propose a Bayesian approach to make an inference on the multivariate monotone regression function, obtain the posterior contraction rate, and construct a universally consistent Bayesian testing procedure for multivariate monotonicity. To facilitate posterior analysis, we set aside the shape restrictions temporarily, and endow a prior on blockwise constant regression functions with heights independently normally distribu…
We prove the weak consistency of the posterior distribution and that of the Bayes estimator for a two-phase piecewise linear regression mdoel where the break-point is unknown. The non-differentiability of the likelihood of the model with regard to the break- point parameter induces technical difficulties that we overcome by creating a regularised version of the problem at hand. We first recover the strong consistency of the quantities of interest for the regularised version, using results about the MLE, and we then prove that the regularised version and the original version of the problem shar…
Asymptotic theory of tail index estimation has been studied extensively in the frequentist literature on extreme values, but rarely in the Bayesian context. We investigate whether popular Bayesian kernel mixture models are able to support heavy tailed distributions and consistently estimate the tail index. We show that posterior inconsistency in tail index is surprisingly common for both parametric and nonparametric mixture models. We then present a set of sufficient conditions under which posterior consistency in tail index can be achieved, and verify these conditions for Pareto mixture model…
The asymmetric Laplace density (ALD) is used as a working likelihood for Bayesian quantile regression. Sriram et al.(2013) derived posterior consistency for Bayesian linear quantile regression based on the misspecified ALD. While their paper also argued for $\sqrt{n}-$consistency, Sriram and Ramamoorthi (2017) highlighted that the argument was only valid for a rate less than $\sqrt{n}$. So, the question of $\sqrt{n}-$rate has remained unaddressed, but is necessary to carry out meaningful Bayesian inference based on the ALD. In this paper, we derive results to obtain posterior consistency rates…
In this work we establish the posterior consistency for a parametrized family of partially observed, fully dominated Markov models. As a main assumption, we suppose that the prior distribution assigns positive probability to all neighborhoods of the true parameter, for a distance induced by the expected Kullback-Leibler divergence between the family members' Markov transition densities. This assumption is easily checked in general. In addition, under some additional, mild assumptions we show that the posterior consistency is implied by the consistency of the maximum likelihood estimator. The l…
Frequentist conditions for asymptotic suitability of Bayesian procedures focus on lower bounds for prior mass in Kullback-Leibler neighbourhoods of the data distribution. The goal of this paper is to investigate the flexibility in criteria for posterior consistency with i.i.d. data. We formulate a versatile posterior consistency theorem that applies both to well- and mis-specified models and which we use to re-derive Schwartz's theorem, consider Kullback-Leibler consistency and formulate consistency theorems in which priors charge metric balls. It is generalized to sieved models with Barron's …
In this paper, we consider the well known problem of estimating a density function under qualitative assumptions. More precisely, we estimate monotone non increasing densities in a Bayesian setting and derive concentration rate for the posterior distribution for a Dirichlet process and finite mixture prior. We prove that the posterior distribution based on both priors concentrates at the rate $(n/\log(n))^{-1/3}$, which is the minimax rate of estimation up to a \log(n)$ factor. We also study the behaviour of the posterior for the point-wise loss at any fixed point of the support the density an…
Quantile regression is a powerful data analysis tool that accommodates heterogeneous covariate-response relationships. We find that by coupling the asymmetric Laplace working likelihood with appropriate shrinkage priors, we can deliver posterior inference that automatically adapts to possible sparsity in quantile regression analysis. After a suitable adjustment on the posterior variance, the posterior inference provides asymptotically valid inference under heterogeneity. Furthermore, the proposed approach leads to oracle asymptotic efficiency for the active (nonzero) quantile regression coeffi…
We examine necessary and sufficient conditions for posterior consistency under $g$-priors, including extensions to hierarchical and empirical Bayesian models. The key features of this article are that we allow the number of regressors to grow at the same rate as the sample size and define posterior consistency under the sup vector norm instead of the more conventional Euclidean norm. We consider in particular the empirical Bayesian model of George and Foster (2000), the hyper-$g$-prior of Liang et al. (2008), and the prior considered by Zellner and Siow (1980).
Estimating the parameters from $k$ independent Bin$(n,p)$ random variables, when both parameters $n$ and $p$ are unknown, is relevant to a variety of applications. It is particularly difficult if $n$ is large and $p$ is small. Over the past decades, several articles have proposed Bayesian approaches to estimate $n$ in this setting, but asymptotic results could only be established recently in \cite{Schneider}. There, posterior contraction for $n$ is proven in the problematic parameter regime where $n\rightarrow\infty$ and $p\rightarrow0$ at certain rates. In this article, we study numerically h…
In this paper, we prove almost surely consistency of a Survival Analysis model, which puts a Gaussian process, mapped to the unit interval, as a prior on the so-called hazard function. We assume our data is given by survival lifetimes $T$ belonging to $\mathbb{R}^{+}$, and covariates on $[0,1]^d$, where $d$ is an arbitrary dimension. We define an appropriate metric for survival functions and prove posterior consistency with respect to this metric. Our proof is based on an extension of the theorem of Schwartz (1965), which gives general conditions for proving almost surely consistency in the se…
We study the behavior of the posterior distribution in high-dimensional Bayesian Gaussian linear regression models having $p\gg n$, with $p$ the number of predictors and $n$ the sample size. Our focus is on obtaining quantitative finite sample bounds ensuring sufficient posterior probability assigned in neighborhoods of the true regression coefficient vector, $β^0$, with high probability. We assume that $β^0$ is approximately $S$-sparse and obtain universal bounds, which provide insight into the role of the prior in controlling concentration of the posterior. Based on these finite sample bound…
This paper addresses the estimation of the nonparametric conditional moment restricted model that involves an infinite-dimensional parameter $g_0$. We estimate it in a quasi-Bayesian way, based on the limited information likelihood, and investigate the impact of three types of priors on the posterior consistency: (i) truncated prior (priors supported on a bounded set), (ii) thin-tail prior (a prior that has very thin tail outside a growing bounded set) and (iii) normal prior with nonshrinking variance. In addition, $g_0$ is allowed to be only partially identified in the frequentist sense, and …
Supremum norm loss is intuitively more meaningful to quantify function estimation error in statistics. In the context of multivariate nonparametric regression with unknown error, we propose a Bayesian procedure based on spike-and-slab prior and wavelet projections to estimate the regression function and all its mixed partial derivatives. We show that their posterior distributions contract to the truth optimally and adaptively under supremum-norm loss. The master theorem through tests with exponential errors used in Bayesian nonparametrics was not adequate to deal with this problem, and we deve…
In the setting of nonparametric multivariate regression with unknown error variance, we study asymptotic properties of a Bayesian method for estimating a regression function f and its mixed partial derivatives. We use a random series of tensor product of B-splines with normal basis coefficients as a prior for f, and the error variance is either estimated using the empirical Bayes approach or is endowed with a suitable prior in a hierarchical Bayes approach. We establish pointwise, L2 and supremum norm posterior contraction rates for f and its mixed partial derivatives, and show that they coinc…
During the past decade, shrinkage priors have received much attention in Bayesian analysis of high-dimensional data. This paper establishes the posterior consistency for high-dimensional linear regression with a class of shrinkage priors, which has a heavy and flat tail and allocates a sufficiently large probability mass in a very small neighborhood of zero. While enjoying its efficiency in posterior simulations, the shrinkage prior can lead to a nearly optimal posterior contraction rate and variable selection consistency as the spike-and-slab prior. Our numerical results show that under the p…
We derive conditions for posterior consistency when the responses are independent but not identically distributed ($i.n.i.d$) and the model is "misspecified" to be a family of densities parametrized by a possibly infinite dimensional parameter. Our approach has connections to key ideas developed for $i.i.d$ models in Kleijn and van der Vaart(2006) and it's subsequent simplification in Ramamoorthi, et al.(2014) (unpublished manuscript). While key results in these two papers rely heavily on the convexity of the specified family of densities, parametric families are seldom convex. In this note, w…
Regression models with fat-tailed error terms are an increasingly popular choice to obtain more robust inference to the presence of outlying observations. This article focuses on Bayesian inference for the Student-$t$ linear regression model under objective priors that are based on the Jeffreys rule. Posterior propriety results presented in Fonseca et al. (2008) are revisited and corrected. In particular, it is shown that the standard Jeffreys-rule prior precludes the existence of a proper posterior distribution.
Since their inception in the 1980's, regression trees have been one of the more widely used non-parametric prediction methods. Tree-structured methods yield a histogram reconstruction of the regression surface, where the bins correspond to terminal nodes of recursive partitioning. Trees are powerful, yet susceptible to over-fitting. Strategies against overfitting have traditionally relied on pruning greedily grown trees. The Bayesian framework offers an alternative remedy against overfitting through priors. Roughly speaking, a good prior charges smaller trees where overfitting does not occur. …
Shape restrictions such as monotonicity on functions often arise naturally in statistical modeling. We consider a Bayesian approach to the problem of estimation of a monotone regression function and testing for monotonicity. We construct a prior distribution using piecewise constant functions. For estimation, a prior imposing monotonicity of the heights of these steps is sensible, but the resulting posterior is harder to analyze theoretically. We consider a ``projection-posterior'' approach, where a conjugate normal prior is used, but the monotonicity constraint is imposed on posterior samples…
We investigate the asymptotic behavior of posterior distributions of regression coefficients in high-dimensional linear models as the number of dimensions grows with the number of observations. We show that the posterior distribution concentrates in neighborhoods of the true parameter under simple sufficient conditions. These conditions hold under popular shrinkage priors given some sparsity assumptions.
We study posterior contraction rates for a class of deep Gaussian process priors applied to the nonparametric regression problem under a general composition assumption on the regression function. It is shown that the contraction rates can achieve the minimax convergence rate (up to $\log n$ factors), while being adaptive to the underlying structure and smoothness of the target function. The proposed framework extends the Bayesian nonparametric theory for Gaussian process priors.
In this article, we investigate posterior convergence in nonparametric regression models where the unknown regression function is modeled by some appropriate stochastic process. In this regard, we consider two setups. The first setup is based on Gaussian processes, where the covariates are either random or non-random and the noise may be either normally or double-exponentially distributed. In the second setup, we assume that the underlying regression function is modeled by some reasonably smooth, but unspecified stochastic process satisfying reasonable conditions. The distribution of the noise…
We study frequentist properties of a Bayesian high-dimensional multivariate linear regression model with correlated responses. The predictors are separated into many groups and the group structure is pre-determined. Two features of the model are unique: (i) group sparsity is imposed on the predictors. (ii) the covariance matrix is unknown and its dimensions can also be high. We choose a product of independent spike-and-slab priors on the regression coefficients and a new prior on the covariance matrix based on its eigendecomposition. Each spike-and-slab prior is a mixture of a point mass at ze…
Strong consistency of the Bayes estimator of a regression curve for the $L^1$-squared loss function is proved. It is also shown the convergence to 0 of the Bayes risk of this estimator both for the $L^1$ and $L^1$-squared loss functions. The Bayes estimator of a regression curve is the regression curve with respect to the posterior predictive distribution.
In this paper, we consider the so-called Shape Invariant Model which stands for the estimation of a function $f^0$ submitted to a random translation of law $g^0$ in a white noise model. We are interested in such a model when the law of the deformations is unknown. We aim to recover the law of the process $\PP_{f^0,g^0}$ as well as $f^0$ and $g^0$. In this perspective, we adopt a Bayesian point of view and find prior on $f$ and $g$ such that the posterior distribution concentrates around $\PP_{f^0,g^0}$ at a polynomial rate when $n$ goes to $+\infty$. We obtain a logarithmic posterior contracti…
Good large sample performance is typically a minimum requirement of any model selection criterion. This article focuses on the consistency property of the Bayes factor, a commonly used model comparison tool, which has experienced a recent surge of attention in the literature. We thoroughly review existing results. As there exists such a wide variety of settings to be considered, e.g. parametric vs. nonparametric, nested vs. non-nested, etc., we adopt the view that a unified framework has didactic value. Using the basic marginal likelihood identity of Chib (1995), we study Bayes factor asymptot…
We study full Bayesian procedures for high-dimensional linear regression. We adopt data-dependent empirical priors introduced in [1]. In their paper, these priors have nice posterior contraction properties and are easy to compute. Our paper extend their theoretical results to the case of unknown error variance . Under proper sparsity assumption, we achieve model selection consistency, posterior contraction rates as well as Bernstein von-Mises theorem by analyzing multivariate t-distribution.
The choice of tuning parameters in Bayesian variable selection is a critical problem in modern statistics. In particular, for Bayesian linear regression with non-local priors, the scale parameter in the non-local prior density is an important tuning parameter which reflects the dispersion of the non-local prior density around zero, and implicitly determines the size of the regression coefficients that will be shrunk to zero. Current approaches treat the scale parameter as given, and suggest choices based on prior coverage/asymptotic considerations. In this paper, we consider the fully Bayesian…
The choice of the prior distribution is a key aspect of Bayesian analysis. For the spatial regression setting a subjective prior choice for the parameters may not be trivial, from this perspective, using the objective Bayesian analysis framework a reference is introduced for the spatial Student-t regression model with unknown degrees of freedom. The spatial Student-t regression model poses two main challenges when eliciting priors: one for the spatial dependence parameter and the other one for the degrees of freedom. It is well-known that the propriety of the posterior distribution over object…
We consider heteroscedastic nonparametric regression models, when both the mean function and variance function are unknown and to be estimated with nonparametric approaches. We derive convergence rates of posterior distributions for this model with different priors, including splines and Gaussian process priors. The results are based on the general ones on the rates of convergence of posterior distributions for independent, non-identically distributed observations, and are established for both of the cases with random covariates, and deterministic covariates. We also illustrate that the result…
This paper studies posterior contraction rates in multi-category logit models with priors incorporating group sparse structures. We consider a general class of logit models that includes the well-known multinomial logit models as a special case. Group sparsity is useful when predictor variables are naturally clustered and particularly useful for variable selection in the multinomial logit models. We provide a unified platform for posterior contraction rates of group-sparse logit models that include binary logistic regression under individual sparsity. No size restriction is directly imposed on…
We consider the nonparametric multivariate isotonic regression problem, where the regression function is assumed to be nondecreasing with respect to each predictor. Our goal is to construct a Bayesian credible interval for the function value at a given interior point with assured limiting frequentist coverage. We put a prior on unrestricted step-functions, but make inference using the induced posterior measure by an "immersion map" from the space of unrestricted functions to that of multivariate monotone functions. This allows maintaining the natural conjugacy for posterior sampling. A natural…
The logistic regression model is the most popular model for analyzing binary data. In the absence of any prior information, an improper flat prior is often used for the regression coefficients in Bayesian logistic regression models. The resulting intractable posterior density can be explored by running Polson et al.'s (2013) data augmentation (DA) algorithm. In this paper, we establish that the Markov chain underlying Polson et al.'s (2013) DA algorithm is geometrically ergodic. Proving this theoretical result is practically important as it ensures the existence of central limit theorems (CLTs…
For ill-posed inverse problems, a regularised solution can be interpreted as a mode of the posterior distribution in a Bayesian framework. This framework enriches the set the solutions, as other posterior estimates can be used as a solution to the inverse problem, such as the posterior mean that can be easier to compute in practice. In this paper we prove consistency of Bayesian solutions of an ill-posed linear inverse problem in the Ky Fan metric for a general class of likelihoods and prior distributions in a finite dimensional setting. This result can be applied to study infinite dimensional…
We consider a sparse linear regression model, when the number of available predictors, $p$, is much larger than the sample size, $n$, and the number of non-zero coefficients, $p_0$, is small. To choose the regression model in this situation, we cannot use classical model selection criteria. In recent years, special methods have been proposed to deal with this type of problem, for example modified versions of Bayesian Information Criterion, like mBIC or mBIC2. It was shown that these criteria are consistent under the assumption that both $n$ and $p$ as well as $p_0$ tend to infinity and the err…
