In this paper we provide general conditions to check on the model and the prior to derive posterior concentration rates for data-dependent priors (or empirical Bayes approaches). We aim at providing conditions that are close to the conditions provided in the seminal paper by Ghosal and van der Vaart (2007a). We then apply the general theorem to two different settings: the estimation of a density using Dirichlet process mixtures of Gaussian random variables with base measure depending on some empirical quantities and the estimation of the intensity of a counting process under the Aalen model. A simulation study for inhomogeneous Poisson processes also illustrates our results. In the former case we also derive some results on the estimation of the mixing density and on the deconvolution problem. In the latter, we provide a general theorem on posterior concentration rates for counting processes with Aalen multiplicative intensity with priors not depending on the data.
The influence of Dirichlet process mixture is ubiquitous in the Bayesian nonparametrics literature. But sampling from its posterior distribution remains a challenge, despite the advent of various Markov chain Monte Carlo methods. The primary challenge is the infinite-dimensional setup, and even if the infinite-dimensional random measure is integrated out, high-dimensionality and discreteness still remain difficult issues to deal with. In this article, exploiting the key ideas proposed in Bhattacharya (2021b), we propose a novel methodology for drawing iid realizations from posteriors of Dirich…
The aim of this paper is to find distributional results for the posterior parameters which arise in the Sethuraman (1994) representation of the Dirichlet process. These results can then be used to derive simply the posterior of the Dirichlet process.
We study nonparametric Bayesian inference with location mixtures of the Laplace density and a Dirichlet process prior on the mixing distribution. We derive a contraction rate of the corresponding posterior distribution, both for the mixing distribution relative to the Wasserstein metric and for the mixed density relative to the Hellinger and $L_q$ metrics.
We study the rates of convergence of the posterior distribution for Bayesian density estimation with Dirichlet mixtures of normal distributions as the prior. The true density is assumed to be twice continuously differentiable. The bandwidth is given a sequence of priors which is obtained by scaling a single prior by an appropriate order. In order to handle this problem, we derive a new general rate theorem by considering a countable covering of the parameter space whose prior probabilities satisfy a summability condition together with certain individual bounds on the Hellinger metric entropy. …
We investigate the posterior rate of convergence for wavelet shrinkage using a Bayesian approach in general Besov spaces. Instead of studying the Bayesian estimator related to a particular loss function, we focus on the posterior distribution itself from a nonparametric Bayesian asymptotics point of view and study its rate of convergence. We obtain the same rate as in \citet{abramovich04} where the authors studied the convergence of several Bayesian estimators.
We consider the asymptotic behaviour of the marginal maximum likelihood empirical Bayes posterior distribution in general setting. First we characterize the set where the maximum marginal likelihood estimator is located with high probability. Then we provide oracle type of upper and lower bounds for the contraction rates of the empirical Bayes posterior. We also show that the hierarchical Bayes posterior achieves the same contraction rate as the maximum marginal likelihood empirical Bayes posterior. We demonstrate the applicability of our general results for various models and prior distributi…
In the sparse normal means model, convergence of the Bayesian posterior distribution associated to spike and slab prior distributions is considered. The key sparsity hyperparameter is calibrated via marginal maximum likelihood empirical Bayes. The plug-in posterior squared-$L^2$ norm is shown to converge at the minimax rate for the euclidean norm for appropriate choices of spike and slab distributions. Possible choices include standard spike and slab with heavy tailed slab, and the spike and slab LASSO of Rocková and George with heavy tailed slab. Surprisingly, the popular Laplace slab is show…
Optimality results for two outstanding Bayesian estimation problems are given in this paper: the estimation of the sampling distribution for the squared total variation function and the estimation of the density for the $L^1$-squared loss function. The posterior predictive distribution provides the solution to these problems. Some examples are presented to illustrate it. The Bayesian estimation problem of a distribution function is also addressed. Consistency of the estimator of the density is proved.
We consider Bayesian inference in inverse regression problems where the objective is to infer about unobserved covariates from observed responses and covariates. We establish posterior consistency of such unobserved covariates in Bayesian inverse regression problemsunder appropriate priors in a leave-one-out cross-validation setup. We relate this to posterior consistency of inverse reference distributions (Bhattacharya (2013)) for assessing model adequacy. We illustrate our theory and methods with various examples of Bayesian inverse regression, along with adequate simulation experiments.
Recently, several researchers have claimed that conclusions obtained from a Bayes factor (or the posterior odds) may contradict those obtained from Bayesian posterior estimation. In this short paper, we wish to point out that no such "incompatibility" exists if one is willing to consistently define one's priors and posteriors. The key for compatibility is that the (implied) prior model odds used for testing are the same as those used for estimation. Our recommendation is simple: If one reports a Bayes factor comparing two models, then one should also report posterior estimates which appropriat…
We consider the problem of sequential learning from categorical observations bounded in [0,1]. We establish an ordering between the Dirichlet posterior over categorical outcomes and a Gaussian posterior under observations with N(0,1) noise. We establish that, conditioned upon identical data with at least two observations, the posterior mean of the categorical distribution will always second-order stochastically dominate the posterior mean of the Gaussian distribution. These results provide a useful tool for the analysis of sequential learning under categorical outcomes.
This paper studies posterior concentration behavior of the base probability measure of a Dirichlet measure, given observations associated with the sampled Dirichlet processes, as the number of observations tends to infinity. The base measure itself is endowed with another Dirichlet prior, a construction known as the hierarchical Dirichlet processes (Teh et al. [J. Amer. Statist. Assoc. 101 (2006) 1566-1581]). Convergence rates are established in transportation distances (i.e., Wasserstein metrics) under various conditions on the geometry of the support of the true base measure. As a consequenc…
Dirichlet process mixture models (DPMM) play a central role in Bayesian nonparametrics, with applications throughout statistics and machine learning. DPMMs are generally used in clustering problems where the number of clusters is not known in advance, and the posterior distribution is treated as providing inference for this number. Recently, however, it has been shown that the DPMM is inconsistent in inferring the true number of components in certain cases. This is an asymptotic result, and it would be desirable to understand whether it holds with finite samples, and to more fully understand t…
We derive the posteror contraction rate for non-parametric Bayesian estimation of a deterministic dispersion coefficient of a linear stochastic differential equation.
We consider the problem of drawing samples from posterior distributions formed under a Dirichlet prior and a truncated multinomial likelihood, by which we mean a Multinomial likelihood function where we condition on one or more counts being zero a priori. Sampling this posterior distribution is of interest in inference algorithms for hierarchical Bayesian models based on the Dirichlet distribution or the Dirichlet process, particularly Gibbs sampling algorithms for the Hierarchical Dirichlet Process Hidden Semi-Markov Model. We provide a data augmentation sampling algorithm that is easy to imp…
Variational and Bayesian methods are two approaches that have been widely used to solve image reconstruction problems. In this paper, we propose original connections between Hamilton--Jacobi (HJ) partial differential equations and a broad class of Bayesian methods and posterior mean estimators with Gaussian data fidelity term and log-concave prior. Whereas solutions to certain first-order HJ PDEs with initial data describe maximum a posteriori estimators in a Bayesian setting, here we show that solutions to some viscous HJ PDEs with initial data describe a broad class of posterior mean estimat…
A Beta-Binomial-Logit model is a Beta-Binomial model with covariate information incorporated via a logistic regression. Posterior propriety of a Bayesian Beta-Binomial-Logit model can be data-dependent for improper hyper-prior distributions. Various researchers in the literature have unknowingly used improper posterior distributions or have given incorrect statements about posterior propriety because checking posterior propriety can be challenging due to the complicated functional form of a Beta-Binomial-Logit model. We derive data-dependent necessary and sufficient conditions for posterior pr…
Sparse Bayesian learning models are typically used for prediction in datasets with significantly greater number of covariates than observations. Such models often take a reproducing kernel Hilbert space (RKHS) approach to carry out the task of prediction and can be implemented using either proper or improper priors. In this article we show that a few sparse Bayesian learning models in the literature, when implemented using improper priors, lead to improper posteriors.
Frequentist-style large-sample properties of Bayesian posterior distributions, such as consistency and convergence rates, are important considerations in nonparametric problems. In this paper we give an analysis of Bayesian asymptotics based primarily on predictive densities. Our analysis is unified in the sense that essentially the same approach can be taken to develop convergence rate results in iid, mis-specified iid, independent non-iid, and dependent data cases.
The study of properties of mean functionals of random probability measures is an important area of research in the theory of Bayesian nonparametric statistics. Many results are now known for random Dirichlet means, but little is known, especially in terms of posterior distributions, for classes of priors beyond the Dirichlet process. In this paper, we consider normalized random measures with independent increments (NRMI's) and mixtures of NRMI. In both cases, we are able to provide exact expressions for the posterior distribution of their means. These general results are then specialized, lead…
We present an alternative approach to the Bayesian nonparametric analysis of conditional species richness under two-parameter Poisson Dirichlet priors. We rely on a known characterization by deletion of classes property and on results for Beta-Binomial distributions. Besides leading to simplified and much more direct proofs, our proposal provides a new scale mixture representation of the conditional asymptotic law.
We study the posterior distribution of the Bayesian multiple change-point regression problem when the number and the locations of the change-points are unknown. While it is relatively easy to apply the general theory to obtain the $O(1/\sqrt{n})$ rate up to some logarithmic factor, showing the exact parametric rate of convergence of the posterior distribution requires additional work and assumptions. Additionally, we demonstrate the asymptotic normality of the segment levels under these assumptions. For inferences on the number of change-points, we show that the Bayesian approach can produce a…
In this paper, we describe a Bayesian nonparametric approach to make inference for a bivariate spherically symmetric distribution. We consider a Dirichlet invariant process prior on the set of all bivariate spherically symmetric distributions and we derive the Dirichlet invariant process posterior. Indeed, our approach is an extension of Dirichlet invariant process for the symmetric distributions on the real line to a bivariate spherically symmetric distribution where the underlying distribution is invariant under a finite group of rotations. Moreover, we obtain the Dirichlet invariant process…
Asymptotic theory of tail index estimation has been studied extensively in the frequentist literature on extreme values, but rarely in the Bayesian context. We investigate whether popular Bayesian kernel mixture models are able to support heavy tailed distributions and consistently estimate the tail index. We show that posterior inconsistency in tail index is surprisingly common for both parametric and nonparametric mixture models. We then present a set of sufficient conditions under which posterior consistency in tail index can be achieved, and verify these conditions for Pareto mixture model…
In recent years, multiple hypothesis testing has come to the forefront of statistical research, ostensibly in relation to applications in genomics and some other emerging fields. The false discovery rate (FDR) and its variants provide very important notions of errors in this context comparable to the role of error probabilities in classical testing problems. Accurate estimation of positive FDR (pFDR), a variant of the FDR, is essential in assessing and controlling this measure. In a recent paper, the authors proposed a model-based nonparametric Bayesian method of estimation of the pFDR functio…
Maximum a posteriori and Bayes estimators are two common methods of point estimation in Bayesian Statistics. It is commonly accepted that maximum a posteriori estimators are a limiting case of Bayes estimators with 0-1 loss. In this paper, we provide a counterexample which shows that in general this claim is false. We then correct the claim that by providing a level-set condition for posterior densities such that the result holds. Since both estimators are defined in terms of optimization problems, the tools of variational analysis find a natural application to Bayesian point estimation.
The classical Bayesian posterior arises naturally as the unique solution of several different optimization problems, without the necessity of interpreting data as conditional probabilities and then using Bayes' Theorem. For example, the classical Bayesian posterior is the unique posterior that minimizes the loss of Shannon information in combining the prior and the likelihood distributions. These results, direct corollaries of recent results about conflations of probability distributions, reinforce the use of Bayesian posteriors, and may help partially reconcile some of the differences between…
Penalized regression methods, such as $L_1$ regularization, are routinely used in high-dimensional applications, and there is a rich literature on optimality properties under sparsity assumptions. In the Bayesian paradigm, sparsity is routinely induced through two-component mixture priors having a probability mass at zero, but such priors encounter daunting computational problems in high dimensions. This has motivated an amazing variety of continuous shrinkage priors, which can be expressed as global-local scale mixtures of Gaussians, facilitating computation. In sharp contrast to the frequent…
We consider the fractional posterior distribution that is obtained by updating a prior distribution via Bayes theorem with a fractional likelihood function, a usual likelihood function raised to a fractional power. First, we analyze the contraction property of the fractional posterior in a general misspecified framework. Our contraction results only require a prior mass condition on certain Kullback-Leibler (KL) neighborhood of the true parameter (or the KL divergence minimizer in the misspecified case), and obviate constructions of test functions and sieves commonly used in the literature for…
The use of non parametric hidden Markov models with finite state space is flourishing in practice while few theoretical guarantees are known in this framework. Here, we study asymptotic guarantees for these models in the Bayesian framework. We obtain posterior concentration rates with respect to the $L_1$-norm on joint marginal densities of consecutive observations in a general theorem. We apply this theorem to two cases and obtain minimax concentration rates. We consider discrete observations with emission distributions distributed from a Dirichlet process and continuous observations with emi…
In the Bayesian approach, the a priori knowledge about the input of a mathematical model is described via a probability measure. The joint distribution of the unknown input and the data is then conditioned, using Bayes' formula, giving rise to the posterior distribution on the unknown input. In this setting we prove posterior consistency for nonlinear inverse problems: a sequence of data is considered, with diminishing fluctuations around a single truth and it is then of interest to show that the resulting sequence of posterior measures arising from this sequence of data concentrates around th…
Mixture models are a natural choice in many applications, but it can be difficult to place an a priori upper bound on the number of components. To circumvent this, investigators are turning increasingly to Dirichlet process mixture models (DPMMs). It is therefore important to develop an understanding of the strengths and weaknesses of this approach. This work considers the MAP (maximum a posteriori) clustering for the Gaussian DPMM (where the cluster means have Gaussian distribution and, for each cluster, the observations within the cluster have Gaussian distribution). Some desirable propertie…
Marginalization techniques are presented for the Bayesian filtering problem under the assumption of Gaussian priors and posteriors and a set of sequentially more constraining state space model assumptions. The techniques provide the capabilities to 1) reduce the filtering problem to essential marginal moment integrals, 2) combine model and numerical approximations to the moment integrals, and 3) decouple modelling and system composition. Closed-form expressions of the posterior means and covariances are developed as functions of subspace projection matrices, subsystem models, and the marginal …
We study the convergence rates of empirical Bayes posterior distributions for nonparametric and high-dimensional inference. We show that as long as the hyperparameter set is discrete, the empirical Bayes posterior distribution induced by the maximum marginal likelihood estimator can be regarded as a variational approximation to a hierarchical Bayes posterior distribution. This connection between empirical Bayes and variational Bayes allows us to leverage the recent results in the variational Bayes literature, and directly obtains the convergence rates of empirical Bayes posterior distributions…
We consider a class of linear ill-posed inverse problems arising from inversion of a compact operator with singular values which decay exponentially to zero. We adopt a Bayesian approach, assuming a Gaussian prior on the unknown function. If the observational noise is assumed to be Gaussian then this prior is conjugate to the likelihood so that the posterior distribution is also Gaussian. We study Bayesian posterior consistency in the small observational noise limit. We assume that the forward operator and the prior and noise covariance operators commute with one another. We show how, for give…
In his 1986 book, Aitchison explains that compositional data is regularly mishandled in statistical analyses, a pattern that continues to this day. The Dirichlet Type I distribution is a multivariate distribution commonly used to model a set of proportions that sum to one. Aitchinson goes on to lament the difficulties of Dirichlet modelling and the scarcity of alternatives. While he addresses the second of these issues, we address the first. The Dirichlet Type II distribution is a transformation of the Dirichlet Type I distribution and is a multivariate distribution on the positive real number…
We consider Bayesian sample size determination using a criterion that utilizes the first two moments of the expected posterior variance. We study the resulting sample size in dependence on the chosen prior and explore the success rate for bounding the posterior variance below a prescribed limit under the true sampling distribution. Compared with sample size determination based on the expected average of the posterior variance the proposed criterion leads to an increase in sample size and significantly improved success rates. Generic asymptotic properties are proven, such as an asymptotic expre…
We show that rate-adaptive multivariate density estimation can be performed using Bayesian methods based on Dirichlet mixtures of normal kernels with a prior distribution on the kernel's covariance matrix parameter. We derive sufficient conditions on the prior specification that guarantee convergence to a true density at a rate that is optimal minimax for the smoothness class to which the true density belongs. No prior knowledge of smoothness is assumed. The sufficient conditions are shown to hold for the Dirichlet location mixture of normals prior with a Gaussian base measure and an inverse-W…
We consider inference in the scalar diffusion model $dX_t=b(X_t)dt+σ(X_t)dW_t$ with discrete data $(X_{jΔ_n})_{0\leq j \leq n}$, $n\to \infty,~Δ_n\to 0$ and periodic coefficients. For $σ$ given, we prove a general theorem detailing conditions under which Bayesian posteriors will contract in $L^2$-distance around the true drift function $b_0$ at the frequentist minimax rate (up to logarithmic factors) over Besov smoothness classes. We exhibit natural nonparametric priors which satisfy our conditions. Our results show that the Bayesian method adapts both to an unknown sampling regime and to unkn…
The main object of Bayesian statistical inference is the determination of posterior distributions. Sometimes these laws are given for quantities devoid of empirical value. This serious drawback vanishes when one confines oneself to considering a finite horizon framework. However, assuming infinite exchangeability gives rise to fairly tractable {\it a posteriori} quantities, which is very attractive in applications. Hence, with a view to a reconciliation between these two aspects of the Bayesian way of reasoning, in this paper we provide quantitative comparisons between posterior distributions …
We consider a Bayesian nonparametric approach to a family of linear inverse problems in a separable Hilbert space setting with Gaussian noise. We assume Gaussian priors, which are conjugate to the model, and present a method of identifying the posterior using its precision operator. Working with the unbounded precision operator enables us to use partial differential equations (PDE) methodology to obtain rates of contraction of the posterior distribution to a Dirac measure centered on the true solution. Our methods assume a relatively weak relation between the prior covariance, noise covariance…
Posterior contractions rates (PCRs) strengthen the notion of Bayesian consistency, quantifying the speed at which the posterior distribution concentrates on arbitrarily small neighborhoods of the true model, with probability tending to 1 or almost surely, as the sample size goes to infinity. Under the Bayesian nonparametric framework, a common assumption in the study of PCRs is that the model is dominated for the observations; that is, it is assumed that the posterior can be written through the Bayes formula. In this paper, we consider the problem of establishing PCRs in Bayesian nonparametric…
The asymmetric Laplace density (ALD) is used as a working likelihood for Bayesian quantile regression. Sriram et al.(2013) derived posterior consistency for Bayesian linear quantile regression based on the misspecified ALD. While their paper also argued for $\sqrt{n}-$consistency, Sriram and Ramamoorthi (2017) highlighted that the argument was only valid for a rate less than $\sqrt{n}$. So, the question of $\sqrt{n}-$rate has remained unaddressed, but is necessary to carry out meaningful Bayesian inference based on the ALD. In this paper, we derive results to obtain posterior consistency rates…
Bayesian model selection is premised on the assumption that the data are generated from one of the postulated models. However, in many applications, all of these models are incorrect (that is, there is misspecification). When the models are misspecified, two or more models can provide a nearly equally good fit to the data, in which case Bayesian model selection can be highly unstable, potentially leading to self-contradictory findings. To remedy this instability, we propose to use bagging on the posterior distribution ("BayesBag") -- that is, to average the posterior model probabilities over m…
In order to estimate model parameters and circumvent possible difficulties encountered with the likelihood function, we propose to replace the likelihood in the formula of the posterior distribution by a function depending on a contrast. The properties of the contrast-based (CB) posterior distribution and MAP estimator are studied to understand what the consequences of incorporating a contrast in the Bayesian formula are. We show that the proposed method can be used to make frequentist inference and allows the reduction of analytical calculations to get the limit variance matrix of the estimat…
Standard Bayesian inference is known to be sensitive to model misspecification, leading to unreliable uncertainty quantification and poor predictive performance. However, finding generally applicable and computationally feasible methods for robust Bayesian inference under misspecification has proven to be a difficult challenge. An intriguing, easy-to-use, and widely applicable approach is to use bagging on the Bayesian posterior ("BayesBag"); that is, to use the average of posterior distributions conditioned on bootstrapped datasets. In this paper, we develop the asymptotic theory of BayesBag,…
Although Bayesian inference is an immensely popular paradigm among a large segment of scientists including statisticians, most applications consider objective priors and need critical investigations (Efron, 2013, Science). While it has several optimal properties, a major drawback of Bayesian inference is the lack of robustness against data contamination and model misspecification, which becomes pernicious in the use of objective priors. This paper presents the general formulation of a Bayes pseudo-posterior distribution yielding robust inference. Exponential convergence results related to the …
We propose a new empirical Bayes approach for inference in the $p \gg n$ normal linear model. The novelty is the use of data in the prior in two ways, for centering and regularization. Under suitable sparsity assumptions, we establish a variety of concentration rate results for the empirical Bayes posterior distribution, relevant for both estimation and model selection. Computation is straightforward and fast, and simulation results demonstrate the strong finite-sample performance of the empirical Bayes model selection procedure.
