We study frequentist properties of Bayesian and $L_0$ model selection, with a focus on (potentially non-linear) high-dimensional regression. We propose a construction to study how posterior probabilities and normalized $L_0$ criteria concentrate on the (Kullback-Leibler) optimal model and other subsets of the model space. When such concentration occurs, one also bounds the frequentist probabilities of selecting the correct model, type I and type II errors. These results hold generally, and help validate the use of posterior probabilities and $L_0$ criteria to control frequentist error probabilities associated to model selection and hypothesis tests. Regarding regression, we help understand the effect of the sparsity imposed by the prior or the $L_0$ penalty, and of problem characteristics such as the sample size, signal-to-noise, dimension and true sparsity. A particular finding is that one may use less sparse formulations than would be asymptotically optimal, but still attain consistency and often also significantly better finite-sample performance. We also prove new results related to misspecifying the mean or covariance structures, and give tighter rates for certain non-local priors than currently available.
The Bayesian posterior probability of the true state is stochastically dominated by that same posterior under the probability law of the true state. This generalizes to notions of "optimism" about posterior probabilities.
In this paper we study posterior consistency for different topologies on the parameters for hidden Markov models with finite state space. We first obtain weak and strong posterior consistency for the marginal density function of finitely many consecutive observations. We deduce posterior consistency for the different components of the parameter. We also obtain posterior consistency for marginal smoothing distributions in the discrete case. We finally apply our results to independent emission probabilities, translated emission probabilities and discrete HMMs, under various types of priors.
This paper adopts a Bayesian nonparametric mixture model where the mixing distribution belongs to the wide class of normalized homogeneous completely random measures. We propose a truncation method for the mixing distribution by discarding the weights of the unnormalized measure smaller than a threshold. We prove convergence in law of our approximation, provide some theoretical properties and characterize its posterior distribution so that a blocked Gibbs sampler is devised. The versatility of the approximation is illustrated by two different applications. In the first the normalized Bessel ra…
This paper focuses on the influence of a misspecified covariance structure on false discovery rate for the large scale multiple testing problem. Specifically, we evaluate the influence on the marginal distribution of local fdr statistics, which are used in many multiple testing procedures and related to Bayesian posterior probabilities. Explicit forms of the marginal distributions under both correctly specified and incorrectly specified models are derived. The Kullback-Leibler divergence is used to quantify the influence caused by a misspecification. Several numerical examples are provided to …
The Bayesian method is noted to produce spuriously high posterior probabilities for phylogenetic trees in analysis of large datasets, but the precise reasons for this over-confidence are unknown. In general, the performance of Bayesian selection of misspecified models is poorly understood, even though this is of great scientific interest since models are never true in real data analysis. Here we characterize the asymptotic behavior of Bayesian model selection and show that when the competing models are equally wrong, Bayesian model selection exhibits surprising and polarized behaviors in large…
Posterior distributions for community structure in sparse planted bi-section models are shown to achieve exact (resp. almost-exact) recovery, with sharp bounds for the sparsity regimes where edge probabilities decrease as $O(\log(n)/n)$ (resp. $O(1/n)$). Assuming posterior recovery, one may interpret credible sets (resp. enlarged credible sets) as asymptotically consistent confidence sets; the diameters of those credible sets are controlled by the rate of posterior concentration. If credible levels are chosen to grow to one quickly enough, corresponding credible sets can be interpreted as freq…
In the standard Bayesian framework data are assumed to be generated by a distribution parametrized by $θ$ in a parameter space $Θ$, over which a prior distribution $π$ is given. A Bayesian statistician quantifies the belief that the true parameter is $θ_{0}$ in $Θ$ by its posterior probability given the observed data. We investigate the behavior of the posterior belief in $θ_{0}$ when the data are generated under some parameter $θ_{1},$ which may or may not be the same as $θ_{0}.$ Starting from stochastic orders, specifically, likelihood ratio dominance, that obtain for resulting distributions…
Consider binary observations whose response probability is an unknown smooth function of a set of covariates. Suppose that a prior on the response probability function is induced by a Gaussian process mapped to the unit interval through a link function. In this paper we study consistency of the resulting posterior distribution. If the covariance kernel has derivatives up to a desired order and the bandwidth parameter of the kernel is allowed to take arbitrarily small values, we show that the posterior distribution is consistent in the $L_1$-distance. As an auxiliary result to our proofs, we sh…
Possible parameter values in a random sampling model are shown by definition to have uniform base-rate prior probabilities. This allows a frequentist posterior probability distribution to be calculated for such possible parameter values conditional solely on actual study observations. If the likelihood probability distribution of a random selection is modelled with a symmetrical continuous function then the frequentist posterior probability of something equal to or more extreme than the null hypothesis will be equal to the P-value; otherwise the P value would be an approximation. An idealistic…
Bayesian model comparison is often based on the posterior distribution over the set of compared models. This distribution is often observed to concentrate on a single model even when other measures of model fit or forecasting ability indicate no strong preference. Furthermore, a moderate change in the data sample can easily shift the posterior model probabilities to concentrate on another model. We document overconfidence in two high-profile applications in economics and neuroscience. To shed more light on the sources of overconfidence we derive the sampling variance of the Bayes factor in uni…
Normalized random measures with independent increments represent a large class of Bayesian nonaprametric priors and are widely used in the Bayesian nonparametric framework. In this paper, we provide the posterior consistency analysis for normalized random measures with independent increments (NRMIs) through the corresponding Levy intensities used to characterize the completely random measures in the construction of NRMIs. Assumptions are introduced on the Levy intensities to analyze the posterior consistency of NRMIs and are verified with multiple interesting examples. A focus of the paper is …
Minimum Bayes factors are commonly used to transform two-sided p-values to lower bounds on the posterior probability of the null hypothesis, as in Pericchi et al. (2017). In this article, we show posterior probabilities of hypothesis by transforming the commonly used -eplog(p), proposed by Vovk (1993) and Sellke et al. (2001). This is achieved after adjusting this minimum Bayes factor with the information to approximate it to an exact Bayes factor, not only when p is a p-value but also when p is a pseudo p-value in the sense of Casella and Berger (2001). Additionally, we show the fit to a refi…
We study the problem of learning Bayesian network structures from data. Koivisto and Sood (2004) and Koivisto (2006) presented algorithms that can compute the exact marginal posterior probability of a subnetwork, e.g., a single edge, in O(n2n) time and the posterior probabilities for all n(n-1) potential edges in O(n2n) total time, assuming that the number of parents per node or the indegree is bounded by a constant. One main drawback of their algorithms is the requirement of a special structure prior that is non uniform and does not respect Markov equivalence. In this paper, we develop an alg…
The statistical inverse problem of estimating the probability distribution of an infinite-dimensional unknown given its noisy indirect observation is studied in the Bayesian framework. In practice, one often considers only finite-dimensional unknowns and investigates numerically their probabilities. As many unknowns are function-valued, it is of interest to know whether the estimated probabilities converge when the finite-dimensional approximations of the unknown are refined. In this work, the generalized Bayes formula is shown to be a powerful tool in the convergence studies. With the help of…
The Australian National University（澳大利亚国立大学）；NICTA（澳大利亚国家信息与通信技术研究院）；Purdue University（普渡大学）
This paper presents theory for Normalized Random Measures (NRMs), Normalized Generalized Gammas (NGGs), a particular kind of NRM, and Dependent Hierarchical NRMs which allow networks of dependent NRMs to be analysed. These have been used, for instance, for time-dependent topic modelling. In this paper, we first introduce some mathematical background of completely random measures (CRMs) and their construction from Poisson processes, and then introduce NRMs and NGGs. Slice sampling is also introduced for posterior inference. The dependency operators in Poisson processes and for the corresponding…
We consider Bayesian nonparametric density estimation using a Pitman-Yor or a normalized inverse-Gaussian process kernel mixture as the prior distribution for a density. The procedure is studied from a frequentist perspective. Using the stick-breaking representation of the Pitman-Yor process or the expression of the finite-dimensional distributions for the normalized-inverse Gaussian process, we prove that, when the data are replicates from an infinitely smooth density, the posterior distribution concentrates on any shrinking $L^p$-norm ball, $1\leq p\leq\infty$, around the sampling density at…
This paper introduces and studies a new class of nonparametric prior distributions. Random probability distribution functions are constructed via normalization of random measures driven by increasing additive processes. In particular, we present results for the distribution of means under both prior and posterior conditions and, via the use of strategic latent variables, undertake a full Bayesian analysis. Our class of priors includes the well-known and widely used mixture of a Dirichlet process.
Bayesian model selection is premised on the assumption that the data are generated from one of the postulated models. However, in many applications, all of these models are incorrect (that is, there is misspecification). When the models are misspecified, two or more models can provide a nearly equally good fit to the data, in which case Bayesian model selection can be highly unstable, potentially leading to self-contradictory findings. To remedy this instability, we propose to use bagging on the posterior distribution ("BayesBag") -- that is, to average the posterior model probabilities over m…
Estimating the marginal likelihoods is an essential feature of model selection in the Bayesian context. It is especially crucial to have good estimates when assessing the number of planets orbiting stars when the models explain the noisy data with different numbers of Keplerian signals. We introduce a simple method for approximating the marginal likelihoods in practice when a statistically representative sample from the parameter posterior density is available. We use our truncated posterior mixture estimate to receive accurate model probabilities for models with differing number of Keplerian …
Reversible jump Markov chain Monte Carlo (RJMCMC) extends ordinary MCMC methods for use in Bayesian multimodel inference. We show that RJMCMC can be implemented as Gibbs sampling with alternating updates of a model indicator and a vector-valued "palette" of parameters denoted $\bm ψ$. Like an artist uses the palette to mix dabs of color for specific needs, we create model-specific parameters from the set available in $\bm ψ$. This description not only removes some of the mystery of RJMCMC, but also provides a basis for fitting models one at a time using ordinary MCMC and computing model weight…
The study of properties of mean functionals of random probability measures is an important area of research in the theory of Bayesian nonparametric statistics. Many results are now known for random Dirichlet means, but little is known, especially in terms of posterior distributions, for classes of priors beyond the Dirichlet process. In this paper, we consider normalized random measures with independent increments (NRMI's) and mixtures of NRMI. In both cases, we are able to provide exact expressions for the posterior distribution of their means. These general results are then specialized, lead…
We put forward a novel calibration of p values, the "Adaptive Robust Lower Bound" (ARLB) which maps p values into approximations of posterior probabilities taking into account the effect of sample sizes. We build on the Robust Lower Bound proposed by Sellke, Bayarri and Berger (2001), but we incorporate a simple power of the sample size to make it adaptive to different amounts of data. We present several illustrations from where it is apparent that the ARLB closely approximates exact Bayes Factors. In particular, it has the same asymptotics as posterior probabilities but avoiding the problems …
An exponential-time exact algorithm is provided for the task of clustering n items of data into k clusters. Instead of seeking one partition, posterior probabilities are computed for summary statistics: the number of clusters, and pairwise co-occurrence. The method is based on subset convolution, and yields the posterior distribution for the number of clusters in O(n * 3^n) operations, or O(n^3 * 2^n) using fast subset convolution. Pairwise co-occurrence probabilities are then obtained in O(n^3 * 2^n) operations. This is considerably faster than exhaustive enumeration of all partitions.
One of the central themes in the classification task is the estimation of class posterior probability at a new point $\bf{x}$. The vast majority of classifiers output a score for $\bf{x}$, which is monotonically related to the posterior probability via an unknown relationship. There are many attempts in the literature to estimate this latter relationship. Here, we provide a way to estimate the posterior probability without resorting to using classification scores. Instead, we vary the prior probabilities of classes in order to derive the ratio of pdf's at point $\bf{x}$, which is directly used…
The classical Bayesian posterior arises naturally as the unique solution of several different optimization problems, without the necessity of interpreting data as conditional probabilities and then using Bayes' Theorem. For example, the classical Bayesian posterior is the unique posterior that minimizes the loss of Shannon information in combining the prior and the likelihood distributions. These results, direct corollaries of recent results about conflations of probability distributions, reinforce the use of Bayesian posteriors, and may help partially reconcile some of the differences between…
Evaluating Posterior Probabilities: Decision Theory, Proper Scoring Rules, and Calibration
Instituto de Ciencias de la Computación（计算机科学研究所）；Universidad de Buenos Aires - CONICET（布宜诺斯艾利斯大学 - 阿根廷国家科学与技术研究理事会）；Escuela Politécnica Superior, Universidad Autónoma de Madrid（马德里自治大学高等理工学院）
In this article, we investigate posterior convergence of nonparametric binary and Poisson regression under possible model misspecification, assuming general stochastic process prior with appropriate properties. Our model setup and objective for binary regression is similar to that of Ghosal and Roy (2006) where the authors have used the approach of entropy bound and exponentially consistent tests with the sieve method to achieve consistency with respect to their Gaussian process prior. In contrast, for both binary and Poisson regression, using general stochastic process prior, our approach inv…
We derive rates of contraction of posterior distributions on nonparametric or semiparametric models based on Gaussian processes. The rate of contraction is shown to depend on the position of the true parameter relative to the reproducing kernel Hilbert space of the Gaussian process and the small ball probabilities of the Gaussian process. We determine these quantities for a range of examples of Gaussian priors and in several statistical settings. For instance, we consider the rate of contraction of the posterior distribution based on sampling from a smooth density model when the prior models t…
This paper concerns the use of Markov chain Monte Carlo methods for posterior sampling in Bayesian nonparametric mixture models with normalized random measure priors. Making use of some recent posterior characterizations for the class of normalized random measures, we propose novel Markov chain Monte Carlo methods of both marginal type and conditional type. The proposed marginal samplers are generalizations of Neal's well-regarded Algorithm 8 for Dirichlet process mixture models, whereas the conditional sampler is a variation of those recently introduced in the literature. For both the margina…
We propose a methodology for modeling and comparing probability distributions within a Bayesian nonparametric framework. Building on dependent normalized random measures, we consider a prior distribution for a collection of discrete random measures where each measure is a linear combination of a set of latent measures, interpretable as characteristic traits shared by different distributions, with positive random weights. The model is non-identified and a method for post-processing posterior samples to achieve identified inference is developed. This uses Riemannian optimization to solve a non-t…
Entropy is a measure of heterogeneity widely used in applied sciences, often when data are collected over space. Recently, a number of approaches has been proposed to include spatial information in entropy. The aim of entropy is to synthesize the observed data in a single, interpretable number. In other studies the objective is, instead, to use data for entropy estimation; several proposals can be found in the literature, which basically are corrections of the estimator based on substituting the involved probabilities with proportions. In this case, independence is assumed and spatial correlat…
Scientific analyses often rely on slow, but accurate forward models for observable data conditioned on known model parameters. While various emulation schemes exist to approximate these slow calculations, these approaches are only safe if the approximations are well understood and controlled. This workshop submission reviews and updates a previously published method, which has been used in cosmological simulations, to (1) train an emulator while simultaneously estimating posterior probabilities with MCMC and (2) explicitly propagate the emulation error into errors on the posterior probabilitie…
Bayesian analysis is increasingly popular for use in social science and other application areas where the data are observations from an informative sample. An informative sampling design leads to inclusion probabilities that are correlated with the response variable of interest. Model inference performed on the observed sample taken from the population will be biased for the population generative model under informative sampling since the balance of information in the sample data is different from that for the population. Typical approaches to account for an informative sampling design under B…
Given a sample of size $n$ from a population of individuals belonging to different species with unknown proportions, a popular problem of practical interest consists in making inference on the probability $D_{n}(l)$ that the $(n+1)$-th draw coincides with a species with frequency $l$ in the sample, for any $l=0,1,\ldots,n$. This paper contributes to the methodology of Bayesian nonparametric inference for $D_{n}(l)$. Specifically, under the general framework of Gibbs-type priors we show how to derive credible intervals for a Bayesian nonparametric estimation of $D_{n}(l)$, and we investigate th…
The detection of change-points in heterogeneous sequences is a statistical challenge with applications across a wide variety of fields. In bioinformatics, a vast amount of methodology exists to identify an ideal set of change-points for detecting Copy Number Variation (CNV). While considerable efficient algorithms are currently available for finding the best segmentation of the data in CNV, relatively few approaches consider the important problem of assessing the uncertainty of the change-point location. Asymptotic and stochastic approaches exist but often require additional model assumptions …
Gaussian covariance graph model is a popular model in revealing underlying dependency structures among random variables. A Bayesian approach to the estimation of covariance structures uses priors that force zeros on some off-diagonal entries of covariance matrices and put a positive definite constraint on matrices. In this paper, we consider a spike and slab prior on off-diagonal entries, which uses a mixture of point-mass and normal distribution. The point-mass naturally introduces sparsity to covariance structures so that the resulting posterior from this prior renders covariance structure l…
We consider the problem of estimation in Hidden Markov models with finite state space and nonparametric emission distributions. Efficient estimators for the transition matrix are exhibited, and a semiparametric Bernstein-von Mises result is deduced. Following from this, we propose a modular approach using the cut posterior to jointly estimate the transition matrix and the emission densities. We derive a general theorem on contraction rates for this approach. We then show how this result may be applied to obtain a contraction rate result for the emission densities in our setting; a key intermed…
Pohang University of Science and Technology（浦项科技大学）
Normalized random measures (NRMs) provide a broad class of discrete random measures that are often used as priors for Bayesian nonparametric models. Dirichlet process is a well-known example of NRMs. Most of posterior inference methods for NRM mixture models rely on MCMC methods since they are easy to implement and their convergence is well studied. However, MCMC often suffers from slow convergence when the acceptance rate is low. Tree-based inference is an alternative deterministic posterior inference method, where Bayesian hierarchical clustering (BHC) or incremental Bayesian hierarchical cl…
Posterior contractions rates (PCRs) strengthen the notion of Bayesian consistency, quantifying the speed at which the posterior distribution concentrates on arbitrarily small neighborhoods of the true model, with probability tending to 1 or almost surely, as the sample size goes to infinity. Under the Bayesian nonparametric framework, a common assumption in the study of PCRs is that the model is dominated for the observations; that is, it is assumed that the posterior can be written through the Bayes formula. In this paper, we consider the problem of establishing PCRs in Bayesian nonparametric…
This paper considers the problem of making statistical inferences about a parameter when a narrow interval centred at a given value of the parameter is considered special, which is interpreted as meaning that there is a substantial degree of prior belief that the true value of the parameter lies in this interval. A clear justification of the practical importance of this problem is provided. The main difficulty with the standard Bayesian solution to this problem is discussed and, as a result, a pseudo-Bayesian solution is put forward based on determining lower limits for the posterior probabili…
Classification approaches based on the direct estimation and analysis of posterior probabilities will degrade if the original class priors begin to change. We prove that a unique (up to scale) solution is possible to recover the data likelihoods for a test example from its original class posteriors and dataset priors. Given the recovered likelihoods and a set of new priors, the posteriors can be re-computed using Bayes' Rule to reflect the influence of the new priors. The method is simple to compute and allows a dynamic update of the original posteriors.
An important task in building regression models is to decide which regressors should be included in the final model. In a Bayesian approach, variable selection can be performed using mixture priors with a spike and a slab component for the effects subject to selection. As the spike is concentrated at zero, variable selection is based on the probability of assigning the corresponding regression effect to the slab component. These posterior inclusion probabilities can be determined by MCMC sampling. In this paper we compare the MCMC implementations for several spike and slab priors with regard t…
Dirichlet process mixture models (DPMM) play a central role in Bayesian nonparametrics, with applications throughout statistics and machine learning. DPMMs are generally used in clustering problems where the number of clusters is not known in advance, and the posterior distribution is treated as providing inference for this number. Recently, however, it has been shown that the DPMM is inconsistent in inferring the true number of components in certain cases. This is an asymptotic result, and it would be desirable to understand whether it holds with finite samples, and to more fully understand t…
In this study, we present a multi-class graphical Bayesian predictive classifier that incorporates the uncertainty in the model selection into the standard Bayesian formalism. For each class, the dependence structure underlying the observed features is represented by a set of decomposable Gaussian graphical models. Emphasis is then placed on the Bayesian model averaging which takes full account of the class-specific model uncertainty by averaging over the posterior graph model probabilities. An explicit evaluation of the model probabilities is well known to be infeasible. To address this issue…
