We propose and analyze nonparametric tests of the null hypothesis that a function belongs to a specified parametric family. The tests are based on BIC approximations, π_{BIC}, to the posterior probability of the null model, and may be carried out in either Bayesian or frequentist fashion. We obtain results on the asymptotic distribution of π_{BIC} under both the null hypothesis and local alternatives. One version of π_{BIC}, call it π_{BIC}^*, uses a class of models that are orthogonal to each other and growing in number without bound as sample size, n, tends to infinity. We show that \sqrtn(1-π_{BIC}^*) converges in distribution to a stable law under the null hypothesis. We also show that π_{BIC}^* can detect local alternatives converging to the null at the rate \sqrt\log n/n. A particularly interesting finding is that the power of the π_{BIC}^*-based test is asymptotically equal to that of a test based on the maximum of alternative log-likelihoods. Simulation results and an example involving variable star data illustrate desirable features of the proposed tests.
The problem of testing mutually exclusive hypotheses with dependent test statistics is considered. Bayesian and frequentist approaches to multiplicity control are studied and compared to help gain understanding as to the effect of test statistic dependence on each approach. The Bayesian approach is shown to have excellent frequentist properties and is argued to be the most effective way of obtaining frequentist multiplicity control, without sacrificing power, when there is considerable test statistic dependence.
There are two main opposing schools of statistical reasoning, Frequentist and Bayesian approaches. Until recent days, the frequentist or classical approach has dominated the scientific research, but Bayesianism has reappeared with a strong impulse that is starting to change the situation. Recently the controversy about the primacy of one of the two approaches seems to be unfinished at a philosophical level, but scientific practices are giving an increasingly important position to the Bayesian approach. This paper eludes philosophical debate to focus on the pragmatic point of view of scientists…
Frequentist coverage of $(1-α)$-highest posterior density (HPD) credible sets is studied in a signal plus noise model under a large class of noise distributions. We consider a specific class of spike-and-slab prior distributions. Different regimes are identified and we derive closed form expressions for the $(1-α)$-HPD on each of these regimes. Similar to the earlier work by Marchand and Strawderman, it is shown that under suitable conditions, the frequentist coverage can drop to $1-3α/2.$
The present paper investigates theoretical performance of various Bayesian wavelet shrinkage rules in a nonparametric regression model with i.i.d. errors which are not necessarily normally distributed. The main purpose is comparison of various Bayesian models in terms of their frequentist asymptotic optimality in Sobolev and Besov spaces. We establish a relationship between hyperparameters, verify that the majority of Bayesian models studied so far achieve theoretical optimality, state which Bayesian models cannot achieve optimal convergence rate and explain why it happens.
In this paper, we study frequentist coverage errors of Bayesian credible sets for an approximately linear regression model with (moderately) high dimensional regressors, where the dimension of the regressors may increase with but is smaller than the sample size. Specifically, we consider quasi-Bayesian inference on the slope vector under the quasi-likelihood with Gaussian error distribution. Under this setup, we derive finite sample bounds on frequentist coverage errors of Bayesian credible rectangles. Derivation of those bounds builds on a novel Berry--Esseen type bound on quasi-posterior dis…
A common concern with Bayesian methodology in scientific contexts is that inferences can be heavily influenced by subjective biases. As presented here, there are two types of bias for some quantity of interest: bias against and bias in favor. Based upon the principle of evidence, it is shown how to measure and control these biases for both hypothesis assessment and estimation problems. Optimality results are established for the principle of evidence as the basis of the approach to these problems. A close relationship is established between measuring bias in Bayesian inferences and frequentist …
For estimating a lower bounded parametric function in the framework of Marchand and Strawderman (2006), we provide through a unified approach a class of Bayesian confidence intervals with credibility $1-α$ and frequentist coverage probability bounded below by $\frac{1-α}{1+α}$. In cases where the underlying pivotal distribution is symmetric, the findings represent extensions with respect to the specification of the credible set achieved through the choice of a {\it spending function}, and include Marchand and Strawderman's HPD procedure result. For non-symmetric cases, the determination of a s…
Bayesian and frequentist methods differ in many aspects, but share some basic optimality properties. In practice, there are situations in which one of the methods is more preferred by some criteria. We consider the case of inference about a set of multiple parameters, which can be divided into two disjoint subsets. On one set, a frequentist method may be favored and on the other, the Bayesian. This motivates a joint estimation procedure in which some of the parameters are estimated Bayesian, and the rest by the maximum-likelihood estimator in the same parametric model, and thus keep the streng…
To the frequentist who computes posteriors, not all priors are useful asymptotically: in this paper Schwartz's 1965 Kullback-Leibler condition is generalised to enable frequentist interpretation of convergence of posterior distributions with the complex models and often dependent datasets in present-day statistical applications. We prove four simple and fully general frequentist theorems, for posterior consistency; for posterior rates of convergence; for consistency of the Bayes factor in hypothesis testing or model selection; and a theorem to obtain confidence sets from credible sets. The lat…
We investigate the relation between frequentist and Bayesian approaches. Namely, we find the "frequentist" Bayes prior π_{f}(λ,x_{obs}) = -\frac{\int_{-\infty}^{x_{obs}}\frac{\partial f(x,λ)}{\partial λ}dx}{f(x_{obs},λ)} (here f(x,λ) is the probability density) for which the results of frequentist and Bayes approaches to the determination of confidence intervals coincide. In many cases (but not always) the "frequentist" prior which reproduces frequentist results coincides with the Jeffreys prior.
Variable selection in the linear regression model takes many apparent faces from both frequentist and Bayesian standpoints. In this paper we introduce a variable selection method referred to as a rescaled spike and slab model. We study the importance of prior hierarchical specifications and draw connections to frequentist generalized ridge regression estimation. Specifically, we study the usefulness of continuous bimodal priors to model hypervariance parameters, and the effect scaling has on the posterior mean through its relationship to penalization. Several model selection strategies, some f…
A theory of quantitative inference about the parameters of sampling distributions is constructed deductively by following very general rules, referred to as the Cox-Polya-Jaynes Desiderata. The inferences are made in terms of probability distributions that are assigned to the parameters. The Desiderata, focusing primarily on consistency of the plausible reasoning, lead to unique assignments of these probabilities in the case of sampling distributions that are invariant under Lie groups. In the scalar cases, e.g. in the case of inferring a single location or scale parameter, the requirement for…
We investigate the frequentist coverage properties of Bayesian credible sets in a general, adaptive, nonparametric framework. It is well known that the construction of adaptive and honest confidence sets is not possible in general. To overcome this problem we introduce an extra assumption on the functional parameters, the so called "general polished tail" condition. We then show that under standard assumptions both the hierarchical and empirical Bayes methods results in honest confidence sets for sieve type of priors in general settings and we characterize their size. We apply the derived abst…
We introduce new estimation methods for a sub-class of the Gaussian scale mixture models for wavelet trees by Wainwright, Simoncelli & Willsky that rely on modern results for composite likelihoods and approximate Bayesian inference. Our methodology is illustrated for denoising and edge detection problems in two-dimensional images.
We investigate the frequentist coverage of Bayesian credible sets in a nonparametric setting. We consider a scale of priors of varying regularity and choose the regularity by an empirical Bayes method. Next we consider a central set of prescribed posterior probability in the posterior distribution of the chosen regularity. We show that such an adaptive Bayes credible set gives correct uncertainty quantification of "polished tail" parameters, in the sense of high probability of coverage of such parameters. On the negative side, we show by theory and example that adaptation of the prior necessar…
For estimating a lower bounded location or mean parameter for a symmetric and logconcave density, we investigate the frequentist performance of the $100(1-α)%$ Bayesian HPD credible set associated with priors which are truncations of flat priors onto the restricted parameter space. Various new properties are obtained. Namely, we identify precisely where the minimum coverage is obtained and we show that this minimum coverage is bounded between $1-\frac{3α}{2}$ and $1-\frac{3α}{2}+\frac{α^2}{1+α}$; with the lower bound $1-\frac{3α}{2}$ improving (for $α\leq 1/3$) on the previously established ([…
A new computation method of frequentist $p$-values and Bayesian posterior probabilities based on the bootstrap probability is discussed for the multivariate normal model with unknown expectation parameter vector. The null hypothesis is represented as an arbitrary-shaped region. We introduce new parametric models for the scaling-law of bootstrap probability so that the multiscale bootstrap method, which was designed for one-sided test, can also computes confidence measures of two-sided test, extending applicability to a wider class of hypotheses. Parameter estimation is improved by the two-step…
Bayesian, frequentist and fiducial (BFF) inferences are much more congruous than they have been perceived historically in the scientific community (cf., Reid and Cox 2015; Kass 2011; Efron 1998). Most practitioners are probably more familiar with the two dominant statistical inferential paradigms, Bayesian inference and frequentist inference. The third, lesser known fiducial inference paradigm was pioneered by R.A. Fisher in an attempt to define an inversion procedure for inference as an alternative to Bayes' theorem. Although each paradigm has its own strengths and limitations subject to thei…
Consider a linear regression model with regression parameter beta and normally distributed errors. Suppose that the parameter of interest is theta = a^T beta where a is a specified vector. Define the parameter tau = c^T beta - t where c and t are specified and a and c are linearly independent. Also suppose that we have uncertain prior information that tau = 0. Kabaila and Giri, 2009, JSPI, describe a new frequentist 1-alpha confidence interval for theta that utilizes this uncertain prior information. We compare this confidence interval with Bayesian 1-alpha equi-tailed and shortest credible in…
In this paper we develop and study adaptive empirical Bayesian smoothing splines. These are smoothing splines with both smoothing parameter and penalty order determined via the empirical Bayes method from the marginal likelihood of the model. The selected order and smoothing parameter are used to construct adaptive credible sets with good frequentist coverage for the underlying regression function. We use these credible sets as a proxy to show the superior performance of adaptive empirical Bayesian smoothing splines compared to frequentist smoothing splines.
Given a sequence ξ_1, ξ_2,... of X-valued, exchangeable random elements, let q(ξ^(n)) and p_m(ξ^(n)) stand for posterior and predictive distribution, respectively, given ξ^(n) = (ξ_1,..., ξ_n). We provide an upper bound for limsup b_n d_[[X]](q(ξ^(n)), δ_\empiricn) and limsup b_n d_[X^m](p_m(ξ^(n)), \empiricn^m), where \empiricn is the empirical measure, b_n is a suitable sequence of positive numbers increasing to +\infty, d_[[X]] and d_[X^m] denote distinguished weak probability distances on [[X]] and [X^m], respectively, with the proviso that [S] denotes the space of all probability measures…
The following zero-sum game between nature and a statistician blends Bayesian methods with frequentist methods such as p-values and confidence intervals. Nature chooses a posterior distribution consistent with a set of possible priors. At the same time, the statistician selects a parameter distribution for inference with the goal of maximizing the minimum Kullback-Leibler information gained over a confidence distribution or other benchmark distribution. An application to testing a simple null hypothesis leads the statistician to report a posterior probability of the hypothesis that is informed…
The major goal of this paper is to study the second order frequentist properties of the marginal posterior distribution of the parametric component in semiparametric Bayesian models, in particular, a second order semiparametric Bernstein-von Mises (BvM) Theorem. Our first contribution is to discover an interesting interference phenomenon between Bayesian estimation and frequentist inferential accuracy: more accurate Bayesian estimation on the nuisance function leads to higher frequentist inferential accuracy on the parametric component. As the second contribution, we propose a new class of dep…
We propose modified frequentist definition for the determination of confidence intervals for the case of Poisson statistics. Namely, we require that 1-β' \geq \sum_{n=o}^{n_{obs}+k} P(n|λ) \geq α'. We show that this definition is equivalent to the Bayesian method with prior π(λ) \sim λ^{k}. We also propose modified frequentist definition for the case of nonzero background.
Bayesian properties of the signed root likelihood ratio statistic are analysed. Conditions for first-order probability matching are derived by the examination of the Bayesian posterior and frequentist means of this statistic. Second-order matching conditions are shown to arise from matching of the Bayesian posterior and frequentist variances of a mean-adjusted version of the signed root statistic. Conditions for conditional probability matching in ancillary statistic models are derived and discussed.
In statistical practice, whether a Bayesian or frequentist approach is used in inference depends not only on the availability of prior information but also on the attitude taken toward partial prior information, with frequentists tending to be more cautious than Bayesians. The proposed framework defines that attitude in terms of a specified amount of caution, thereby enabling data analysis at the level of caution desired and on the basis of any prior information. The caution parameter represents the attitude toward partial prior information in much the same way as a loss function represents th…
Between Bayesian and frequentist inference, it's commonly believed that the former is for cases where one has a prior and the latter is for cases where one has no prior. But the prior/no-prior classification isn't exhaustive, and most real-world applications fit somewhere in between these two extremes. That neither of the two dominant schools of thought are suited for these applications creates confusion and slows progress. A key observation here is that ``no prior information'' actually means no prior distribution can be ruled out, so the classically-frequentist context is best characterized …
University of Tübingen（图宾根大学）；MPI for Intelligent Systems, Tübingen（图宾根智能系统研究所）
Despite their compelling theoretical properties, Bayesian neural networks (BNNs) tend to perform worse than frequentist methods in classification-based uncertainty quantification (UQ) tasks such as out-of-distribution (OOD) detection. In this paper, based on empirical findings in prior works, we hypothesize that this issue is because even recent Bayesian methods have never considered OOD data in their training processes, even though this "OOD training" technique is an integral part of state-of-the-art frequentist UQ methods. To validate this, we treat OOD data as a first-class citizen in BNN t…
Using a collection of simulated an real benchmarks, we compare Bayesian and frequentist regularization approaches under a low informative constraint when the number of variables is almost equal to the number of observations on simulated and real datasets. This comparison includes new global noninformative approaches for Bayesian variable selection built on Zellner's g-priors that are similar to Liang et al. (2008). The interest of those calibration-free proposals is discussed. The numerical experiments we present highlight the appeal of Bayesian regularization methods, when compared with non-B…
This paper explores the intrinsic connections between the Bayesian false discovery rate (FDR) control procedures and their counterpart of frequentist procedures. We attempt to offer a unified view of FDR control within and beyond the setting of testing exchangeable hypotheses. Under the standard two-groups model and the Oracle condition, we show that the Bayesian and the frequentist methods can achieve asymptotically equivalent FDR control at arbitrary levels. Built on this result, we further illustrate that rigorous post-fitting model diagnosis is necessary and effective to ensure robust FDR …
There are three principle paradigms of statistics: Bayesian, frequentist and information-based inference. Although these paradigms are in agreement in some contexts, the Lindley paradox describes a class of problems, models of unknown dimension, where conflicting conclusions are generated by frequentist and Bayesian inference. This conflict can materially affect the scientific conclusions. Understanding the Lindley paradox---where it applies, why it occurs, and how it can be avoided---is therefore essential to the understanding of statistical analysis. In this paper, we revisit the Lindley par…
Analogues of the frequentist chi-square and F tests are proposed for testing goodness-of-fit and consistency for Bayesian models. Simple examples exhibit these tests' detection of inconsistency between consecutive experiments with identical parameters, when the first experiment provides the prior for the second. In a related analysis, a quantitative measure is derived for judging the degree of tension between two different experiments with partially overlapping parameter vectors.
We study frequentist asymptotic properties of Bayesian procedures for high-dimensional Gaussian sparse regression when unknown nuisance parameters are involved. Nuisance parameters can be finite-, high-, or infinite-dimensional. A mixture of point masses at zero and continuous distributions is used for the prior distribution on sparse regression coefficients, and appropriate prior distributions are used for nuisance parameters. The optimal posterior contraction of sparse regression coefficients, hampered by the presence of nuisance parameters, is also examined and discussed. It is shown that t…
The primary objective of this paper is to revisit and make a case for the merits of R.A. Fisher's objections to the decision-theoretic framing of frequentist inference. It is argued that this framing is congruent with the Bayesian but incongruent with the frequentist inference. It provides the Bayesian approach with a theory of optimal inference, but it misrepresents the theory of optimal frequentist inference by framing inferences solely in terms of the universal quantifier `for all values of theta in the parameter space'. This framing is at odds with the primary objective of model-based freq…
Approximate confidence distribution computing (ACDC) offers a new take on the rapidly developing field of likelihood-free inference from within a frequentist framework. The appeal of this computational method for statistical inference hinges upon the concept of a confidence distribution, a special type of estimator which is defined with respect to the repeated sampling principle. An ACDC method provides frequentist validation for computational inference in problems with unknown or intractable likelihoods. The main theoretical contribution of this work is the identification of a matching condit…
In many application domains, such as computational biology, the goal of graphical model structure learning is to uncover discrete relationships between entities. For example, in our problem of interest concerning HIV vaccine design, we want to infer which HIV peptides interact with which immune system molecules (HLA molecules). For problems of this nature, we are interested in determining the number of nonspurious arcs in a learned graphical model. We describe both a Bayesian and frequentist approach to this problem. In the Bayesian approach, we use the posterior distribution over model struct…
Suppose that a compound Poisson process is observed discretely in time and assume that its jump distribution is supported on the set of natural numbers. In this paper we propose a non-parametric Bayesian approach to estimate the intensity of the underlying Poisson process and the distribution of the jumps. We provide a MCMC scheme for obtaining samples from the posterior. We apply our method on both simulated and real data examples, and compare its performance with the frequentist plug-in estimator proposed by Buchmann and Grübel. On a theoretical side, we study the posterior from the frequent…
