In a smooth semiparametric estimation problem, the marginal posterior for the parameter of interest is expected to be asymptotically normal and satisfy frequentist criteria of optimality if the model is endowed with a suitable prior. It is shown that, under certain straightforward and interpretable conditions, the assertion of Le Cam's acclaimed, but strictly parametric, Bernstein-von Mises theorem [Univ. California Publ. Statist. 1 (1953) 277-329] holds in the semiparametric situation as well. As a consequence, Bayesian point-estimators achieve efficiency, for example, in the sense of Hájek's convolution theorem [Z. Wahrsch. Verw. Gebiete 14 (1970) 323-330]. The model is required to satisfy differentiability and metric entropy conditions, while the nuisance prior must assign nonzero mass to certain Kullback-Leibler neighborhoods [Ghosal, Ghosh and van der Vaart Ann. Statist. 28 (2000) 500-531]. In addition, the marginal posterior is required to converge at parametric rate, which appears to be the most stringent condition in examples. The results are applied to estimation of the linear coefficient in partial linear regression, with a Gaussian prior on a smoothness class for the nuisance.
A Bernstein-von Mises theorem is derived for general semiparametric functionals. The result is applied to a variety of semiparametric problems in i.i.d. and non-i.i.d. situations. In particular, new tools are developed to handle semiparametric bias, in particular for nonlinear functionals and in cases where regularity is possibly low. Examples include the squared $L^2$-norm in Gaussian white noise, nonlinear functionals in density estimation, as well as functionals in autoregressive models. For density estimation, a systematic study of BvM results for two important classes of priors is provide…
This paper brings a contribution to the Bayesian theory of nonparametric and semiparametric estimation. We are interested in the asymptotic normality of the posterior distribution in Gaussian linear regression models when the number of regressors increases with the sample size. Two kinds of Bernstein-von Mises Theorems are obtained in this framework: nonparametric theorems for the parameter itself, and semiparametric theorems for functionals of the parameter. We apply them to the Gaussian sequence model and to the regression of functions in Sobolev and $C^α$ classes, in which we get the minima…
Bernstein-von Mises theorems for nonparametric Bayes priors in the Gaussian white noise model are proved. It is demonstrated how such results justify Bayes methods as efficient frequentist inference procedures in a variety of concrete nonparametric problems. Particularly Bayesian credible sets are constructed that have asymptotically exact $1-α$ frequentist coverage level and whose $L^2$-diameter shrinks at the minimax rate of convergence (within logarithmic factors) over Hölder balls. Other applications include general classes of linear and nonlinear functionals and credible bands for auto-co…
The prominent Bernstein -- von Mises (BvM) result claims that the posterior distribution after centering by the efficient estimator and standardizing by the square root of the total Fisher information is nearly standard normal. In particular, the prior completely washes out from the asymptotic posterior distribution. This fact is fundamental and justifies the Bayes approach from the frequentist viewpoint. In the nonparametric setup the situation changes dramatically and the impact of prior becomes essential even for the contraction of the posterior; see [vdV2008], [Bo2011], [CaNi2013,CaNi2014]…
Maximum likelihood estimators are proposed for the parameters and the densities in a semiparametric density ratio model in which the nonparametric baseline density is approximated by the Bernstein polynomial model. The EM algorithm is used to obtain the maximum approximate Bernstein likelihood estimates. Simulation study shows that the performance of the proposed method is much better than the existing ones. The proposed method is illustrated by real data examples. Some asymptotic results are also presented and proved.
In this paper, we consider nonparametric multidimensional finite mixture models and we are interested in the semiparametric estimation of the population weights. Here, the i.i.d. observations are assumed to have at least three components which are independent given the population. We approximate the semiparametric model by projecting the conditional distributions on step functions associated to some partition. Our first main result is that if we refine the partition slowly enough, the associated sequence of maximum likelihood estimators of the weights is asymptotically efficient, and the poste…
Bernstein estimators are well-known to avoid the boundary bias problem of traditional kernel estimators. The theoretical properties of these estimators have been studied extensively on compact intervals and hypercubes, but never on the simplex, except for the mean squared error of the density estimator in Tenbusch (1994) when $d = 2$. The simplex is an important case as it is the natural domain of compositional data. In this paper, we make an effort to prove several asymptotic results (bias, variance, mean squared error (MSE), mean integrated squared error (MISE), asymptotic normality, uniform…
We provide a general theoretical framework to derive Bernstein-von Mises theorems for matrix functionals. The conditions on functionals and priors are explicit and easy to check. Results are obtained for various functionals including entries of covariance matrix, entries of precision matrix, quadratic forms, log-determinant, eigenvalues in the Bayesian Gaussian covariance/precision matrix estimation setting, as well as for Bayesian linear and quadratic discriminant analysis.
The major goal of this paper is to study the second order frequentist properties of the marginal posterior distribution of the parametric component in semiparametric Bayesian models, in particular, a second order semiparametric Bernstein-von Mises (BvM) Theorem. Our first contribution is to discover an interesting interference phenomenon between Bayesian estimation and frequentist inferential accuracy: more accurate Bayesian estimation on the nuisance function leads to higher frequentist inferential accuracy on the parametric component. As the second contribution, we propose a new class of dep…
In this paper, we study the asymptotic posterior distribution of linear functionals of the density. In particular, we give general conditions to obtain a semiparametric version of the Bernstein-Von Mises theorem. We then apply this general result to nonparametric priors based on infinite dimensional exponential families. As a byproduct, we also derive adaptive nonparametric rates of concentration of the posterior distributions under these families of priors on the class of Sobolev and Besov spaces.
This paper revisits the prominent Fisher, Wilks, and Bernstein -- von Mises (BvM) results from different viewpoints. Particular issues to address are: nonasymptotic framework with just one finite sample, possible model misspecification, and a large parameter dimension. In particular, in the case of an i.i.d. sample, the mentioned results can be stated for any smooth parametric family provided that the dimension \(p \) of the parameter space satisfies the condition "\(p^{2}/n \) is small" for the Fisher expansion, while the Wilks and the BvM results require "\(p^{3}/n \) is small".
We prove a Bernstein-von Mises theorem for a general class of high dimensional nonlinear Bayesian inverse problems in the vanishing noise limit. We propose a sufficient condition on the growth rate of the number of unknown parameters under which the posterior distribution is asymptotically normal. This growth condition is expressed explicitly in terms of the model dimension, the degree of ill-posedness of the inverse problem and the noise parameter. The theoretical results are applied to a Bayesian estimation of the medium parameter in an elliptic problem.
We investigate Bernstein-von Mises theorems for adaptive nonparametric Bayesian procedures in the canonical Gaussian white noise model. We consider both a Hilbert space and multiscale setting with applications in $L^2$ and $L^\infty$ respectively. This provides a theoretical justification for plug-in procedures, for example the use of certain credible sets for sufficiently smooth linear functionals. We use this general approach to construct optimal frequentist confidence sets based on the posterior distribution. We also provide simulations to numerically illustrate our approach and obtain a vi…
We continue the investigation of Bernstein-von Mises theorems for nonparametric Bayes procedures from [Ann. Statist. 41 (2013) 1999-2028]. We introduce multiscale spaces on which nonparametric priors and posteriors are naturally defined, and prove Bernstein-von Mises theorems for a variety of priors in the setting of Gaussian nonparametric regression and in the i.i.d. sampling model. From these results we deduce several applications where posterior-based inference coincides with efficient frequentist procedures, including Donsker- and Kolmogorov-Smirnov theorems for the random posterior cumula…
In the recent Bayesian nonparametric literature, many examples have been reported in which Bayesian estimators and posterior distributions do not achieve the optimal convergence rate, indicating that the Bernstein-von Mises theorem does not hold. In this article, we give a positive result in this direction by showing that the Bernstein-von Mises theorem holds in survival models for a large class of prior processes neutral to the right. We also show that, for an arbitrarily given convergence rate n^{-α} with 0<α\leq 1/2, a prior process neutral to the right can be chosen so that its posterior d…
We demonstrate that a prior influence on the posterior distribution of covariance matrix vanishes as sample size grows. The assumptions on a prior are explicit and mild. The results are valid for a finite sample and admit the dimension $p$ growing with the sample size $n$. We exploit the described fact to derive the finite sample Bernstein-von Mises theorem for functionals of covariance matrix (e.g. eigenvalues) and to find the posterior distribution of the Frobenius distance between spectral projector and empirical spectral projector. This can be useful for constructing sharp confidence sets …
This note considers a finite dimensional statistical model for the Calderón problem with piecewise constant conductivities. In this setting it is shown that injectivity of the forward map and its linearisation suffice to prove the invertibility of the information operator, resulting in a Bernstein--von-Mises theorem and optimality guarantees for estimation by Bayesian posterior means.
The classical parametric and semiparametric Bernstein -- von Mises (BvM) results are reconsidered in a non-classical setup allowing finite samples and model misspecification. In the case of a finite dimensional nuisance parameter we obtain an upper bound on the error of Gaussian approximation of the posterior distribution for the target parameter which is explicit in the dimension of the nuisance and target parameters. This helps to identify the so called \emph{critical dimension} $ p $ of the full parameter for which the BvM result is applicable. In the important i.i.d. case, we show that the…
In a smooth semiparametric model, the marginal posterior distribution of the finite dimensional parameter of interest is expected to be asymptotically equivalent to the sampling distribution of frequentist's efficient estimators. This is the assertion of the so-called Bernstein-von Mises theorem, and recently, it has been proved in many interesting semiparametric models. In this thesis, we consider the semiparametric Bernstein-von Mises theorem in some models which have symmetric errors. The simplest example of these models is the symmetric location model that has 1-dimensional location parame…
We study large sample properties of Bayesian analysis of the proportional hazard model with neutral to the right process priors on the baseline hazard function. We show that the posterior distribution of the baseline cumulative hazard function and regression coefficients centered at the maximum likelihood estimator is jointly asymptotically equivalent to the sampling distribution of the maximum likelihood estimator.
In a smooth semi-parametric model, the marginal posterior distribution for a finite dimensional parameter of interest is expected to be asymptotically equivalent to the sampling distribution of any efficient point-estimator. The assertion leads to asymptotic equivalence of credible and confidence sets for the parameter of interest and is known as the semi-parametric Bernstein-von Mises theorem. In recent years, it has received much attention and has been applied in many examples. We consider models in which errors with symmetric densities play a role; more specifically, it is shown that the ma…
We study the asymptotic behaviour of the posterior distribution in a broad class of statistical models where the "true" solution occurs on the boundary of the parameter space. We show that in this case Bayesian inference is consistent, and that the posterior distribution has not only Gaussian components as in the case of regular models (the Bernstein-von Mises theorem) but also has Gamma distribution components whose form depends on the behaviour of the prior distribution near the boundary and have a faster rate of convergence. We also demonstrate a remarkable property of Bayesian inference, t…
We establish that Laplace transforms of the posterior Dirichlet process converge to those of the limiting Brownian bridge process in a neighbourhood about zero, uniformly over Glivenko-Cantelli function classes. For real-valued random variables and functions of bounded variation, we strengthen this result to hold for all real numbers. This last result is proved via an explicit strong approximation coupling inequality.
We consider the statistical inverse problem of recovering an unknown function $f$ from a linear measurement corrupted by additive Gaussian white noise. We employ a nonparametric Bayesian approach with standard Gaussian priors, for which the posterior-based reconstruction of $f$ corresponds to a Tikhonov regulariser $\bar f$ with a reproducing kernel Hilbert space norm penalty. We prove a semiparametric Bernstein-von Mises theorem for a large collection of linear functionals of $f$, implying that semiparametric posterior estimation and uncertainty quantification are valid and optimal from a fre…
We review the Bayesian theory of semiparametric inference following Bickel and Kleijn (2012) and Kleijn and Knapik (2013). After an overview of efficiency in parametric and semiparametric estimation problems, we consider the Bernstein-von Mises theorem (see, e.g., Le Cam and Yang (1990)) and generalize it to (LAN) regular and (LAE) irregular semiparametric estimation problems. We formulate a version of the semiparametric Bernstein-von Mises theorem that does not depend on least-favourable submodels, thus bypassing the most restrictive condition in the presentation of Bickel and Kleijn (2012). …
Few methods in Bayesian non-parametric statistics/ machine learning have received as much attention as Bayesian Additive Regression Trees (BART). While BART is now routinely performed for prediction tasks, its theoretical properties began to be understood only very recently. In this work, we continue the theoretical investigation of BART initiated by Rockova and van der Pas (2017). In particular, we study the Bernstein-von Mises (BvM) phenomenon (i.e. asymptotic normality) for smooth linear functionals of the regression surface within the framework of non-parametric regression with fixed covar…
We study nonparametric Bayesian statistical inference for the parameters governing a pure jump process of the form $$Y_t = \sum_{k=1}^{N(t)} Z_k,~~~ t \ge 0,$$ where $N(t)$ is a standard Poisson process of intensity $λ$, and $Z_k$ are drawn i.i.d.~from jump measure $μ$. A high-dimensional wavelet series prior for the Lévy measure $ν= λμ$ is devised and the posterior distribution arises from observing discrete samples $Y_Δ, Y_{2Δ}, \dots, Y_{nΔ}$ at fixed observation distance $Δ$, giving rise to a nonlinear inverse inference problem. We derive contraction rates in uniform norm for the posterior…
Many useful parameters depend on nonparametric first steps. Examples include games, dynamic discrete choice, average exact consumer surplus, and treatment effects. Often estimators of these parameters are asymptotically equivalent to a sample average of an object referred to as the influence function. The influence function is useful in local policy analysis, in evaluating local sensitivity of estimators, constructing debiased machine learning estimators, in efficiency comparisons, and in formulating primitive regularity conditions for asymptotic normality, We show that the influence function …
The Pitman-Yor process is a random probability distribution, that can be used as a prior distribution in a nonparametric Bayesian analysis. The process is of species sampling type and generates discrete distributions, which yield of the order $n^σ$ different values ("species") in a random sample of size $n$, if the type $σ$ is positive. Thus this type parameter can be set to target true distributions of various levels of discreteness, making the Pitman-Yor process an interesting prior in this case. It was previously shown that the resulting posterior distribution is consistent if and only if t…
Generalized likelihoods are commonly used to obtain consistent estimators with attractive computational and robustness properties. Formally, any generalized likelihood can be used to define a generalized posterior distribution, but an arbitrarily defined "posterior" cannot be expected to appropriately quantify uncertainty in any meaningful sense. In this article, we provide sufficient conditions under which generalized posteriors exhibit concentration, asymptotic normality (Bernstein-von Mises), an asymptotically correct Laplace approximation, and asymptotically correct frequentist coverage. W…
Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Bernstein-von Mises results (BvM) establish that the Laplace approximation is asymptotically correct in the large-data limit. However, these results are inappropriate for computational purposes since they only hold over most, and not all, datasets and involve hard-to-estimate constants. In this article, I present a new BvM theorem which bounds the Kullback-Leibler (KL) divergence between a fixed log-concave density $f\left(\boldsymbolθ\right)$ and its Laplace approximation. The bound goes to $0$ as the higher-derivatives of $f\left(\boldsymbolθ\right)$ tend to $0$ and $f\left(\boldsymbolθ\righ…
We investigate the asymptotic normality of the posterior distribution in the discrete setting, when model dimension increases with sample size. We consider a probability mass function $θ_0$ on $\mathbbm{N}\setminus \{0\}$ and a sequence of truncation levels $(k_n)_n$ satisfying $k_n^3\leq n\inf_{i\leq k_n}θ_0(i).$ Let $\hatθ$ denote the maximum likelihood estimate of $(θ_0(i))_{i\leq k_n}$ and let $Δ_n(θ_0)$ denote the $k_n$-dimensional vector which $i$-th coordinate is defined by \sqrt{n} (\hatθ_n(i)-θ_0(i)) for $1\leq i\leq k_n.$ We check that under mild conditions on $θ_0$ and on the sequen…
