This paper studies decision theoretic properties of benchmarked estimators which are of some importance in small area estimation problems. Benchmarking is intended to improve certain aggregate properties (such as study-wide averages) when model based estimates have been applied to individual small areas. We study decision-theoretic properties of such estimators by reducing the problem to one of studying these problems in a related derived problem. For certain such problems, we show that unconstrained solutions in the original (unbenchmarked) problem give unconstrained Bayes and improved estimators which automatically satisfy the benchmark constraint. Also, dominance properties of constrained empirical Bayes estimators are shown in the Fay-Herriot model, a frequently used model in small area estimation.
The problem of estimating a normal covariance matrix is considered from a decision-theoretic point of view, where the dimension of the covariance matrix is larger than the sample size. This paper addresses not only the nonsingular case but also the singular case in terms of the covariance matrix. Based on James and Stein's minimax estimator and on an orthogonally invariant estimator, some classes of estimators are unifiedly defined for any possible ordering on the dimension, the sample size and the rank of the covariance matrix. Unified dominance results on such classes are provided under a St…
We construct empirical Bayes estimators for the reproduction parameter of Borel-Tanner distribution assuming LINEX loss and prove their asymptotic optimality. Some properties of the estimators regret risk are illustrated through simulations.
Strong consistency of the Bayes estimator of a regression curve for the $L^1$-squared loss function is proved. It is also shown the convergence to 0 of the Bayes risk of this estimator both for the $L^1$ and $L^1$-squared loss functions. The Bayes estimator of a regression curve is the regression curve with respect to the posterior predictive distribution.
Under mild conditions, it is shown the strong consistency of the Bayes estimator of the density. Moreover, the Bayes risk (for some common loss functions) of the Bayes estimator of the density (i.e. the posterior predictive density) reaches zero when the sample size goes to $\infty$. In passing, a similar result is obtained for the estimation of the sampling distribution.
A monotone version of an empirical Bayes estimator for the parameter of the Borel-Tanner distribution is constructed. Some properties of the estimator's regret risk are illustrated through simulations.
This article focuses on the performance of Bayes estimators, in comparison with the MLE, in multinomial models with a relatively large number of cells. The prior for the Bayes estimator is taken to be the conjugate Dirichlet, i.e., the multivariate Beta, with exchangeable distributions over the coordinates, including the non-informative uniform distribution. The choice of the multinomial is motivated by its many applications in business and industry, but also by its use in providing a simple nonparametric estimator of an unknown distribution. It is striking that the Bayes procedure outperforms…
Optimality results for two outstanding Bayesian estimation problems are given in this paper: the estimation of the sampling distribution for the squared total variation function and the estimation of the density for the $L^1$-squared loss function. The posterior predictive distribution provides the solution to these problems. Some examples are presented to illustrate it. The Bayesian estimation problem of a distribution function is also addressed. Consistency of the estimator of the density is proved.
In the estimation of the mean matrix in a multivariate normal distribution, the generalized Bayes estimators with closed forms are provided, and the sufficient conditions for their minimaxity are derived relative to both matrix and scalar quadratic loss functions. The generalized Bayes estimators of the covariance matrix are also given with closed forms, and the dominance properties are discussed for the Stein loss function.
We derive minimax generalized Bayes estimators of regression coefficients in the general linear model with spherically symmetric errors under invariant quadratic loss for the case of unknown scale. The class of estimators generalizes the class considered in Maruyama and Strawderman (2005) to include non-monotone shrinkage functions.
Although approximate Bayesian computation (ABC) has become a popular technique for performing parameter estimation when the likelihood functions are analytically intractable there has not as yet been a complete investigation of the theoretical properties of the resulting estimators. In this paper we give a theoretical analysis of the asymptotic properties of ABC based parameter estimators for hidden Markov models and show that ABC based estimators satisfy asymptotically biased versions of the standard results in the statistical literature.
In this article, we consider two forms of shrinkage estimators of the mean $θ$ of a multivariate normal distribution $X\sim N_{p}\left(θ, σ^{2}I_{p}\right)$ where $σ^{2}$ is unknown. We take the prior law $θ\sim N_{p}\left(\upsilon, τ^{2}I_{p}\right)$ and we constuct a Modified Bayes estimator $δ_{B}^{\ast}$ and an Empirical Modified Bayes estimator $δ_{EB}^{\ast}$. We are interested in studying the minimaxity and the limits of risks ratios of these estimators, to the maximum likelihood estimator $X$, when $n$ and $p$ tend to infinity.
Aggregating estimators using exponential weights depending on their risk appears optimal in expectation but not in probability. We use here a slight overpenalization to obtain oracle inequality in probability for such an explicit aggregation procedure. We focus on the fixed design regression framework and the aggregation of affine estimators and obtain results for a large family of affine estimators under a non necessarily independent sub-Gaussian noise assumptions.
The Bayesian approach is effective for inverse problems. The posterior density distribution provides useful information of the unknowns. However, for problems with non-unique solutions, the classical estimators such as the maximum a posterior (MAP) and conditional mean (CM) are not enough. We introduce two new estimators, the local maximum a posterior (LMAP) and local conditional mean (LCM). Their applications are demonstrated by three inverse problems: an inverse spectral problem, an inverse source problem, and an inverse medium problem.
In this paper, we consider the estimation of a mean vector of a multivariate normal population where the mean vector is suspected to be nearly equal to mean vectors of $k-1$ other populations. As an alternative to the preliminary test estimator based on the test statistic for testing hypothesis of equal means, we derive empirical and hierarchical Bayes estimators which shrink the sample mean vector toward a pooled mean estimator given under the hypothesis. The minimaxity of those Bayesian estimators are shown, and their performances are investigated by simulation.
This paper is concerned with the simultaneous estimation of $k$ population means when one suspects that the $k$ means are nearly equal. As an alternative to the preliminary test estimator based on the test statistics for testing hypothesis of equal means, we derive Bayesian and minimax estimators which shrink individual sample means toward a pooled mean estimator given under the hypothesis. It is shown that both the preliminary test estimator and the Bayesian minimax shrinkage estimators are further improved by shrinking the pooled mean estimator. The performance of the proposed shrinkage esti…
We give a sufficient condition for admissibility of generalized Bayes estimators of the location vector of spherically symmetric distribution under squared error loss. Compared to the known results for the multivariate normal case, our sufficient condition is very tight and is close to being a necessary condition. In particular we establish the admissibility of generalized Bayes estimators with respect to the harmonic prior and priors with slightly heavier tail than the harmonic prior. We use the theory of regularly varying functions to construct a sequence of smooth proper priors approaching …
Variational and Bayesian methods are two approaches that have been widely used to solve image reconstruction problems. In this paper, we propose original connections between Hamilton--Jacobi (HJ) partial differential equations and a broad class of Bayesian methods and posterior mean estimators with Gaussian data fidelity term and log-concave prior. Whereas solutions to certain first-order HJ PDEs with initial data describe maximum a posteriori estimators in a Bayesian setting, here we show that solutions to some viscous HJ PDEs with initial data describe a broad class of posterior mean estimat…
In this article we perform an asymptotic analysis of Bayesian parallel kernel density estimators introduced by Neiswanger, Wang and Xing (2014). We derive the asymptotic expansion of the mean integrated squared error for the full data posterior estimator and investigate the properties of asymptotically optimal bandwidth parameters. Our analysis demonstrates that partitioning data into subsets requires a non-trivial choice of bandwidth parameters that optimizes the estimation error.
We consider the asymptotic behaviour of the marginal maximum likelihood empirical Bayes posterior distribution in general setting. First we characterize the set where the maximum marginal likelihood estimator is located with high probability. Then we provide oracle type of upper and lower bounds for the contraction rates of the empirical Bayes posterior. We also show that the hierarchical Bayes posterior achieves the same contraction rate as the maximum marginal likelihood empirical Bayes posterior. We demonstrate the applicability of our general results for various models and prior distributi…
The Bayesian posterior probability of the true state is stochastically dominated by that same posterior under the probability law of the true state. This generalizes to notions of "optimism" about posterior probabilities.
We obtain a limit of a hierarchical Bayes estimator of a finite population mean when the sample size is large. The limit is in the sense of ordinary calculus, where the sample observations are treated as fixed quantities. Our result suggests a simple way to correct the hierarchical Bayes estimator to achieve design-consistency, a well-known property in the traditional randomization approach to finite population sampling. We also suggest three different measures of uncertainty of our proposed estimator.
Maximum a posteriori and Bayes estimators are two common methods of point estimation in Bayesian Statistics. It is commonly accepted that maximum a posteriori estimators are a limiting case of Bayes estimators with 0-1 loss. In this paper, we provide a counterexample which shows that in general this claim is false. We then correct the claim that by providing a level-set condition for posterior densities such that the result holds. Since both estimators are defined in terms of optimization problems, the tools of variational analysis find a natural application to Bayesian point estimation.
We consider the problem of empirical Bayes estimation of multiple variances when provided with sample variances. Assuming an arbitrary prior on the variances, we derive different versions of the Bayes estimators using different loss functions. For one particular loss function, the resulting Bayes estimator relies on the marginal cumulative distribution function of the sample variances only. When replacing it with the empirical distribution function, we obtain an empirical Bayes version called F-modeling based empirical Bayes estimator of variances. We provide theoretical properties of this est…
This article considers the parametric estimation of $Pr(X<Y<Z)$ and its generalizations based on several well-known one-parameter and two-parameter continuous distributions. It is shown that for some one-parameter distributions and when there is a common known parameter in some two-parameter distributions, the uniformly minimum variance unbiased estimator can be expressed as a linear combination of the Appell hypergeometric function of the first type, $F_{1}$ and the hypergeometric functions $_{2}F_{1}$ and $_{3}F_{2}.$ The Bayes estimator based on conjugate gamma priors and Jefferys' non-info…
In the present paper, we derive lower bounds for the risk of the nonparametric empirical Bayes estimators. In order to attain the optimal convergence rate, we propose generalization of the linear empirical Bayes estimation method which takes advantage of the flexibility of the wavelet techniques. We present an empirical Bayes estimator as a wavelet series expansion and estimate coefficients by minimizing the prior risk of the estimator. As a result, estimation of wavelet coefficients requires solution of a well-posed low-dimensional sparse system of linear equations. The dimension of the syste…
We consider the problem of estimating the error variance in a general linear model when the error distribution is assumed to be spherically symmetric, but not necessary Gaussian. In particular we study the case of a scale mixture of Gaussians including the particularly important case of the multivariate-t distribution. Under Stein's loss, we construct a class of estimators that improve on the usual best unbiased (and best equivariant) estimator. Our class has the interesting double robustness property of being simultaneously generalized Bayes (for the same generalized prior) and minimax over t…
In this paper we deal with the regression problem in a random design setting. We investigate asymptotic optimality under minimax point of view of various Bayesian rules based on warped wavelets and show that they nearly attain optimal minimax rates of convergence over the Besov smoothness class considered. Warped wavelets have been introduced recently, they offer very good computable and easy-to-implement properties while being well adapted to the statistical problem at hand. We particularly put emphasis on Bayesian rules leaning on small and large variance Gaussian priors and discuss their si…
Favaro, Lijoi, and Pruenster (2012, Biometrics, 68, 1188--1196) derive a novel Bayesian nonparametric estimator of the probability of detecting at the $(n+m+1)$th observation a species already observed with any given frequency in an enlarged sample of size $n+m$, conditionally on a basic sample of size $n$. Unfortunately the general result under Gibbs priors (Theorem 2), and consequently the explicit result under $(α, θ)$ Poisson-Dirichlet priors (Proposition 3), appear to be wrong. Here we provide the correct formulas for both the results, obtained by means of a new technique devised in Cerqu…
The Bayesian predictive density has complex representation and does not belong to any finite-dimensional statistical model except for in limited situations. In this paper, we introduce its simple approximate representation employing its projection onto a finite-dimensional exponential family. Its theoretical properties are established parallelly to those of the Bayesian predictive density when the model belongs to curved exponential families. It is also demonstrated that the projection asymptotically coincides with the plugin density with the posterior mean of the expectation parameter of the …
Mixture models are well-known for their versatility, and the Bayesian paradigm is a suitable platform for mixture analysis, particularly when the number of components is unknown. Bhattacharya (2008) introduced a mixture model based on the Dirichlet process, where an upper bound on the unknown number of components is to be specified. Here we consider a Bayesian asymptotic framework for objectively specifying the upper bound, which we assume to depend on the sample size. In particular, we define a Bayesian analogue of the mean integrated squared error (Bayesian MISE), and select that form of the…
In order to estimate model parameters and circumvent possible difficulties encountered with the likelihood function, we propose to replace the likelihood in the formula of the posterior distribution by a function depending on a contrast. The properties of the contrast-based (CB) posterior distribution and MAP estimator are studied to understand what the consequences of incorporating a contrast in the Bayesian formula are. We show that the proposed method can be used to make frequentist inference and allows the reduction of analytical calculations to get the limit variance matrix of the estimat…
In frequentist inference, minimizing the Hellinger distance between a kernel density estimate and a parametric family produces estimators that are both robust to outliers and statistically efficienty when the parametric model is correct. This paper seeks to extend these results to the use of nonparametric Bayesian density estimators within disparity methods. We propose two estimators: one replaces the kernel density estimator with the expected posterior density from a random histogram prior; the other induces a posterior over parameters through the posterior for the random histogram. We show t…
We consider the problem of estimating a random state vector when there is information about the maximum distances between its subvectors. The estimation problem is posed in a Bayesian framework in which the minimum mean square error (MMSE) estimate of the state is given by the conditional mean. Since finding the conditional mean requires multidimensional integration, an approximate MMSE estimator is proposed. The performance of the proposed estimator is evaluated in a positioning problem. Finally, the application of the estimator in inequality constrained recursive filtering is illustrated by …
Assuming squared error loss, we show that finding unbiased estimators and Bayes estimators can be treated as using a pair of linear operators that operate between two Hilbert spaces. We note that these integral operators are adjoint and then investigate some consequences of this fact.
We introduce a Bayesian estimator of the underlying class structure in the stochastic block model, when the number of classes is known. The estimator is the posterior mode corresponding to a Dirichlet prior on the class proportions, a generalized Bernoulli prior on the class labels, and a beta prior on the edge probabilities. We show that this estimator is strongly consistent when the expected degree is at least of order $\log^2{n}$, where $n$ is the number of nodes in the network.
We investigate the posterior rate of convergence for wavelet shrinkage using a Bayesian approach in general Besov spaces. Instead of studying the Bayesian estimator related to a particular loss function, we focus on the posterior distribution itself from a nonparametric Bayesian asymptotics point of view and study its rate of convergence. We obtain the same rate as in \citet{abramovich04} where the authors studied the convergence of several Bayesian estimators.
This paper investigates estimation of the mean vector under invariant quadratic loss for a spherically symmetric location family with a residual vector with density of the form $ f(x,u)=η^{(p+n)/2}f(η\{\|x-θ\|^2+\|u\|^2\}) $, where $η$ is unknown. We show that the natural estimator $x$ is admissible for $p=1,2$. Also, for $p\geq 3$, we find classes of generalized Bayes estimators that are admissible within the class of equivariant estimators of the form $\{1-ξ(x/\|u\|)\}x$. In the Gaussian case, a variant of the James--Stein estimator, $[1-\{(p-2)/(n+2)\}/\{\|x\|^2/\|u\|^2+(p-2)/(n+2)+1\}]x$, …
This paper investigates the {\em nonasymptotic} properties of Bayes procedures for estimating an unknown distribution from $n$ i.i.d.\ observations. We assume that the prior is supported by a model $(\scr{S},h)$ (where $h$ denotes the Hellinger distance) with suitable metric properties involving the number of small balls that are needed to cover larger ones. We also require that the prior put enough probability on small balls. We consider two different situations. The simplest case is the one of a parametric model containing the target density for which we show that the posterior concentrates …
We derive the Bayes estimator for the parameters of a multinomial distribution under two loss functions ($1-B$ and $1-B^2$) that are based on the Bhattacharyya coefficient $B(\vec{p},\vec{q}) = \sum{\sqrt{p_kq_k}}$. We formulate a non-commutative generalization relevant to quantum probability theory as an open problem. As an example application, we use our solution to find minimax estimators for a binomial parameter under Bhattacharyya loss ($1-B^2$).
In many statistical problems, stochastic signals can be represented as a sequence of noisy wavelet coefficients. In this paper, we develop general empirical Bayes methods for the estimation of true signal. Our estimators approximate certain oracle separable rules and achieve adaptation to ideal risks and exact minimax risks in broad collections of classes of signals. In particular, our estimators are uniformly adaptive to the minimum risk of separable estimators and the exact minimax risks simultaneously in Besov balls of all smoothness and shape indices, and they are uniformly superefficient …
The paper deals with the non-parametric estimation in the regression with the multiplicative noise. Using the local polynomial fitting and the bayesian approach, we construct the minimax on isotropic Hölder class estimator. Next applying Lepski's method, we propose the estimator which is optimally adaptive over the collection of isotropic Hölder classes. To prove the optimality of the proposed procedure we establish, in particular, the exponential inequality for the deviation of locally bayesian estimator from the parameter to be estimated. These theoretical results are illustrated by simulati…
We present an alternative approach to the Bayesian nonparametric analysis of conditional species richness under two-parameter Poisson Dirichlet priors. We rely on a known characterization by deletion of classes property and on results for Beta-Binomial distributions. Besides leading to simplified and much more direct proofs, our proposal provides a new scale mixture representation of the conditional asymptotic law.
Squared error loss remains the most commonly used loss function for constructing a Bayes estimator of the parameter of interest. However, it can lead to sub-optimal solutions when a parameter is defined in a restricted space. It can also be an inappropriate choice in the context when an extreme overestimation and/or underestimation results in severe consequences and a more conservative estimator is preferred. We advocate a class of loss functions for parameters defined on restricted spaces which infinitely penalize boundary decisions like the squared error loss does on the real line. We also r…
The negative multinomial distribution is a multivariate generalization of the negative binomial distribution. In this paper, we consider the problem of estimating an unknown matrix of probabilities on the basis of observations of negative multinomial variables under the standardized squared error loss. First, a general sufficient condition for a shrinkage estimator to dominate the UMVU estimator is derived and an empirical Bayes estimator satisfying the condition is constructed. Next, a hierarchical shrinkage prior is introduced, an associated Bayes estimator is shown to dominate the UMVU esti…
Formulating a statistical inverse problem as one of inference in a Bayesian model has great appeal, notably for what this brings in terms of coherence, the interpretability of regularisation penalties, the integration of all uncertainties, and the principled way in which the set-up can be elaborated to encompass broader features of the context, such as measurement error, indirect observation, etc. The Bayesian formulation comes close to the way that most scientists intuitively regard the inferential task, and in principle allows the free use of subject knowledge in probabilistic model building…
The topics dicussed in this paper take their origin inthe estimation of the Gram matrix of a random vector from a sample made of n independent copies. They comprise the estimation of the covariance matrix and the study of least squares regression with a random design. We propose four types of results, based on non-asymptotic PAC-Bayesian generalization bounds: a new robust estimator of the Gram matrix and of the covariance matrix, new results on the empirical Gram matrix, new robust least squares estimators and new results on the ordinary least squares estimator, including its exact rate of co…
This paper considers estimation of the predictive density for a normal linear model with unknown variance under alpha-divergence loss for -1 <= alpha <= 1. We first give a general canonical form for the problem, and then give general expressions for the generalized Bayes solution under the above loss for each alpha. For a particular class of hierarchical generalized priors studied in Maruyama and Strawderman (2005, 2006) for the problems of estimating the mean vector and the variance respectively, we give the generalized Bayes predictive density. Additionally, we show that, for a subclass of t…
