We consider Bayesian shrinkage predictions for the Normal regression problem under the frequentist Kullback-Leibler risk function.
Firstly, we consider the multivariate Normal model with an unknown mean and a known covariance. While the unknown mean is fixed, the covariance of future samples can be different from training samples. We show that the Bayesian predictive distribution based on the uniform prior is dominated by that based on a class of priors if the prior distributions for the covariance and future covariance matrices are rotation invariant.
Then, we consider a class of priors for the mean parameters depending on the future covariance matrix. With such a prior, we can construct a Bayesian predictive distribution dominating that based on the uniform prior.
Lastly, applying this result to the prediction of response variables in the Normal linear regression model, we show that there exists a Bayesian predictive distribution dominating that based on the uniform prior. Minimaxity of these Bayesian predictions follows from these results.
We investigate shrinkage priors for constructing Bayesian predictive distributions. It is shown that there exist shrinkage predictive distributions asymptotically dominating Bayesian predictive distributions based on the Jeffreys prior or other vague priors if the model manifold satisfies some differential geometric conditions. Kullback--Leibler divergence from the true distribution to a predictive distribution is adopted as a loss function. Conformal transformations of model manifolds corresponding to vague priors are introduced. We show several examples where shrinkage predictive distributio…
During the past decade, shrinkage priors have received much attention in Bayesian analysis of high-dimensional data. This paper establishes the posterior consistency for high-dimensional linear regression with a class of shrinkage priors, which has a heavy and flat tail and allocates a sufficiently large probability mass in a very small neighborhood of zero. While enjoying its efficiency in posterior simulations, the shrinkage prior can lead to a nearly optimal posterior contraction rate and variable selection consistency as the spike-and-slab prior. Our numerical results show that under the p…
We develop singular value shrinkage priors for the mean matrix parameters in the matrix-variate normal model with known covariance matrices. Our priors are superharmonic and put more weight on matrices with smaller singular values. They are a natural generalization of the Stein prior. Bayes estimators and Bayesian predictive densities based on our priors are minimax and dominate those based on the uniform prior in finite samples. In particular, our priors work well when the true value of the parameter has low rank.
The present paper investigates theoretical performance of various Bayesian wavelet shrinkage rules in a nonparametric regression model with i.i.d. errors which are not necessarily normally distributed. The main purpose is comparison of various Bayesian models in terms of their frequentist asymptotic optimality in Sobolev and Besov spaces. We establish a relationship between hyperparameters, verify that the majority of Bayesian models studied so far achieve theoretical optimality, state which Bayesian models cannot achieve optimal convergence rate and explain why it happens.
Shrinkage prior are becoming more and more popular in Bayesian modeling for high dimensional sparse problems due to its computational efficiency. Recent works show that a polynomially decaying prior leads to satisfactory posterior asymptotics under regression models. In the literature, statisticians have investigated how the global shrinkage parameter, i.e., the scale parameter, in a heavy tail prior affects the posterior contraction. In this work, we explore how the shape of the prior, or more specifically, the polynomial order of the prior tail affects the posterior. We discover that, under …
Penalized regression methods, such as $L_1$ regularization, are routinely used in high-dimensional applications, and there is a rich literature on optimality properties under sparsity assumptions. In the Bayesian paradigm, sparsity is routinely induced through two-component mixture priors having a probability mass at zero, but such priors encounter daunting computational problems in high dimensions. This has motivated an amazing variety of continuous shrinkage priors, which can be expressed as global-local scale mixtures of Gaussians, facilitating computation. In sharp contrast to the correspo…
We consider nonparametric Bayesian estimation and prediction for nonhomogeneous Poisson process models with unknown intensity functions. We propose a class of improper priors for intensity functions. Nonparametric Bayesian inference with kernel mixture based on the class improper priors is shown to be useful, although improper priors have not been widely used for nonparametric Bayes problems. Several theorems corresponding to those for finite-dimensional independent Poisson models hold for nonhomogeneous Poisson process models with infinite-dimensional parameter spaces. Bayesian estimation and…
In this article, we consider two forms of shrinkage estimators of the mean $θ$ of a multivariate normal distribution $X\sim N_{p}\left(θ, σ^{2}I_{p}\right)$ where $σ^{2}$ is unknown. We take the prior law $θ\sim N_{p}\left(\upsilon, τ^{2}I_{p}\right)$ and we constuct a Modified Bayes estimator $δ_{B}^{\ast}$ and an Empirical Modified Bayes estimator $δ_{EB}^{\ast}$. We are interested in studying the minimaxity and the limits of risks ratios of these estimators, to the maximum likelihood estimator $X$, when $n$ and $p$ tend to infinity.
We use Levy processes to generate joint prior distributions, and therefore penalty functions, for a location parameter as p grows large. This generalizes the class of local-global shrinkage rules based on scale mixtures of normals, illuminates new connections among disparate methods, and leads to new results for computing posterior means and modes under a wide class of priors. We extend this framework to large-scale regularized regression problems where p>n, and provide comparisons with other methodologies.
We consider the problem of learning the structure of a high dimensional precision matrix under sparsity assumptions. We propose to use a shrinkage prior, called the DL-graphical prior based on the Dirichlet-Laplace prior used for the Gaussian mean problem. A posterior sampling scheme based on Gibbs sampling is also provided along with theoretical guarantees of the method by obtaining the posterior convergence rate of the precision matrix.
High-dimensional data, where the number of variables exceeds or is comparable to the sample size, is now pervasive in many scientific applications. In recent years, Bayesian shrinkage models have been developed as effective and computationally feasible tools to analyze such data, especially in the context of linear regression. In this paper, we focus on the Normal-Gamma shrinkage model developed by Griffin and Brown. This model subsumes the popular Bayesian lasso model, and a three-block Gibbs sampling algorithm to sample from the resulting intractable posterior distribution has been developed…
In this paper, we treat estimation and prediction problems where negative multinomial variables are observed and in particular consider unbalanced settings. First, the problem of estimating multiple negative multinomial parameter vectors under the standardized squared error loss is treated and a new empirical Bayes estimator which dominates the UMVU estimator under suitable conditions is derived. Second, we consider estimation of the joint predictive density of several multinomial tables under the Kullback-Leibler divergence and obtain a sufficient condition under which the Bayesian predictive…
Monash University（莫纳什大学）；The University of Melbourne（墨尔本大学）
Global-local shrinkage hierarchies are an important innovation in Bayesian estimation. We propose the use of log-scale distributions as a novel basis for generating familes of prior distributions for local shrinkage hyperparameters. By varying the scale parameter one may vary the degree to which the prior distribution promotes sparsity in the coefficient estimates. By examining the class of distributions over the logarithm of the local shrinkage parameter that have log-linear, or sub-log-linear tails, we show that many standard prior distributions for local shrinkage parameters can be unified …
The negative multinomial distribution is a multivariate generalization of the negative binomial distribution. In this paper, we consider the problem of estimating an unknown matrix of probabilities on the basis of observations of negative multinomial variables under the standardized squared error loss. First, a general sufficient condition for a shrinkage estimator to dominate the UMVU estimator is derived and an empirical Bayes estimator satisfying the condition is constructed. Next, a hierarchical shrinkage prior is introduced, an associated Bayes estimator is shown to dominate the UMVU esti…
We consider the problem of estimation and structure learning of high dimensional signals via a normal sequence model, where the underlying parameter vector is piecewise constant, or has a block structure. We develop a Bayesian fusion estimation method by using the Horseshoe prior to induce a strong shrinkage effect on successive differences in the mean parameters, simultaneously imposing sufficient prior concentration for non-zero values of the same. The proposed method thus facilitates consistent estimation and structure recovery of the signal pieces. We provide theoretical justifications of …
This paper is concerned with the simultaneous estimation of $k$ population means when one suspects that the $k$ means are nearly equal. As an alternative to the preliminary test estimator based on the test statistics for testing hypothesis of equal means, we derive Bayesian and minimax estimators which shrink individual sample means toward a pooled mean estimator given under the hypothesis. It is shown that both the preliminary test estimator and the Bayesian minimax shrinkage estimators are further improved by shrinking the pooled mean estimator. The performance of the proposed shrinkage esti…
Shrinkage prior has gained great successes in many data analysis, however, its applications mostly focus on the Bayesian modeling of sparse parameters. In this work, we will apply Bayesian shrinkage to model high dimensional parameter that possesses an unknown blocking structure. We propose to impose heavy-tail shrinkage prior, e.g., $t$ prior, on the differences of successive parameter entries, and such a fusion prior will shrink successive differences towards zero and hence induce posterior blocking. Comparing to conventional Bayesian fused lasso which implements Laplace fusion prior, $t$ fu…
The present work describes simulation studies to compare the performances of bayesian wavelet shrinkage methods in estimating component curves from aggregated functional data. To do so, five methods were considered: the bayesian shrinkage rule under logistic prior by Sousa (2020), bayesian shrinkage rule under beta prior by Sousa et al. (2020), Large Posterior Mode method by Cutillo et al. (2008), Amplitude-scale invariant Bayes Estimator by Figueiredo and Nowak (2001) and Bayesian Adaptive Multiresolution Smoother by Vidakovic and Ruggeri (2001). Further, the so called Donoho-Johnstone test f…
Wavelet shrinkage estimators are widely applied in several fields of science for denoising data in wavelet domain by reducing the magnitudes of empirical coefficients. In nonparametric regression problem, most of the shrinkage rules are derived from models composed by an unknown function with additive gaussian noise. Although gaussian noise assumption is reasonable in several real data analysis, mainly for large sample sizes, it is not general. Contaminated data with positive noise can occur in practice and nonparametric regression models with positive noise bring challenges in wavelet shrinka…
Selecting a subset of variables for linear models remains an active area of research. This paper reviews many of the recent contributions to the Bayesian model selection and shrinkage prior literature. A posterior variable selection summary is proposed, which distills a full posterior distribution over regression coefficients into a sequence of sparse linear predictors.
In wavelet shrinkage and thresholding, most of the standard techniques do not consider information that wavelet coefficients might be bounded, although information about bounded energy in signals can be readily available. To address this, we present a Bayesian approach for shrinkage of bounded wavelet coefficients in the context of non-parametric regression. We propose the use of a zero-contaminated beta distribution with a support symmetric around zero as the prior distribution for the location parameter in the wavelet domain in models with additive gaussian errors. The hyperparameters of the…
The estimation of the mean matrix of the multivariate normal distribution is addressed in the high dimensional setting. Efron-Morris-type linear shrinkage estimators based on ridge estimators for the precision matrix instead of the Moore-Penrose generalized inverse are considered, and the weights in the ridge-type linear shrinkage estimators are estimated in terms of minimizing the Stein unbiased risk estimators under the quadratic loss. It is shown that the ridge-type linear shrinkage estimators with the estimated weights are minimax, and that the estimated weights converge to the optimal wei…
We investigate predictive densities for multivariate normal models with unknown mean vectors and known covariance matrices. Bayesian predictive densities based on shrinkage priors often have complex representations, although they are effective in various problems. We consider extended normal models with mean vectors and covariance matrices as parameters, and adopt predictive densities that belong to the extended models including the original normal model. We adopt predictive densities that are optimal with respect to the posterior Bayes risk in the extended models. The proposed predictive dens…
The paper discusses shrinkage priors which impose increasing shrinkage in a sequence of parameters. We review the cumulative shrinkage process (CUSP) prior of Legramanti et al. (2020), which is a spike-and-slab shrinkage prior where the spike probability is stochastically increasing and constructed from the stick-breaking representation of a Dirichlet process prior. As a first contribution, this CUSP prior is extended by involving arbitrary stick-breaking representations arising from beta distributions. As a second contribution, we prove that exchangeable spike-and-slab priors, which are popul…
Department of Statistics, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校统计学系）
We consider a Bayesian framework for estimating a high-dimensional sparse precision matrix, in which adaptive shrinkage and sparsity are induced by a mixture of Laplace priors. Besides discussing our formulation from the Bayesian standpoint, we investigate the MAP (maximum a posteriori) estimator from a penalized likelihood perspective that gives rise to a new non-convex penalty approximating the $\ell_0$ penalty. Optimal error rates for estimation consistency in terms of various matrix norms along with selection consistency for sparse structure recovery are shown for the unique MAP estimator …
It is increasingly being realised that many real world time series are not stationary and exhibit evolving second-order autocovariance or spectral structure. This article introduces a Bayesian approach for modelling the evolving wavelet spectrum of a locally stationary wavelet time series. Our new method works by combining the advantages of a Haar-Fisz transformed spectrum with a simple, but powerful, Bayesian wavelet shrinkage method. Our new method produces excellent and stable spectral estimates and this is demonstrated via simulated data and on differenced infant ECG data. A major addition…
Most estimates for penalised linear regression can be viewed as posterior modes for an appropriate choice of prior distribution. Bayesian shrinkage methods, particularly the horseshoe estimator, have recently attracted a great deal of attention in the problem of estimating sparse, high-dimensional linear models. This paper extends these ideas, and presents a Bayesian grouped model with continuous global-local shrinkage priors to handle complex group hierarchies that include overlapping and multilevel group structures. As the posterior mean is never a sparse estimate of the linear model coeffic…
NTT Advanced Technology Corporation（NTT先进技术公司）；The University of Electro-Communications（电气通信大学）
Sparse convex clustering is to cluster observations and conduct variable selection simultaneously in the framework of convex clustering. Although a weighted $L_1$ norm is usually employed for the regularization term in sparse convex clustering, its use increases the dependence on the data and reduces the estimation accuracy if the sample size is not sufficient. To tackle these problems, this paper proposes a Bayesian sparse convex clustering method based on the ideas of Bayesian lasso and global-local shrinkage priors. We introduce Gibbs sampling algorithms for our method using scale mixtures …
Bayesian nonparametric regression with dependent wavelets has dual shrinkage properties: there is shrinkage through a dependent prior put on functional differences, and shrinkage through the setting of most of the wavelet coefficients to zero through Bayesian variable selection methods. The methodology can deal with unequally spaced data and is efficient because of the existence of fast moves in model space for the MCMC computation. The methodology is illustrated on the problem of modeling the oscillations of Cepheid variable stars; these are a class of pulsating variable stars with the useful…
Kyoto University（京都大学）；National Institute of Informatics（国立信息学研究所）
A common strategy for sparse linear regression is to introduce regularization, which eliminates irrelevant features by letting the corresponding weights be zeros. However, regularization often shrinks the estimator for relevant features, which leads to incorrect feature selection. Motivated by the above-mentioned issue, we propose Bayesian masking (BM), a sparse estimation method which imposes no regularization on the weights. The key concept of BM is to introduce binary latent variables that randomly mask features. Estimating the masking rates determines the relevance of the features automati…
We investigate the asymptotic behavior of posterior distributions of regression coefficients in high-dimensional linear models as the number of dimensions grows with the number of observations. We show that the posterior distribution concentrates in neighborhoods of the true parameter under simple sufficient conditions. These conditions hold under popular shrinkage priors given some sparsity assumptions.
We provide a framework for assessing the default nature of a prior distribution using the property of regular variation, which we study for global-local shrinkage priors. In particular, we demonstrate the horseshoe priors, originally designed to handle sparsity, also possess regular variation and thus are appropriate for default Bayesian analysis. To illustrate our methodology, we solve a problem of non-informative priors due to Efron (1973), who showed standard flat non-informative priors in high-dimensional normal means model can be highly informative for nonlinear parameters of interest. We…
Shrinkage priors for Bayesian Substitute Confounders
School of Mathematical Sciences, University of Nottingham, Nottingham, UK（诺丁汉大学数学科学学院）；Department of Statistics, Rice University, USA（里士满大学统计学系；伯克利国家实验室）；Lawrence Berkeley National Laboratory, USA（洛斯阿拉莫斯国家实验室统计科学组）；Statistical Sciences Group, Los Alamos National Laboratory, USA
In this paper, we consider Bayesian variable selection problem of linear regression model with global-local shrinkage priors on the regression coefficients. We propose a variable selection procedure that select a variable if the ratio of the posterior mean to the ordinary least square estimate of the corresponding coefficient is greater than $1/2$. Under the assumption of orthogonal designs, we show that if the local parameters have polynomial-tailed priors, our proposed method enjoys the oracle property in the sense that it can achieve variable selection consistency and optimal estimation rat…
Prior distributions for high-dimensional linear regression require specifying a joint distribution for the unobserved regression coefficients, which is inherently difficult. We instead propose a new class of shrinkage priors for linear regression via specifying a prior first on the model fit, in particular, the coefficient of determination, and then distributing through to the coefficients in a novel way. The proposed method compares favourably to previous approaches in terms of both concentration around the origin and tail behavior, which leads to improved performance both in posterior contra…
Large Bayesian VARs are now widely used in empirical macroeconomics. One popular shrinkage prior in this setting is the natural conjugate prior as it facilitates posterior simulation and leads to a range of useful analytical results. This is, however, at the expense of modeling flexibility, as it rules out cross-variable shrinkage -- i.e., shrinking coefficients on lags of other variables more aggressively than those on own lags. We develop a prior that has the best of both worlds: it can accommodate cross-variable shrinkage, while maintaining many useful analytical results, such as a closed-f…
The present paper proposes a bayesian approach for wavelet shrinkage with the use of a shrinkage prior based on the generalized secant hyperbolic distribution symmetric around zero in a nonparemetric regression problem. This shrinkage prior allows the control of the kurtosis of the coefficients, which impacts on the level of shrinkage on its extreme values. Statistical properties such as bias, variance, classical and bayesian risks of the rule are analyzed and performances of the proposed rule are obtained in simulations studies involving the Donoho-Johnstone test functions. Application of the…
Macroeconomists using large datasets often face the choice of working with either a large Vector Autoregression (VAR) or a factor model. In this paper, we develop methods for combining the two using a subspace shrinkage prior. Subspace priors shrink towards a class of functions rather than directly forcing the parameters of a model towards some pre-specified location. We develop a conjugate VAR prior which shrinks towards the subspace which is defined by a factor model. Our approach allows for estimating the strength of the shrinkage as well as the number of factors. After establishing the the…
This paper extends the idea of decoupling shrinkage and sparsity for continuous priors to Bayesian Quantile Regression (BQR). The procedure follows two steps: In the first step, we shrink the quantile regression posterior through state of the art continuous priors and in the second step, we sparsify the posterior through an efficient variant of the adaptive lasso, the signal adaptive variable selection (SAVS) algorithm. We propose a new variant of the SAVS which automates the choice of penalisation through quantile specific loss-functions that are valid in high dimensions. We show in large sca…
