We propose a new prediction method for multivariate linear regression problems where the number of features is less than the sample size but the number of outcomes is extremely large. Many popular procedures, such as penalized regression procedures, require parameter tuning that is computationally untenable in such large-scale problems. We take a different approach, motivated by ideas from simultaneous estimation problems, that performs linear shrinkage on ordinary least squares parameter estimates. Our approach is extremely computationally efficient and tuning-free. We show that it can asymptotically outperform ordinary least squares without any structural assumptions on the true regression coefficients and illustrate its good performance in simulations and an analysis of single-cell RNA-seq data.
In this work, the estimation of the multivariate normal mean by different classes of shrinkage estimators is investigated. The risk associated with the balanced loss function is used to compare two estimators. We start by considering estimators that generalize the James-Stein estimator and show that these estimators dominate the maximum likelihood estimator (MLE), therefore are minimax, when the shrinkage function satisfies some conditions. Then, we treat estimators of polynomial form and prove the increase of the degree of the polynomial allows us to build a better estimator from the one prev…
We study shrinkage estimation of the mean parameters of a class of multivariate distributions for which the diagonal entries of the corresponding covariance matrix are certain quadratic functions of the mean parameter. This class of distributions includes the diagonal multivariate natural exponential families. We propose two classes of semi-parametric shrinkage estimators for the mean and construct unbiased estimators of the corresponding risk. We establish the asymptotic consistency and convergence rates for these shrinkage estimators under squared error loss as both $n$, the sample size, and…
We are interested in testing general linear hypotheses in a high-dimensional multivariate linear regression model. The framework includes many well-studied problems such as two-sample tests for equality of population means, MANOVA and others as special cases. A family of rotation-invariant tests is proposed that involves a flexible spectral shrinkage scheme applied to the sample error covariance matrix. The asymptotic normality of the test statistic under the null hypothesis is derived in the setting where dimensionality is comparable to sample sizes, assuming the existence of certain moments …
We propose a new class of estimators of the multivariate response linear regression coefficient matrix that exploits the assumption that the response and predictors have a joint multivariate Normal distribution. This allows us to indirectly estimate the regression coefficient matrix through shrinkage estimation of the parameters of the inverse regression, or the conditional distribution of the predictors given the responses. We establish a convergence rate bound for estimators in our class and we study two examples. The first example estimator exploits an assumption that the inverse regression…
A collection of robust Mahalanobis distances for multivariate outlier detection is proposed, based on the notion of shrinkage. Robust intensity and scaling factors are optimally estimated to define the shrinkage. Some properties are investigated, such as affine equivariance and breakdown value. The performance of the proposal is illustrated through the comparison to other techniques from the literature, in a simulation study and with a real dataset. The behavior when the underlying distribution is heavy-tailed or skewed, shows the appropriateness of the method when we deviate from the common a…
This work is motivated by the need to accurately model a vector of responses related to pediatric functional status using administrative health data from inpatient rehabilitation visits. The components of the responses have known and structured interrelationships. To make use of these relationships in modeling, we develop a two-pronged regularization approach to borrow information across the responses. The first component of our approach encourages joint selection of the effects of each variable across possibly overlapping groups related responses and the second component encourages shrinkage …
Statistical inference for sparse covariance matrices is crucial to reveal dependence structure of large multivariate data sets, but lacks scalable and theoretically supported Bayesian methods. In this paper, we propose beta-mixture shrinkage prior, computationally more efficient than the spike and slab prior, for sparse covariance matrices and establish its minimax optimality in high-dimensional settings. The proposed prior consists of beta-mixture shrinkage and gamma priors for off-diagonal and diagonal entries, respectively. To ensure positive definiteness of the resulting covariance matrix,…
We consider sparse Bayesian estimation in the classical multivariate linear regression model with $p$ regressors and $q$ response variables. In univariate Bayesian linear regression with a single response $y$, shrinkage priors which can be expressed as scale mixtures of normal densities are popular for obtaining sparse estimates of the coefficients. In this paper, we extend the use of these priors to the multivariate case to estimate a $p \times q$ coefficients matrix $\mathbf{B}$. We derive sufficient conditions for posterior consistency under the Bayesian multivariate linear regression frame…
The James-Stein estimator is an estimator of the multivariate normal mean and dominates the maximum likelihood estimator (MLE) under squared error loss. The original work inspired great interest in developing shrinkage estimators for a variety of problems. Nonetheless, research on shrinkage estimation for manifold-valued data is scarce. In this paper, we propose shrinkage estimators for the parameters of the Log-Normal distribution defined on the manifold of $N \times N$ symmetric positive-definite matrices. For this manifold, we choose the Log-Euclidean metric as its Riemannian metric since i…
In a remarkable series of papers beginning in 1956, Charles Stein set the stage for the future development of minimax shrinkage estimators of a multivariate normal mean under quadratic loss. More recently, parallel developments have seen the emergence of minimax shrinkage estimators of multivariate normal predictive densities under Kullback--Leibler risk. We here describe these parallels emphasizing the focus on Bayes procedures and the derivation of the superharmonic conditions for minimaxity as well as further developments of new minimax shrinkage predictive density estimators including mult…
In the course of evolution, proteins show a remarkable conservation of their three-dimensional structure and their biological function, leading to strong evolutionary constraints on the sequence variability between homologous proteins. Our method aims at extracting such constraints from rapidly accumulating sequence data, and thereby at inferring protein structure and function from sequence information alone. Recently, global statistical inference methods (e.g. direct-coupling analysis, sparse inverse covariance estimation) have achieved a breakthrough towards this aim, and their predictions h…
We investigate predictive densities for multivariate normal models with unknown mean vectors and known covariance matrices. Bayesian predictive densities based on shrinkage priors often have complex representations, although they are effective in various problems. We consider extended normal models with mean vectors and covariance matrices as parameters, and adopt predictive densities that belong to the extended models including the original normal model. We adopt predictive densities that are optimal with respect to the posterior Bayes risk in the extended models. The proposed predictive dens…
The possibility of improving on the usual multivariate normal confidence was first discussed in Stein (1962). Using the ideas of shrinkage, through Bayesian and empirical Bayesian arguments, domination results, both analytic and numerical, have been obtained. Here we trace some of the developments in confidence set estimation.
In this paper, a nonparametric estimator is proposed for estimating the L1-median for multivariate conditional distribution when the covariates take values in an infinite dimensional space. The multivariate case is more appropriate to predict the components of a vector of random variables simultaneously rather than predicting each of them separately. While estimating the conditional L1-median function using the well-known Nadarya-Waston estimator, we establish the strong consistency of this estimator as well as the asymptotic normality. We also present some simulations and provide how to built…
Portfolio balancing requires estimates of covariance between asset returns. Returns data have histories which greatly vary in length, since assets begin public trading at different times. This can lead to a huge amount of missing data--too much for the conventional imputation-based approach. Fortunately, a well-known factorization of the MVN likelihood under the prevailing historical missingness pattern leads to a simple algorithm of OLS regressions that is much more reliable. When there are more assets than returns, however, OLS becomes unstable. Gramacy, et al. (2008), showed how classical s…
We consider the problem of adaptive estimation of the functional component in a multivariate partial linear model where the argument of the function is defined on a $q$-dimensional grid. Obtaining an adaptive estimator of this functional component is an important practical problem in econometrics where exact distributions of random errors and the parametric component are mostly unknown and cannot safely assumed to be normal. An estimator of the functional component that is adaptive in the mean squared sense over the wide range of multivariate Besov classes and robust to a wide choice of distri…
Shrinkage for time-varying parameter (TVP) models is investigated within a Bayesian framework, with the aim to automatically reduce time-varying parameters to static ones, if the model is overfitting. This is achieved through placing the double gamma shrinkage prior on the process variances. An efficient Markov chain Monte Carlo scheme is developed, exploiting boosting based on the ancillarity-sufficiency interweaving strategy. The method is applicable both to TVP models for univariate as well as multivariate time series. Applications include a TVP generalized Phillips curve for EU area inflat…
A robust estimator is proposed for the parameters that characterize the linear regression problem. It is based on the notion of shrinkages, often used in Finance and previously studied for outlier detection in multivariate data. A thorough simulation study is conducted to investigate: the efficiency with normal and heavy-tailed errors, the robustness under contamination, the computational times, the affine equivariance and breakdown value of the regression estimator. Two classical data-sets often used in the literature and a real socio-economic data-set about the Living Environment Deprivation…
Factor Analysis is a popular method for modeling dependence in multivariate data. However, determining the number of factors and obtaining a sparse orientation of the loadings are still major challenges. In this paper, we propose a decision-theoretic approach that brings to light the relation between a sparse representation of the loadings and factor dimension. This relation is done through a summary from information contained in the multivariate posterior. To construct such summary, we introduce a three-step approach. In the first step, the model is fitted with a conservative factor dimension…
Spectral density matrix estimation of multivariate time series is a classical problem in time series and signal processing. In modern neuroscience, spectral density based metrics are commonly used for analyzing functional connectivity among brain regions. In this paper, we develop a non-asymptotic theory for regularized estimation of high-dimensional spectral density matrices of Gaussian and linear processes using thresholded versions of averaged periodograms. Our theoretical analysis ensures that consistent estimation of spectral density matrix of a $p$-dimensional time series using $n$ sampl…
A uniform shrinkage prior (USP) distribution on the unknown variance component of a random-effects model is known to produce good frequency properties. The USP has a parameter that determines the shape of its density function, but it has been neglected whether the USP can maintain such good frequency properties regardless of the choice for the shape parameter. We investigate which choice for the shape parameter of the USP produces Bayesian interval estimates of random effects that meet their nominal confidence levels better than several existent choices in the literature. Using univariate and …
We propose a flexible class of models based on scale mixture of uniform distributions to construct shrinkage priors for covariance matrix estimation. This new class of priors enjoys a number of advantages over the traditional scale mixture of normal priors, including its simplicity and flexibility in characterizing the prior density. We also exhibit a simple, easy to implement Gibbs sampler for posterior simulation which leads to efficient estimation in high dimensional problems. We first discuss the theory and computational details of this new approach and then extend the basic model to a new…
Methods for choosing a fixed set of knot locations in additive spline models are fairly well established in the statistical literature. While most of these methods are in principle directly extendable to non-additive surface models, they are less likely to be successful in that setting because of the curse of dimensionality, especially when there are more than a couple of covariates. We propose a regression model for a multivariate Gaussian response that combines both additive splines and interactive splines, and a highly efficient MCMC algorithm that updates all the knot locations jointly. We…
This paper deals with the problem of the multivariate copula density estimation. Using wavelet methods we provide two shrinkage procedures based on thresholding rules for which the knowledge of the regularity of the copula density to be estimated is not necessary. These methods, said to be adaptive, are proved to perform very well when adopting the minimax and the maxiset approaches. Moreover we show that these procedures can be discriminated in the maxiset sense. We produce an estimation algorithm whose qualities are evaluated thanks some simulation. Last, we propose a real life application f…
The percentage of patients hospitalized due to hyponatremia is getting higher. Hyponatremia is the deficiency of sodium electrolyte in the human serum. This deficiency might indulge adverse effects and also associated with longer hospital stay or mortality, if it wasnt actively treated and managed. This work predicts the futuristic sodium levels of patients based on their history of health problems using multilayer perceptron and multivariate linear regression algorithm. This work analyses the patients age, information about other disease such as diabetes, pneumonia, liver-disease, malignancy,…
The negative multinomial distribution is a multivariate generalization of the negative binomial distribution. In this paper, we consider the problem of estimating an unknown matrix of probabilities on the basis of observations of negative multinomial variables under the standardized squared error loss. First, a general sufficient condition for a shrinkage estimator to dominate the UMVU estimator is derived and an empirical Bayes estimator satisfying the condition is constructed. Next, a hierarchical shrinkage prior is introduced, an associated Bayes estimator is shown to dominate the UMVU esti…
The known connection between shrinkage estimation, empirical Bayes, and mixed effects models is explored and applied to balanced and unbalanced designs in which the responses are correlated. As an illustration, a mixed model is proposed for predicting the outcome of English Premier League games that takes into account both home and away team effects. Results based on empirical best linear unbiased predictors obtained from fitting mixed linear models are compared with fully Bayesian predictors that utilize prior information from the previous season.
In this work, we study the positive definiteness (PDness) problem in covariance matrix estimation. For high dimensional data, many regularized estimators are proposed under structural assumptions on the true covariance matrix including sparsity. They are shown to be asymptotically consistent and rate-optimal in estimating the true covariance matrix and its structure. However, many of them do not take into account the PDness of the estimator and produce a non-PD estimate. To achieve the PDness, researchers consider additional regularizations (or constraints) on eigenvalues, which make both the …
Multivariate compositional count data arise in many applications including ecology, microbiology, genetics, and paleoclimate. A frequent question in the analysis of multivariate compositional count data is what values of a covariate(s) give rise to the observed composition. Learning the relationship between covariates and the compositional count allows for inverse prediction of unobserved covariates given compositional count observations. Gaussian processes provide a flexible framework for modeling functional responses with respect to a covariate without assuming a functional form. Many scient…
This paper describes a method to estimate a production frontier that satisfies the axioms of monotonicity and concavity in a non-parametric Bayesian setting. An inefficiency term that allows for significant departure from prior distributional assumptions is jointly estimated in a single stage with parametric prior assumptions. We introduce heteroscedasticity into the inefficiency terms by local hyperplane-specific shrinkage hyperparameters and impose monotonicity using bound-constrained local nonlinear regression. Our minimum-of-hyperplanes estimator imposes concavity. Our Monte Carlo simulati…
We propose a nonparametric method for detecting nonlinear causal relationship within a set of multidimensional discrete time series, by using sparse additive models (SpAMs). We show that, when the input to the SpAM is a $β$-mixing time series, the model can be fitted by first approximating each unknown function with a linear combination of a set of B-spline bases, and then solving a group-lasso-type optimization problem with nonconvex regularization. Theoretically, we characterize the oracle statistical properties of the proposed sparse estimator in function estimation and model selection. Num…
Physical processes rarely occur in isolation, rather they influence and interact with one another. Thus, there is great benefit in modeling potential dependence between both spatial locations and different processes. It is the interaction between these two dependencies that is the focus of Genton and Kleiber's paper under discussion. We see the problem of ensuring that any multivariate spatial covariance matrix is nonnegative definite as important, but we also see it as a means to an end. That "end" is solving the scientific problem of predicting a multivariate field. [arXiv:1507.08017].
We show how to extract the implicit copula of a response vector from a Bayesian regularized regression smoother with Gaussian disturbances. The copula can be used to compare smoothers that employ different shrinkage priors and function bases. We illustrate with three popular choices of shrinkage priors --- a pairwise prior, the horseshoe prior and a g prior augmented with a point mass as employed for Bayesian variable selection --- and both univariate and multivariate function bases. The implicit copulas are high-dimensional, have flexible dependence structures that are far from that of a Gaus…
This paper proposes a new method and algorithm for predicting multivariate responses in a regression setting. Research into classification of High Dimension Low Sample Size (HDLSS) data, in particular microarray data, has made considerable advances, but regression prediction for high-dimensional data with continuous responses has had less attention. Recently Bair et al (2006) proposed an efficient prediction method based on supervised principal component regression (PCR). Motivated by the fact that a larger number of principal components results in better regression performance, this paper ext…
Since the advent of the horseshoe priors for regularization, global-local shrinkage methods have proved to be a fertile ground for the development of Bayesian methodology in machine learning, specifically for high-dimensional regression and classification problems. They have achieved remarkable success in computation, and enjoy strong theoretical support. Most of the existing literature has focused on the linear Gaussian case; see Bhadra et al. (2019b) for a systematic survey. The purpose of the current article is to demonstrate that the horseshoe regularization is useful far more broadly, by …
