Let X be a n*p matrix and l_1 the largest eigenvalue of the covariance matrix X^{*}*X. The "null case" where X_{i,j} are independent Normal(0,1) is of particular interest for principal component analysis. For this model, when n, p tend to infinity and n/p tends to gamma in (0,\infty), it was shown in Johnstone (2001) that l_1, properly centered and scaled, converges to the Tracy-Widom law. We show that with the same centering and scaling, the result is true even when p/n or n/p tends to infinity. The derivation uses ideas and techniques quite similar to the ones presented in Johnstone (2001). Following Soshnikov (2002), we also show that the same is true for the joint distribution of the k largest eigenvalues, where k is a fixed integer. Numerical experiments illustrate the fact that the Tracy-Widom approximation is reasonable even when one of the dimension is "small".
The estimation of the covariance matrix is an initial step in many multivariate statistical methods such as principal components analysis and factor analysis, but in many practical applications the dimensionality of the sample space is large compared to the number of samples, and the usual maximum likelihood estimate is poor. Typically, improvements are obtained by modelling or regularization. From a practical point of view, these methods are often computationally heavy and rely on approximations. As a fast substitute, we propose an easily calculable maximum a posteriori (MAP) estimator based …
In this paper, we propose a class of Bayes estimators for the covariance matrix of graphical Gaussian models Markov with respect to a decomposable graph $G$. Working with the $W_{P_G}$ family defined by Letac and Massam [Ann. Statist. 35 (2007) 1278--1323] we derive closed-form expressions for Bayes estimators under the entropy and squared-error losses. The $W_{P_G}$ family includes the classical inverse of the hyper inverse Wishart but has many more shape parameters, thus allowing for flexibility in differentially shrinking various parts of the covariance matrix. Moreover, using this family a…
This paper develops on-line inference for the multivariate local level model, with the focus being placed on covariance estimation of the innovations. We assess the application of the inverse Wishart prior distribution in this context and find it too restrictive since the serial correlation structure of the observation and state innovations is forced to be the same. We generalize the inverse Wishart distribution to allow for a more convenient correlation structure, but still retaining approximate conjugacy. We prove some relevant results for the new distribution and we develop approximate Baye…
Gaussian covariance graph models encode marginal independence among the components of a multivariate random vector by means of a graph $G$. These models are distinctly different from the traditional concentration graph models (often also referred to as Gaussian graphical models or covariance selection models) since the zeros in the parameter are now reflected in the covariance matrix $Σ$, as compared to the concentration matrix $Ω=Σ^{-1}$. The parameter space of interest for covariance graph models is the cone $P_G$ of positive definite matrices with fixed zeros corresponding to the missing ed…
We study the distribution of the ratio of two central Wishart matrices with different covariance matrices. We first derive the density function of a particular matrix form of the ratio and show that its cumulative distribution function can be expressed in terms of the hypergeometric function 2F1 of a matrix argument. Then we apply the holonomic gradient method for numerical evaluation of the hypergeometric function. This approach enables us to compute the power function of Roy's maximum root test for testing the equality of two covariance matrices.
We derive concentration inequalities for the spectral measure of large random matrices, allowing for certain forms of dependence. Our main focus is on empirical covariance (Wishart) matrices, but general symmetric random matrices are also considered.
A holonomic system for the probability density function of the largest eigenvalue of a non-central complex Wishart distribution with identity covariance matrix is derived. Furthermore a new determinantal formula for the probability density function is derived (for m=2,3) or conjectured.
In this paper, we obtain a property of the expectation of the inverse of compound Wishart matrices which results from their orthogonal invariance. Using this property as well as results from random matrix theory (RMT), we derive the asymptotic effect of the noise induced by estimating the covariance matrix on computing the risk of the optimal portfolio. This in turn enables us to get an asymptotically unbiased estimator of the risk of the optimal portfolio not only for the case of independent observations but also in the case of correlated observations. This improvement provides a new approach…
For a random matrix following a Wishart distribution, we derive formulas for the expectation and the covariance matrix of compound matrices. The compound matrix of order $m$ is populated by all $m\times m$-minors of the Wishart matrix. Our results yield first and second moments of the minors of the sample covariance matrix for multivariate normal observations. This work is motivated by the fact that such minors arise in the expression of constraints on the covariance matrix in many classical multivariate problems.
We study the tail bound of the emperical covariance of multivariate normal distribution. Following the work of (Gittens & Tropp, 2011), we provide a tail bound with a small constant.
We consider two types of averaging of complex covariance matrices, a sample mean (average) and the sample Fréchet mean. We analyse the performance of these quantities as estimators for the true covariance matrix via `intrinsic' versions of bias and mean square error, a methodology which takes account of geometric structure. We derive simple expressions for the intrinsic bias in both cases, and the simple average is seen to be preferable. The same is true for the asymptotic Riemannian risk, and for the Riemannian risk itself in the scalar case. Combined with a similar preference for the simple …
Consider a standard white Wishart matrix with parameters $n$ and $p$. Motivated by applications in high-dimensional statistics and signal processing, we perform asymptotic analysis on the maxima and minima of the eigenvalues of all the $m \times m$ principal minors, under the asymptotic regime that $n,p,m$ go to infinity. Asymptotic results concerning extreme eigenvalues of principal minors of real Wigner matrices are also obtained. In addition, we discuss an application of the theoretical results to the construction of compressed sensing matrices, which provides insights to compressed sensing…
To explore the limits of a stochastic gradient method, it may be useful to consider an example consisting of an infinite number of quadratic functions. In this context, it is appropriate to determine the expected value and the covariance matrix of the stochastic noise, i.e. the difference of the true gradient and the approximated gradient generated from a finite sample. When specifying the covariance matrix, the expected value of a quadratic form QBQ is needed, where Q is a Wishart distributed random matrix and B is an arbitrary fixed symmetric matrix. After deriving an expression for E(QBQ) a…
We consider the properties of the bootstrap as a tool for inference concerning the eigenvalues of a sample covariance matrix computed from an $n\times p$ data matrix $X$. We focus on the modern framework where $p/n$ is not close to 0 but remains bounded as $n$ and $p$ tend to infinity. Through a mix of numerical and theoretical considerations, we show that the bootstrap is not in general a reliable inferential tool in the setting we consider. However, in the case where the population covariance matrix is well-approximated by a finite rank matrix, the bootstrap performs as it does in finite dim…
This study derives a new property of the Wishart distribution when the degree-of-freedom and the size of the matrix parameter of the distribution grow simultaneoulsy. Particularly, the asymptotic normality of the product of four independent Wishart matrices is shown under a high dimensional asymptotic regime. As an application of the result, a statistical test procedure for the common principal components hypothesis is proposed. For this problem, the proposed test statistic is asymptotically normal under the null hypothesis. In addition, the proposed test statistic diverges to positive infinit…
We consider settings where the observations are drawn from a zero-mean multivariate (real or complex) normal distribution with the population covariance matrix having eigenvalues of arbitrary multiplicity. We assume that the eigenvectors of the population covariance matrix are unknown and focus on inferential procedures that are based on the sample eigenvalues alone (i.e., "eigen-inference"). Results found in the literature establish the asymptotic normality of the fluctuation in the trace of powers of the sample covariance matrix. We develop concrete algorithms for analytically computing the …
The sum of independent Wishart matrices, taken from distributions with unequal covariance matrices, plays a crucial role in multivariate statistics, and has applications in the fields of quantitative finance and telecommunication. However, analytical results concerning the corresponding eigenvalue statistics have remained unavailable, even for the sum of two Wishart matrices. This can be attributed to the complicated and rotationally noninvariant nature of the matrix distribution that makes extracting the information about eigenvalues a nontrivial task. Using a generalization of the Harish-Cha…
The noncentral Wishart distribution has become more mainstream in statistics as the prevalence of applications involving sample covariances with underlying multivariate Gaussian populations as dramatically increased since the advent of computers. Multiple sources in the literature deal with local approximations of the noncentral Wishart distribution with respect to its central counterpart. However, no source has yet developed explicit local approximations for the (central) Wishart distribution in terms of a normal analogue, which is important since Gaussian distributions are at the heart of th…
We compute the spectral statistics of the sum H of two independent complex Wishart matrices, each of which is correlated with a different covariance matrix. Random matrix theory enjoys many applications including sums and products of random matrices. Typically ensembles with correlations among the matrix elements are much more difficult to solve. Using a combination of supersymmetry, superbosonisation and bi-orthogonal functions we are able to determine all spectral k-point density correlation functions of H for arbitrary matrix size N. In the half-degenerate case, when one of the covariance m…
In this paper we derive some new and practical results on testing and interval estimation problems for the population eigenvalues of a Wishart matrix based on the asymptotic theory for block-wise infinite dispersion of the population eigenvalues. This new type of asymptotic theory has been developed by the present authors in Takemura and Sheena (2005) and Sheena and Takemura (2007a,b) and in these papers it was applied to point estimation problem of population covariance matrix in a decision theoretic framework. In this paper we apply it to some testing and interval estimation problems. We sho…
Consider the classical Gaussian unitary ensemble of size $N$ and the real Wishart ensemble $W_N(n,I)$. In the limits as $N \to \infty$ and $N/n \to γ> 0$, the expected number of eigenvalues that exit the upper bulk edge is less than one, 0.031 and 0.170 respectively, the latter number being independent of $γ$. These statements are consequences of quantitative bounds on tail sums of eigenvalues outside the bulk which are established here for applications in high dimensional covariance matrix estimation.
The joint distribution of two off-diagonal Wishart matrix elements was useful in recent work on geometric probability [Finch 2010]. Not finding such formulas in the literature, we report these here.
Vinberg cones and the ambient vector spaces are important in modern statistics of sparse models and of graphical models. The aim of this paper is to study eigenvalue distributions of Gaussian, Wigner and covariance matrices related to growing Vinberg matrices, corresponding to growing daisy graphs. For Gaussian or Wigner ensembles, we give an explicit formula for the limiting distribution. For Wishart ensembles defined naturally on Vinberg cones, their limiting Stieltjes transforms, support and atom at 0 are described explicitly in terms of the Lambert-Tsallis functions, which are defined by u…
Let A be a p-variate real Wishart matrix on n degrees of freedom with identity covariance. The distribution of the largest eigenvalue in A has important applications in multivariate statistics. Consider the asymptotics when p grows in proportion to n, it is known from Johnstone (2001) that after centering and scaling, these distributions approach the orthogonal Tracy-Widom law for real-valued data, which can be numerically evaluated and tabulated in software. Under the same assumption, we show that more carefully chosen centering and scaling constants improve the accuracy of the distributional…
In this paper we consider some hypothesis tests within a family of Wishart distributions, where both the sample space and the parameter space are symmetric cones. For such testing problems, we first derive the joint density of the ordered eigenvalues of the generalized Wishart distribution and propose a test statistic analog to that of classical multivariate statistics for testing homoscedasticity of covariance matrix. In this generalization of Bartlett's test for equality of variances to hypotheses of real, complex, quaternion, Lorentz and octonion types of covariance structures.
Let $\mathbf{W}$ be a correlated complex non-central Wishart matrix defined through $\mathbf{W}=\mathbf{X}^H\mathbf{X}$, where $\mathbf{X}$ is $n\times m \, (n\geq m)$ complex Gaussian with non-zero mean $\boldsymbolΥ$ and non-trivial covariance $\boldsymbolΣ$. We derive exact expressions for the cumulative distribution functions (c.d.f.s) of the extreme eigenvalues (i.e., maximum and minimum) of $\mathbf{W}$ for some particular cases. These results are quite simple, involving rapidly converging infinite series, and apply for the practically important case where $\boldsymbolΥ$ has rank one. We…
This paper studies the problem of estimating the covariance of a collection of vectors using only highly compressed measurements of each vector. An estimator based on back-projections of these compressive samples is proposed and analyzed. A distribution-free analysis shows that by observing just a single linear measurement of each vector, one can consistently estimate the covariance matrix, in both infinity and spectral norm, and this same analysis leads to precise rates of convergence in both norms. Via information-theoretic techniques, lower bounds showing that this estimator is minimax-opti…
We derive a maximum a posteriori estimator for the linear observation model, where the signal and noise covariance matrices are both uncertain. The uncertainties are treated probabilistically by modeling the covariance matrices with prior inverse-Wishart distributions. The nonconvex problem of jointly estimating the signal of interest and the covariance matrices is tackled by a computationally efficient fixed-point iteration as well as an approximate variational Bayes solution. The statistical performance of estimators is compared numerically to state-of-the-art estimators from the literature …
The distributions of the largest and the smallest eigenvalues of a $p$-variate sample covariance matrix $S$ are of great importance in statistics. Focusing on the null case where $nS$ follows the standard Wishart distribution $W_p(I,n)$, we study the accuracy of their scaling limits under the setting: $n/p\rightarrow γ\in(0,\infty)$ as $n\rightarrow \infty$. The limits here are the orthogonal Tracy--Widom law and its reflection about the origin. With carefully chosen rescaling constants, the approximation to the rescaled largest eigenvalue distribution by the limit attains accuracy of order ${…
We consider the five classes of multivariate statistical problems identified by James (1964), which together cover much of classical multivariate analysis, plus a simpler limiting case, symmetric matrix denoising. Each of James' problems involves the eigenvalues of $E^{-1}H$ where $H$ and $E$ are proportional to high dimensional Wishart matrices. Under the null hypothesis, both Wisharts are central with identity covariance. Under the alternative, the non-centrality or the covariance parameter of $H$ has a single eigenvalue, a spike, that stands alone. When the spike is smaller than a case-spec…
We consider the problem of estimating the mean vector of a p-variate normal $(θ,Σ)$ distribution under invariant quadratic loss, $(δ-θ)'Σ^{-1}(δ-θ)$, when the covariance is unknown. We propose a new class of estimators that dominate the usual estimator $δ^0(X)=X$. The proposed estimators of $θ$ depend upon X and an independent Wishart matrix S with n degrees of freedom, however, S is singular almost surely when p>n. The proof of domination involves the development of some new unbiased estimators of risk for the p>n setting. We also find some relationships between the amount of domination and t…
We consider Bayesian inference of banded covariance matrices and propose a post-processed posterior. The post-processing of the posterior consists of two steps. In the first step, posterior samples are obtained from the conjugate inverse-Wishart posterior which does not satisfy any structural restrictions. In the second step, the posterior samples are transformed to satisfy the structural restriction through a post-processing function. The conceptually straightforward procedure of the post-processed posterior makes its computation efficient and can render interval estimators of functionals of …
Let $\mathbf{X}_p=(\mathbf{s}_1,...,\mathbf{s}_n)=(X_{ij})_{p \times n}$ where $X_{ij}$'s are independent and identically distributed (i.i.d.) random variables with $EX_{11}=0,EX_{11}^2=1$ and $EX_{11}^4<\infty$. It is showed that the largest eigenvalue of the random matrix $\mathbf{A}_p=\frac{1}{2\sqrt{np}}(\mathbf{X}_p\mathbf{X}_p^{\prime}-n\mathbf{I}_p)$ tends to 1 almost surely as $p\rightarrow\infty,n\rightarrow\infty$ with $p/n\rightarrow0$.
We establish a quantitative version of the Tracy--Widom law for the largest eigenvalue of high dimensional sample covariance matrices. To be precise, we show that the fluctuations of the largest eigenvalue of a sample covariance matrix $X^*X$ converge to its Tracy--Widom limit at a rate nearly $N^{-1/3}$, where $X$ is an $M \times N$ random matrix whose entries are independent real or complex random variables, assuming that both $M$ and $N$ tend to infinity at a constant rate. This result improves the previous estimate $N^{-2/9}$ obtained by Wang [73]. Our proof relies on a Green function comp…
In this paper we relate the matrix $S_B$ of the second moments of a spherically truncated normal multivariate to its full covariance matrix $Σ$ and present an algorithm to invert the relation and reconstruct $Σ$ from $S_B$. While the eigenvectors of $Σ$ are left invariant by the truncation, its eigenvalues are non-uniformly damped. We show that the eigenvalues of $Σ$ can be reconstructed from their truncated counterparts via a fixed point iteration, whose convergence we prove analytically. The procedure requires the computation of multidimensional Gaussian integrals over a Euclidean ball, for …
Let $X_1, \ldots, X_n$ be i.i.d. sample in $\mathbb{R}^p$ with zero mean and the covariance matrix $\mathbf{Σ^*}$. The classical PCA approach recovers the projector $\mathbf{P^*_{\mathcal{J}}}$ onto the principal eigenspace of $\mathbf{Σ^*}$ by its empirical counterpart $\mathbf{\widehat{P}_{\mathcal{J}}}$. Recent paper [Koltchinskii, Lounici (2017)] investigated the asymptotic distribution of the Frobenius distance between the projectors $\| \mathbf{\widehat{P}_{\mathcal{J}}} - \mathbf{P^*_{\mathcal{J}}} \|_2$, while [Naumov et al. (2017)] offered a bootstrap procedure to measure uncertainty …
In this paper, we establish some new central limit theorems for certain spectral statistics of a high-dimensional sample covariance matrix under a divergent spectral norm population model. This model covers the divergent spiked population model as a special case. Meanwhile, the number of the spiked eigenvalues can either be fixed or grow to infinity. It is seen from our theorems that the divergence of population spectral norm affects the fluctuations of the linear spectral statistics in a fickle way, depending on the divergence rate.
Let $\mathbf{X}\in\mathbb{C}^{n\times m}$ ($m\geq n$) be a random matrix with independent columns each distributed as complex multivariate Gaussian with zero mean and {\it single-spiked} covariance matrix $\mathbf{I}_n+ η\mathbf{u}\mathbf{u}^*$, where $\mathbf{I}_n$ is the $n\times n$ identity matrix, {\color{blue}$\mathbf{u}\in\mathbb{C}^{n\times 1}$} is an arbitrary vector with unit Euclidean norm, $η\geq 0$ is a non-random parameter, and $(\cdot)^*$ represents the conjugate-transpose. This paper investigates the distribution of the random quantity $κ_{\text{SC}}^2(\mathbf{X})=\sum_{k=1}^n λ…
We propose a new Conditional BEKK matrix-F (CBF) model for the time-varying realized covariance (RCOV) matrices. This CBF model is capable of capturing heavy-tailed RCOV, which is an important stylized fact but could not be handled adequately by the Wishart-based models. To further mimic the long memory feature of the RCOV, a special CBF model with the conditional heterogeneous autoregressive (HAR) structure is introduced. Moreover, we give a systematical study on the probabilistic properties and statistical inferences of the CBF model, including exploring its stationarity, establishing the as…
This paper is the second chapter of three of the author's undergraduate thesis. In this paper, we consider the random matrix ensemble given by $(d_b, d_w)$-regular graphs on $M$ black vertices and $N$ white vertices, where $d_b \in [N^γ, N^{2/3 - γ}]$ for any $γ> 0$. We simultaneously prove that the bulk eigenvalue correlation statistics for both normalized adjacency matrices and their corresponding covariance matrices are stable for short times. Combined with an ergodicity analysis of the Dyson Brownian motion in another paper, this proves universality of bulk eigenvalue correlation statistic…
Under the high-dimensional setting that data dimension and sample size tend to infinity proportionally, we derive the central limit theorem (CLT) for linear spectral statistics (LSS) of large-dimensional sample covariance matrix. Different from existing literature, our results do not require the assumption that the population covariance matrices are bounded. Moreover, many common kernel functions in the real data such as logarithmic functions and polynomial functions are allowed in this paper. In our model, the number of spiked eigenvalues can be fixed or tend to infinity. One salient feature …
We introduce a new random matrix model called distance covariance matrix in this paper, whose normalized trace is equivalent to the distance covariance. We first derive a deterministic limit for the eigenvalue distribution of the distance covariance matrix when the dimensions of the vectors and the sample size tend to infinity simultaneously. This limit is valid when the vectors are independent or weakly dependent through a finite-rank perturbation. It is also universal and independent of the details of the distributions of the vectors. Furthermore, the top eigenvalues of this distance covaria…
We study the $k$-largest eigenvalues of heavy-tailed sample covariance matrices of the form $\bX\bX^\T$ in an asymptotic framework, where the dimension of the data and the sample size tend to infinity. To this end, we assume that the rows of $\bX$ are given by independent copies of some stationary process with regularly varying marginals with index $α\in(0,2)$ satisfying large deviation and mixing conditions. We apply these general results to stochastic volatility and GARCH processes.
In this article we investigate high-dimensional banded sample covariance matrices under the regime that the sample size $n$, the dimension $p$ and the bandwidth $d$ tend simultaneously to infinity such that $$n/p\to 0 \ \ \text{and} \ \ 2d/n\to y>0.$$ It is shown that the empirical spectral distribution of those matrices almost surely converges weakly to some deterministic probability measure which is characterized by its moments. Certain restricted compositions of natural numbers play a crucial role in the evaluation of the expected moments of the empirical spectral distribution.
We consider the joint sparse estimation of regression coefficients and the covariance matrix for covariates in a high-dimensional regression model, where the predictors are both relevant to a response variable of interest and functionally related to one another via a Gaussian directed acyclic graph (DAG) model. Gaussian DAG models introduce sparsity in the Cholesky factor of the inverse covariance matrix, and the sparsity pattern in turn corresponds to specific conditional independence assumptions on the underlying predictors. A variety of methods have been developed in recent years for Bayesi…
We propose a Bayesian methodology for estimating spiked covariance matrices with jointly sparse structure in high dimensions. The spiked covariance matrix is reparametrized in terms of the latent factor model, where the loading matrix is equipped with a novel matrix spike-and-slab LASSO prior, which is a continuous shrinkage prior for modeling jointly sparse matrices. We establish the rate-optimal posterior contraction for the covariance matrix with respect to the operator norm as well as that for the principal subspace with respect to the projection operator norm loss. We also study the poste…
