This paper is concerned with the limiting spectral behaviors of large dimensional Kendall's rank correlation matrices generated by samples with independent and continuous components. We do not require the components to be identically distributed, and do not need any moment conditions, which is very different from the assumptions imposed in the literature of random matrix theory. The statistical setting in this paper covers a wide range of highly skewed and heavy-tailed distributions. We establish the central limit theorem (CLT) for the linear spectral statistics of the Kendall's rank correlation matrices under the Marchenko-Pastur asymptotic regime, in which the dimension diverges to infinity proportionally with the sample size. We further propose three nonparametric procedures for high dimensional independent test and their limiting null distributions are derived by implementing this CLT. Our numerical comparisons demonstrate the robustness and superiority of our proposed test statistics under various mixed and heavy-tailed cases.
In a sparse high-dimensional elliptical model we consider a hard threshold estimator for the correlation matrix based on Kendall's tau with threshold level $α(\frac{\log p}{n})^{1/2}$. Parameters $α$ are identified such that the threshold estimator achieves the minimax rate under the squared Frobenius norm and the squared spectral norm. This allows a reasonable calibration of the estimator without any quantitative information about the tails of the underlying distribution. For Gaussian observations we even establish a critical threshold constant $α^\ast$ under the squared Frobenius norm, i.e. …
In the present paper, we first discuss the Kendall rank correlation coefficient. In continuous case, we define the Kendall rank correlation coefficient in terms of the concomitants of order statistics, find the expected value of the Kendall rank correlation coefficient and show that the later is free of n. We also prove that in continuous case the Kendall correlation coefficient converges in probability to its expected value. We then propose to consider the expected value of the Kendall rank correlation coefficient as a new theoretical correlation coefficient which can be an alternative to the…
This paper discusses the statistical inference problem associated with testing for dependence between two continuous random variables using Kendall's $τ$ in the context of the missing data problem. We prove the worst-case identified set for this measure of association always includes zero. The consequence of this result is that robust inference for dependence using Kendall's $τ$, where robustness is with respect to the form of the missingness-generating process, is impossible.
For a bivariate time series $((X_i,Y_i))_{i=1,...,n}$ we want to detect whether the correlation between $X_i$ and $Y_i$ stays constant for all $i = 1,...,n$. We propose a nonparametric change-point test statistic based on Kendall's tau and derive its asymptotic distribution under the null hypothesis of no change by means a new U-statistic invariance principle for dependent processes. The asymptotic distribution depends on the long run variance of Kendall's tau, for which we propose an estimator and show its consistency. Furthermore, assuming a single change-point, we show that the location of …
Methods are developed for checking and completing systems of bivariate and multivariate Kendall's tau concordance measures in applications where only partial information about dependencies between variables is available. The concept of a concordance signature of a multivariate continuous distribution is introduced; this is the vector of concordance probabilities for margins of all orders. It is shown that every attainable concordance signature is equal to the concordance signature of a unique mixture of the extremal copulas, that is the copulas with extremal correlation matrices consisting exc…
This paper investigates limiting spectral distribution of a high-dimensional Kendall's rank correlation matrix. The underlying population is allowed to have general dependence structure. The result no longer follows the generalized Marucenko-Pastur law, which is brand new. It's the first result on rank correlation matrices with dependence. As applications, we study the Kendall's rank correlation matrix for multivariate normal distributions with a general covariance matrix. From these results, we further gain insights on Kendall's rank correlation matrix and its connections with the sample cova…
Joint modeling of a large number of variables often requires dimension reduction strategies that lead to structural assumptions of the underlying correlation matrix, such as equal pair-wise correlations within subsets of variables. The underlying correlation matrix is thus of interest for both model specification and model validation. In this paper, we develop tests of the hypothesis that the entries of the Kendall rank correlation matrix are linear combinations of a smaller number of parameters. The asymptotic behavior of the proposed test statistics is investigated both when the dimension is…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Heidelberg Institute for Theoretical Studies（海德堡理论研究所）；Goethe University Frankfurt（法兰克福歌德大学）；Helmut-Schmidt-University（赫尔穆特·施密特大学）
A necessary and suffcient condition for Pitman's asymptotic relative effciency (ARE) of the Kendall and Spearman correlation statistics for the independence test to be 1 is given, in terms of certain smoothness and nondegeneracy properties of the model. Corresponding easy to use and broadly applicable sufficient conditions are obtained, which are then illustrated on several known models of dependence. Effects of the presence or absence of the smoothness and/or nondegeneracy parts of the mentioned necessary and suffcient condition are demonstrated using certain specially constructed dependence …
In this paper, we study a high-dimensional random matrix model from nonparametric statistics called the Kendall rank correlation matrix, which is a natural multivariate extension of the Kendall rank correlation coefficient. We establish the Tracy-Widom law for its largest eigenvalue. It is the first Tracy-Widom law for a nonparametric random matrix model, and also the first Tracy-Widom law for a high-dimensional U-statistic.
We treat the problem of testing independence between m continuous variables when m can be larger than the available sample size n. We consider three types of test statistics that are constructed as sums or sums of squares of pairwise rank correlations. In the asymptotic regime where both m and n tend to infinity, a martingale central limit theorem is applied to show that the null distributions of these statistics converge to Gaussian limits, which are valid with no specific distributional or moment assumptions on the data. Using the framework of U-statistics, our result covers a variety of ran…
The goal of this study was to test the equality of two covariance matrices by using modified Pillai's trace statistics under a high-dimensional framework, i.e., the dimension and sample sizes go to infinity proportionally. In this paper, we introduce two modified Pillai's trace statistics and obtain their asymptotic distributions under the null hypothesis. The benefits of the proposed statistics include the following: (1) the sample size can be smaller than the dimensions; (2) the limiting distributions of the proposed statistics are universal; and (3) we do not restrict the structure of the p…
The Riemannian geometry of covariance matrices has been essential to several successful applications, in computer vision, biomedical signal and image processing, and radar data processing. For these applications, an important ongoing challenge is to develop Riemannian-geometric tools which are adapted to structured covariance matrices. The present paper proposes to meet this challenge by introducing a new class of probability distributions, Gaussian distributions of structured covariance matrices. These are Riemannian analogs of Gaussian distributions, which only sample from covariance matrice…
We consider general high-dimensional spiked sample covariance models and show that their leading sample spiked eigenvalues and their linear spectral statistics are asymptotically independent when the sample size and dimension are proportional to each other. As a byproduct, we also establish the central limit theorem of the leading sample spiked eigenvalues by removing the block diagonal assumption on the population covariance matrix, which is commonly needed in the literature. Moreover, we propose consistent estimators of the $L_4$ norm of the spiked population eigenvectors. Based on these res…
In this paper, the fused graphical lasso (FGL) method is used to estimate multiple precision matrices from multiple populations simultaneously. The lasso penalty in the FGL model is a restraint on sparsity of precision matrices, and a moderate penalty on the two precision matrices from distinct groups restrains the similar structure across multiple groups. In high-dimensional settings, an oracle inequality is provided for FGL estimators, which is necessary to establish the central limit law. We not only focus on point estimation of a precision matrix, but also work on hypothesis testing for a …
We develop adaptive estimation and inference methods for high-dimensional Gaussian copula regression that achieve the same performance without the knowledge of the marginal transformations as that for high-dimensional linear regression. Using a Kendall's tau based covariance matrix estimator, an $\ell_1$ regularized estimator is proposed and a corresponding de-biased estimator is developed for the construction of the confidence intervals and hypothesis tests. Theoretical properties of the procedures are studied and the proposed estimation and inference methods are shown to be adaptive to the u…
Let $\mathbf{X}_n=(x_{ij})$ be a $k \times n$ data matrix with complex-valued, independent and standardized entries satisfying a Lindeberg-type moment condition. We consider simultaneously $R$ sample covariance matrices $\mathbf{B}_{nr}=\frac1n \mathbf{Q}_r \mathbf{X}_n \mathbf{X}_n^*\mathbf{Q}_r^\top,~1\le r\le R$, where the $\mathbf{Q}_{r}$'s are nonrandom real matrices with common dimensions $p\times k~(k\geq p)$. Assuming that both the dimension $p$ and the sample size $n$ grow to infinity, the limiting distributions of the eigenvalues of the matrices $\{\mathbf{B}_{nr}\}$ are identified, …
Standard Gini covariance and Gini correlation play important roles in measuring the dependence of random variables with heavy tails. However, the asymmetry brings a substantial difficulty in interpretation. In this paper, we propose a symmetric Gini-type covariance and a symmetric Gini correlation ($ρ_g$) based on the joint rank function. The proposed correlation $ρ_g$ is more robust than the Pearson correlation but less robust than the Kendall's $τ$ correlation. We establish the relationship between $ρ_g$ and the linear correlation $ρ$ for a class of random vectors in the family of elliptical…
Maximum pseudo-likelihood (MPL) is a semiparametric estimation method often used to obtain the dependence parameters in copula models from data. It has been shown that despite being consistent, and in some cases efficient, MPL estimation can overestimate the level of dependence especially for small weakly dependent samples. We show that the MPL method uses the expected value of order statistics and we propose to use instead the median or the mode of the same order statistics. In a simulation study we compare the finite-sample performance of the proposed estimators with that of the original MPL…
Sphericity test plays a key role in many statistical problems. We propose Spearman's rho-type rank test and Kendall's tau-type rank test for sphericity in the high dimensional settings. We show that these two tests are equivalent. Thanks to the "blessing of dimension", we do not need to estimate any nuisance parameters. Without estimating the location parameter, we can allow the dimension to be arbitrary large. Asymptotic normality of these two tests are also established under elliptical distributions. Simulations demonstrate that they are very robust and efficient in a wide range of settings.
We prove that Kendall's Rank correlation matrix converges to the Marčenko-Pastur law, under the assumption that the observations are i.i.d random vectors $X_1$, $\dots$, $X_n$ with components that are independent and absolutely continuous with respect to the Lebesgue measure. This is the first result on the empirical spectral distribution of a multivariate $U$-statistic.
In this paper, we study the empirical spectral distribution of Spearman's rank correlation matrices, under the assumption that the observations are independent and identically distributed random vectors and the features are correlated. We show that the limiting spectral distribution is the generalized Marucenko-Pastur law with the covariance matrix of the observation after standardized transformation. With these results, we compare several classical covariance/correlation matrices including the sample covariance matrix, Pearson's correlation matrix, Kendall's correlation matrix and Spearman's …
Department of Mathematics, University of Louisiana at Lafayette（路易斯安那州拉法叶大学数学系）；Department of Statistics, formerly of Presidency College（统计系，原为 Presidency College）
In this paper we propose methodology for inference of binary-valued adjacency matrices from various measures of the strength of association between pairs of network nodes, or more generally pairs of variables. This strength of association can be quantified by sample covariance and correlation matrices, and more generally by test-statistics and hypothesis test p-values from arbitrary distributions. Community detection methods such as block modelling typically require binary-valued adjacency matrices as a starting point. Hence, a main motivation for the methodology we propose is to obtain binary…
There has been an increasing interest in testing the equality of large Pearson's correlation matrices. However, in many applications it is more important to test the equality of large rank-based correlation matrices since they are more robust to outliers and nonlinearity. Unlike the Pearson's case, testing the equality of large rank-based statistics has not been well explored and requires us to develop new methods and theory. In this paper, we provide a framework for testing the equality of two large U-statistic based correlation matrices, which include the rank-based correlation matrices as s…
We consider the testing of mutual independence among all entries in a $d$-dimensional random vector based on $n$ independent observations. We study two families of distribution-free test statistics, which include Kendall's tau and Spearman's rho as important examples. We show that under the null hypothesis the test statistics of these two families converge weakly to Gumbel distributions, and propose tests that control the type I error in the high-dimensional setting where $d>n$. We further show that the two tests are rate-optimal in terms of power against sparse alternatives, and outperform co…
Low-rank matrices are pervasive throughout statistics, machine learning, signal processing, optimization, and applied mathematics. In this paper, we propose a novel and user-friendly Euclidean representation framework for low-rank matrices. Correspondingly, we establish a collection of technical and theoretical tools for analyzing the intrinsic perturbation of low-rank matrices in which the underlying referential matrix and the perturbed matrix both live on the same low-rank matrix manifold. Our analyses show that, locally around the referential matrix, the sine-theta distance between subspace…
This paper is the third chapter of three of the author's undergraduate thesis. In this paper, we study the convergence of local bulk statistics for linearized covariance matrices under Dyson's Brownian motion. We consider deterministic initial data $V$ approximate the Dyson Brownian motion for linearized covariance matrices by the Wigner flow. Using universality results for the Wigner flow, we deduce universality for the linearized covariance matrices. We deduce bulk universality of averaged bulk correlation functions for both biregular bipartite graphs and honest covariance matrices. We also …
Testing large covariance matrices is of fundamental importance in statistical analysis with high-dimensional data. In the past decade, three types of test statistics have been studied in the literature: quadratic form statistics, maximum form statistics, and their weighted combination. It is known that quadratic form statistics would suffer from low power against sparse alternatives and maximum form statistics would suffer from low power against dense alternatives. The weighted combination methods were introduced to enhance the power of quadratic form statistics or maximum form statistics when…
Let $F_N$ and $F$ be the empirical and limiting spectral distributions of an $N\times N$ Wigner matrix. The Cramér-von Mises (CvM) statistic is a classical goodness-of-fit statistic that characterizes the distance between $F_N$ and $F$ in $\ell^2$-norm. In this paper, we consider a mesoscopic approximation of the CvM statistic for Wigner matrices, and derive its limiting distribution. In the appendix, we also give the limiting distribution of the CvM statistic (without approximation) for the toy model CUE.
The sum of independent Wishart matrices, taken from distributions with unequal covariance matrices, plays a crucial role in multivariate statistics, and has applications in the fields of quantitative finance and telecommunication. However, analytical results concerning the corresponding eigenvalue statistics have remained unavailable, even for the sum of two Wishart matrices. This can be attributed to the complicated and rotationally noninvariant nature of the matrix distribution that makes extracting the information about eigenvalues a nontrivial task. Using a generalization of the Harish-Cha…
