Sphericity test plays a key role in many statistical problems. We propose Spearman's rho-type rank test and Kendall's tau-type rank test for sphericity in the high dimensional settings. We show that these two tests are equivalent. Thanks to the "blessing of dimension", we do not need to estimate any nuisance parameters. Without estimating the location parameter, we can allow the dimension to be arbitrary large. Asymptotic normality of these two tests are also established under elliptical distributions. Simulations demonstrate that they are very robust and efficient in a wide range of settings.
This paper focuses on the prominent sphericity test when the dimension $p$ is much lager than sample size $n$. The classical likelihood ratio test(LRT) is no longer applicable when $p\gg n$. Therefore a Quasi-LRT is proposed and asymptotic distribution of the test statistic under the null when $p/n\rightarrow\infty, n\rightarrow\infty$ is well established in this paper. Meanwhile, John's test has been found to possess the powerful {\it dimension-proof} property, which keeps exactly the same limiting distribution under the null with any $(n,p)$-asymptotic, i.e. $p/n\rightarrow[0,\infty]$, $n\ri…
In this paper, we propose corrections to the likelihood ratio test and John's test for sphericity in large-dimensions. New formulas for the limiting parameters in the CLT for linear spectral statistics of sample covariance matrices with general fourth moments are first established. Using these formulas, we derive the asymptotic distribution of the two proposed test statistics under the null. These asymptotics are valid for general population, i.e. not necessarily Gaussian, provided a finite fourth-moment. Extensive Monte-Carlo experiments are conducted to assess the quality of these tests with…
This paper studies the asymptotic power of tests of sphericity against perturbations in a single unknown direction as both the dimensionality of the data and the number of observations go to infinity. We establish the convergence, under the null hypothesis and contiguous alternatives, of the log ratio of the joint densities of the sample covariance eigenvalues to a Gaussian process indexed by the norm of the perturbation. When the perturbation norm is larger than the phase transition threshold studied in Baik, Ben Arous and Peche [Ann. Probab. 33 (2005) 1643-1697] the limiting process is degen…
In this paper we study the existence of locally most powerful invariant tests (LMPIT) for the problem of testing the covariance structure of a set of Gaussian random vectors. The LMPIT is the optimal test for the case of close hypotheses, among those satisfying the invariances of the problem, and in practical scenarios can provide better performance than the typically used generalized likelihood ratio test (GLRT). The derivation of the LMPIT usually requires one to find the maximal invariant statistic for the detection problem and then derive its distribution under both hypotheses, which in ge…
This paper considers testing the covariance matrices structure based on Wald's score test in large dimensional setting. The hypothesis $H_0: Σ=Σ_0 $ for a given matrix $Σ_0$, which covers the identity hypothesis test and sphericity hypothesis test as the special cases, is reviewed by the generalized CLT (Central Limit Theorem) for the linear spectral statistics of large dimensional sample covariance matrices from Jiang(2015) . The proposed tests can be applicable for large dimensional non-Gaussian variables in a wider range. Furthermore, the simulation study is provided to compare the proposed…
Based on a generalized cosine measure between two symmetric matrices, we propose a general framework for one-sample and two-sample tests of covariance and correlation matrices. We also develop a set of associated permutation algorithms for some common one-sample tests, such as the tests of sphericity, identity and compound symmetry, and the $K$-sample tests of multivariate equality of covariance or correlation matrices. The proposed method is very flexible in the sense that it does not assume any underlying distributions and data generation models. Moreover, it allows data to have different ma…
We develop tests for high-dimensional covariance matrices under a generalized elliptical model. Our tests are based on a central limit theorem (CLT) for linear spectral statistics of the sample covariance matrix based on self-normalized observations. For testing sphericity, our tests neither assume specific parametric distributions nor involve the kurtosis of data. More generally, we can test against any non-negative definite matrix that can even be not invertible. As an interesting application, we illustrate in empirical studies that our tests can be used to test uncorrelatedness among idiosy…
Nonparametric tests provide robust and powerful alternatives to the corresponding least squares methods. There are two approaches to nonparametric pairwise comparisons of treatment effects, the method based on pairwise rankings and the method based on overall ranking. The former is generally recommended in the literature because of its strong control of familywise error rate. However, this method is developed only for one-way layouts and randomized complete blocks. By combining the method of aligned ranks and pairwise ranking, we extend the Steel-Dwass pairwise comparisons to the analysis of c…
This paper discusses fluctuations of linear spectral statistics of high-dimensional sample covariance matrices when the underlying population follows an elliptical distribution. Such population often possesses high order correlations among their coordinates, which have great impact on the asymptotic behaviors of linear spectral statistics. Taking such kind of dependency into consideration, we establish a new central limit theorem for the linear spectral statistics in this paper for a class of elliptical populations. This general theoretical result has wide applications and, as an example, it i…
This article concerns tests for the two-sample location problem when the dimension is larger than the sample size. The traditional multivariate-rank-based procedures cannot be used in high dimensional settings because the sample scatter matrix is not available. We propose a novel high-dimensional spatial rank test in this article. The asymptotic normality is established. We can allow the dimension being almost the exponential rate of the sample sizes. Simulations demonstrate that it is very robust and efficient in a wide range of distributions.
We consider the problem of testing the equality of conditional distributions of a response variable given a vector of covariates between two populations. Such a hypothesis testing problem can be motivated from various machine learning and statistical inference scenarios, including transfer learning and causal predictive inference. We develop a nonparametric test procedure inspired from the conformal prediction framework. The construction of our test statistic combines recent developments in conformal prediction with a novel choice of conformity score, resulting in a weighted rank-sum test stat…
Ranking institutions such as medical centers or universities is based on an indicator accompanied with an uncertainty measure such as a standard deviation, and confidence intervals should be calculated to assess the quality of these ranks. We consider the problem of constructing simultaneous confidence intervals for the ranks of centers based on an observed sample. We present in this paper a novel method based on multiple testing which uses the partitioning principle and employs the likelihood ratio (LR) test on the partitions. The complexity of the algorithm is super exponential. We present s…
Joint modeling of a large number of variables often requires dimension reduction strategies that lead to structural assumptions of the underlying correlation matrix, such as equal pair-wise correlations within subsets of variables. The underlying correlation matrix is thus of interest for both model specification and model validation. In this paper, we develop tests of the hypothesis that the entries of the Kendall rank correlation matrix are linear combinations of a smaller number of parameters. The asymptotic behavior of the proposed test statistics is investigated both when the dimension is…
We consider the problem of testing uniformity on high-dimensional unit spheres. We are primarily interested in non-null issues. We show that rotationally symmetric alternatives lead to two Local Asymptotic Normality (LAN) structures. The first one is for fixed modal location $θ$ and allows to derive locally asymptotically most powerful tests under specified $θ$. The second one, that addresses the Fisher-von Mises-Langevin (FvML) case, relates to the unspecified-$θ$ problem and shows that the high-dimensional Rayleigh test is locally asymptotically most powerful invariant. Under mild assumption…
Conformal prediction has been a very popular method of distribution-free predictive inference in recent years in machine learning and statistics. Its popularity stems from the fact that it works as a wrapper around any prediction algorithm such as neural networks or random forests. Exchangeability is at the core of the validity of conformal prediction. The concept of exchangeability is also at the core of rank tests widely known in nonparametric statistics. In this paper, we review the concept of exchangeability and discuss the implications for conformal prediction and rank tests. We provide a…
This paper deals with the local asymptotic structure, in the sense of Le Cam's asymptotic theory of statistical experiments, of the signal detection problem in high dimension. More precisely, we consider the problem of testing the null hypothesis of sphericity of a high-dimensional covariance matrix against an alternative of (unspecified) multiple symmetry-breaking directions (\textit{multispiked} alternatives). Simple analytical expressions for the asymptotic power envelope and the asymptotic powers of previously proposed tests are derived. These asymptotic powers are shown to lie very substa…
Several statistics-based detectors, based on unimodal matrix models, for determining the number of sources in a field are designed. A new variance ratio statistic is proposed, and its asymptotic distribution is analyzed. The variance ratio detector is shown to outperform the alternatives. It is shown that further improvements are achievable via optimally selected rotations. Numerical experiments demonstrate the performance gains of our detection methods over the baseline approach.
A rank-based test of the null hypothesis that a regressor has no effect on a response variable is proposed and analyzed. This test is identical in structure to the order selection test but with the raw data replaced by ranks. The test is nonparametric in that it is consistent against virtually any smooth alternative, and is completely distribution free for all sample sizes. The asymptotic distribution of the rank-based order selection statistic is obtained and seen to be the same as that of its raw data counterpart. Exact small sample critical values of the test statistic are provided as well.…
In this paper, we extend the recently proposed multivariate rank energy distance, based on the theory of optimal transport, for statistical testing of distributional similarity, to soft rank energy distance. Being differentiable, this in turn allows us to extend the rank energy to a subspace robust rank energy distance, dubbed Projected soft-Rank Energy distance, which can be computed via optimization over the Stiefel manifold. We show via experiments that using projected soft rank energy one can trade-off the detection power vs the false alarm via projections onto an appropriately selected lo…
This paper investigates limiting properties of eigenvalues of multivariate sample spatial-sign covariance matrices when both the number of variables and the sample size grow to infinity. The underlying p-variate populations are general enough to include the popular independent components model and the family of elliptical distributions. A first result of the paper establishes that the distribution of the eigenvalues converges to a deterministic limit that belongs to the family of generalized Marcenko-Pastur distributions. Furthermore, a new central limit theorem is established for a class of l…
Dept. of ECE and CSL, UIUC（伊利诺伊大学厄巴纳-香槟分校电气与计算机工程系及协调科学实验室）
This paper concerns the construction of tests for universal hypothesis testing problems, in which the alternate hypothesis is poorly modeled and the observation space is large. The mismatched universal test is a feature-based technique for this purpose. In prior work it is shown that its finite-observation performance can be much better than the (optimal) Hoeffding test, and good performance depends crucially on the choice of features. The contributions of this paper include: 1) We obtain bounds on the number of εdistinguishable distributions in an exponential family. 2) This motivates a new f…
Extending to dimension 2 and higher the dual univariate concepts of ranks and quantiles has remained an open problem for more than half a century. Based on measure transportation results, a solution has been proposed recently under the name center-outward ranks and quantiles which, contrary to previous proposals, enjoys all the properties that make univariate ranks a successful tool for statistical inference. Just as their univariate counterparts (to which they reduce in dimension one), center-outward ranks allow for the construction of distribution-free and asymptotically efficient tests for …
We treat the problem of testing independence between m continuous variables when m can be larger than the available sample size n. We consider three types of test statistics that are constructed as sums or sums of squares of pairwise rank correlations. In the asymptotic regime where both m and n tend to infinity, a martingale central limit theorem is applied to show that the null distributions of these statistics converge to Gaussian limits, which are valid with no specific distributional or moment assumptions on the data. Using the framework of U-statistics, our result covers a variety of ran…
This paper investigates the problem of testing independence of two random vectors of general dimensions. For this, we give for the first time a distribution-free consistent test. Our approach combines distance covariance with the center-outward ranks and signs developed in Hallin (2017). In technical terms, the proposed test is consistent and distribution-free in the family of multivariate distributions with nonvanishing (Lebesgue) probability densities. Exploiting the (degenerate) U-statistic structure of the distance covariance and the combinatorial nature of Hallin's center-outward ranks an…
This paper provides parametric and rank-based optimal tests for eigenvectors and eigenvalues of covariance or scatter matrices in elliptical families. The parametric tests extend the Gaussian likelihood ratio tests of Anderson (1963) and their pseudo-Gaussian robustifications by Davis (1977) and Tyler (1981, 1983). The rank-based tests address a much broader class of problems, where covariance matrices need not exist and principal components are associated with more general scatter matrices. The proposed tests are shown to outperform daily practice both from the point of view of validity as fr…
In this paper, we use and further develop upon a recently proposed multivariate, distribution-free Goodness-of-Fit (GoF) test based on the theory of Optimal Transport (OT) called the Rank Energy (RE) [1], for non-parametric and unsupervised Change Point Detection (CPD) in multivariate time series data. We show that directly using RE leads to high sensitivity to very small changes in distributions (causing high false alarms) and it requires large sample complexity and huge computational cost. To alleviate these drawbacks, we propose a new GoF test statistic called as soft-Rank Energy (sRE) that…
We propose a class of locally and asymptotically optimal tests, based on multivariate ranks and signs for the homogeneity of scatter matrices in $m$ elliptical populations. Contrary to the existing parametric procedures, these tests remain valid without any moment assumptions, and thus are perfectly robust against heavy-tailed distributions (validity robustness). Nevertheless, they reach semiparametric efficiency bounds at correctly specified elliptical densities and maintain high powers under all (efficiency robustness). In particular, their normal-score version outperforms traditional Gaussi…
Matrix-covariate is now frequently encountered in many biomedical researches. It is common to fit conventional statistical models by vectorizing matrix-covariate. This strategy, however, results in a large number of parameters, while the available sample size is relatively too small to have reliable analysis results. To overcome the problem of high-dimensionality in hypothesis testing, variance component test has been proposed with promise detection power, but is not straightforward to provide estimates of effect size. In this work, we overcome the problem of high-dimensionality by utilizing t…
Within the nonparametric regression model with unknown regression function $l$ and independent, symmetric errors, a new multiscale signed rank statistic is introduced and a conditional multiple test of the simple hypothesis $l=0$ against a nonparametric alternative is proposed. This test is distribution-free and exact for finite samples even in the heteroscedastic case. It adapts in a certain sense to the unknown smoothness of the regression function under the alternative, and it is uniformly consistent against alternatives whose sup-norm tends to zero at the fastest possible rate. The test is…
The problem of testing the reliability of ensemble forecasting systems is revisited. A popular tool to assess the reliability of ensemble forecasting systems (for scalar verifications) is the rank histogram, this histogram is expected to be more or less flat, since for a reliable ensemble, the ranks are uniformly distributed among their possible outcomes. Quantitative tests for flatness (e.g.\ Pearson's goodness--of--fit test) have been suggested, without exception though, these tests assume the ranks to be a sequence of independent random variables, which is not the case in general as can be …
Statistical Inference for Rank Allocation in Low-Rank Adaptation
Department of Mathematics, National University of Singapore（数学系，新加坡国立大学）；Department of Mathematics and Department of Electrical and Computer Engineering, National University of Singapore（数学系和电气与计算机工程系，新加坡国立大学）
We develop a class of tests for semiparametric vector autoregressive (VAR) models with unspecified innovation densities, based on the recent measure-transportation-based concepts of multivariate {\it center-outward ranks} and {\it signs}. We show that these concepts, combined with Le Cam's asymptotic theory of statistical experiments, yield novel testing procedures, which (a)~are valid under a broad class of innovation densities (possibly non-elliptical, skewed, and/or with infinite moments), (b)~are optimal (locally asymptotically maximin or most stringent) at selected ones, and (c) are robus…
By studying the family of $p$-dimensional scale mixtures, this paper shows for the first time a non trivial example where the eigenvalue distribution of the corresponding sample covariance matrix {\em does not converge} to the celebrated Marčenko-Pastur law. A different and new limit is found and characterized. The reasons of failure of the Marčenko-Pastur limit in this situation are found to be a strong dependence between the $p$-coordinates of the mixture. Next, we address the problem of testing whether the mixture has a spherical covariance matrix. To analize the traditional John's type tes…
Bayesian inference for rank-order problems is frustrated by the absence of an explicit likelihood function. This hurdle can be overcome by assuming a latent normal representation that is consistent with the ordinal information in the data: the observed ranks are conceptualized as an impoverished reflection of an underlying continuous scale, and inference concerns the parameters that govern the latent representation. We apply this generic data-augmentation method to obtain Bayes factors for three popular rank-based tests: the rank sum test, the signed rank test, and Spearman's $ρ_s$.
In this paper, we propose a power comparison between high dimensional t-test, sign and signed rank test for the one sample mean test. We show that the high dimensional signed rank test is superior to a high dimensional t test, but inferior to a high dimensional sign test.
We study a rank based univariate two-sample distribution-free test. The test statistic is the difference between the average of between-group rank distances and the average of within-group rank distances. This test statistic is closely related to the two-sample Cramér-von Mises criterion. They are different empirical versions of a same quantity for testing the equality of two population distributions. Although they may be different for finite samples, they share the same expected value, variance and asymptotic properties. The advantage of the new rank based test over the classical one is its e…
Testing Covariance Separability in High Dimensions
Institute for Statistics and Mathematics, WU Wien（维也纳经济大学统计与数学研究所）；Institute of Statistics and Mathematical Methods in Economics, TU Wien（维也纳工业大学经济与统计及数学方法研究所）
