Multivariate extreme value theory is concerned with modeling the joint tail behavior of several random variables. Existing work mostly focuses on asymptotic dependence, where the probability of observing a large value in one of the variables is of the same order as observing a large value in all variables simultaneously. However, there is growing evidence that asymptotic independence is equally important in real world applications. Available statistical methodology in the latter setting is scarce and not well understood theoretically. We revisit non-parametric estimation and introduce rank-based M-estimators for parametric models that simultaneously work under asymptotic dependence and asymptotic independence, without requiring prior knowledge on which of the two regimes applies. Asymptotic normality of the proposed estimators is established under weak regularity conditions. We further show how bivariate estimators can be leveraged to obtain parametric estimators in spatial tail models, and again provide a thorough theoretical justification for our approach.
This paper investigates the problem of testing independence of two random vectors of general dimensions. For this, we give for the first time a distribution-free consistent test. Our approach combines distance covariance with the center-outward ranks and signs developed in Hallin (2017). In technical terms, the proposed test is consistent and distribution-free in the family of multivariate distributions with nonvanishing (Lebesgue) probability densities. Exploiting the (degenerate) U-statistic structure of the distance covariance and the combinatorial nature of Hallin's center-outward ranks an…
We treat the problem of testing independence between m continuous variables when m can be larger than the available sample size n. We consider three types of test statistics that are constructed as sums or sums of squares of pairwise rank correlations. In the asymptotic regime where both m and n tend to infinity, a martingale central limit theorem is applied to show that the null distributions of these statistics converge to Gaussian limits, which are valid with no specific distributional or moment assumptions on the data. Using the framework of U-statistics, our result covers a variety of ran…
We congratulate Engelke and Hitz on a thought-provoking paper on graphical models for extremes. A key contribution of the paper is the introduction of a novel definition of conditional independence for a multivariate Pareto distribution. Here, we outline a proposal for independence and conditional independence of general random variables whose support is a general set Omega in multidimensional real number space. Our proposal includes the authors' definition of conditional independence, and the analogous definition of independence as special cases. By making our proposal independent of the cont…
Ultrahigh-dimensional variable selection plays an increasingly important role in contemporary scientific discoveries and statistical research. Among others, Fan and Lv [J. R. Stat. Soc. Ser. B Stat. Methodol. 70 (2008) 849-911] propose an independent screening framework by ranking the marginal correlations. They showed that the correlation ranking procedure possesses a sure independence screening property within the context of the linear model with Gaussian covariates and responses. In this paper, we propose a more general version of the independent learning with ranking the maximum marginal l…
It is well known that an extreme order statistic and a central order statistic (os) as well as an intermediate os and a central os from a sample of iid univariate random variables get asymptotically independent as the sample size increases. We extend this result to bivariate random variables, where the os are taken componentwise. An explicit representation of the conditional distribution of bivariate os turns out to be a powerful tool.
Independence screening is a variable selection method that uses a ranking criterion to select significant variables, particularly for statistical models with nonpolynomial dimensionality or "large p, small n" paradigms when p can be as large as an exponential of the sample size n. In this paper we propose a robust rank correlation screening (RRCS) method to deal with ultra-high dimensional data. The new procedure is based on the Kendall τcorrelation coefficient between response and predictor variables rather than the Pearson correlation of existing methods. The new method has four desirable fe…
The maximum likelihood degree (ML degree) measures the algebraic complexity of a fundamental optimization problem in statistics: maximum likelihood estimation. In this problem, one maximizes the likelihood function over a statistical model. The ML degree of a model is an upper bound to the number of local extrema of the likelihood function and can be expressed as a weighted sum of Euler characteristics. The independence model (i.e. rank one matrices over the probability simplex) is well known to have an ML degree of one, meaning their is a unique local maxima of the likelihood function. Howeve…
Conditional independence, graphical models and sparsity are key notions for parsimonious statistical models and for understanding the structural relationships in the data. The theory of multivariate and spatial extremes describes the risk of rare events through asymptotically justified limit models such as max-stable and multivariate Pareto distributions. Statistical modelling in this field has been limited to moderate dimensions so far, partly owing to complicated likelihoods and a lack of understanding of the underlying probabilistic structures. We introduce a general theory of conditional i…
One of the most popular class of tests for independence between two random variables is the general class of rank statistics which are invariant under permutations. This class contains Spearman's coefficient of rank correlation statistic, Fisher-Yates statistic, weighted Mann statistic and others. Under the null hypothesis of independence these test statistics have a permutation distribution that usually the normal asymptotic theory used to approximate the p-values for these tests. In this note we suggest using a saddlepoint approach that almost exact and need no extensive simulation calculati…
In this paper we propose a class of weighted rank correlation coefficients extending the Spearman's rho. The proposed class constructed by giving suitable weights to the distance between two sets of ranks to place more emphasis on items having low rankings than those have high rankings or vice versa. The asymptotic distribution of the proposed measures and properties of the parameters estimated by them are studied through the associated copula. A simulation study is performed to compare the performance of the proposed statistics for testing independence using asymptotic relative efficiency cal…
We study the spherical cap packing problem with a probabilistic approach. Such probabilistic considerations result in an asymptotic sharp universal uniform bound on the maximal inner product between any set of unit vectors and a stochastically independent uniformly distributed unit vector. When the set of unit vectors are themselves independently uniformly distributed, we further develop the extreme value distribution limit of the maximal inner product, which characterizes its uncertainty around the bound. As applications of the above asymptotic results, we derive (1) an asymptotic sharp unive…
Verifying probabilistic forecasts for extreme events is a highly active research area because popular media and public opinions are naturally focused on extreme events, and biased conclusions are readily made. In this context, classical verification methods tailored for extreme events, such as thresholded and weighted scoring rules, have undesirable properties that cannot be mitigated, and the well-known continuous ranked probability score (CRPS) is no exception. In this paper, we define a formal framework for assessing the behavior of forecast evaluation procedures with respect to extreme eve…
This paper investigates the use of extreme value theory for modelling the distribution of demand-net-of-wind for capacity adequacy assessment. Extreme value theory approaches are well-established and mathematically justified methods for estimating the tails of a distribution and so are ideally suited for problems in capacity adequacy, where normally only the tails of the relevant distributions are significant. The extreme value theory peaks over threshold approach is applied directly to observations of demand-net-of-wind, meaning that no assumption is needed about the nature of any dependence …
A possible drawback of the ordinary correlation coefficient $ρ$ for two real random variables $X$ and $Y$ is that zero correlation does not imply independence. In this paper we introduce a new correlation coefficient $ρ^*$ which assumes values between zero and one, equalling zero iff the two variables are independent and equalling one iff the two variables are linearly related. The coefficients $ρ^*$ and $ρ^2$ are shown to be closely related algebraically, and they coincide for distributions on a $2\times 2$ contingency table. We derive an orthogonal decomposition of $ρ^*$ as a positively weig…
We study the problem of detecting an abrupt change to the signal covariance matrix. In particular, the covariance changes from a "white" identity matrix to an unknown spiked or low-rank matrix. Two sequential change-point detection procedures are presented, based on the largest and the smallest eigenvalues of the sample covariance matrix. To control false-alarm-rate, we present an accurate theoretical approximation to the average-run-length (ARL) and expected detection delay (EDD) of the detection, leveraging the extreme eigenvalue distributions from random matrix theory and by capturing a non…
The classical multivariate extreme value theory tries to capture the extremal dependence between the components under a multivariate domain of attraction condition and it requires each of the components to be in the domain of attraction of a univariate extreme value distribution as well. The multivariate extreme value (MEV) model has a rich theory but has some limitations as it fails to capture the dependence structure in presence of asymptotic independence. A different approach to MEV was given by Heffernan and Tawn (2004), where they examined MEV distributions by conditioning on one of the c…
We study the spherical cap packing problem with a probabilistic approach. Such probabilistic considerations result in an asymptotic sharp universal uniform bound on the maximal inner product between any set of unit vectors and a stochastically independent uniformly distributed unit vector. When the set of unit vectors are themselves independently uniformly distributed, we further develop the extreme value distribution limit of the maximal inner product, which characterizes its uncertainty around the bound. As applications of the above asymptotic results, we derive (1) an asymptotic sharp unive…
There is an increasing interest to understand the dependence structure of a random vector not only in the center of its distribution but also in the tails. Extreme-value theory tackles the problem of modelling the joint tail of a multivariate distribution by modelling the marginal distributions and the dependence structure separately. For estimating dependence at high levels, the stable tail dependence function and the spectral measure are particularly convenient. These objects also lie at the basis of nonparametric techniques for modelling the dependence among extremes in the max-domain of at…
Extremes play a special role in Anomaly Detection. Beyond inference and simulation purposes, probabilistic tools borrowed from Extreme Value Theory (EVT), such as the angular measure, can also be used to design novel statistical learning methods for Anomaly Detection/ranking. This paper proposes a new algorithm based on multivariate EVT to learn how to rank observations in a high dimensional space with respect to their degree of 'abnormality'. The procedure relies on an original dimension-reduction technique in the extreme domain that possibly produces a sparse representation of multivariate e…
Rank correlations have found many innovative applications in the last decade. In particular, suitable rank correlations have been used for consistent tests of independence between pairs of random variables. Using ranks is especially appealing for continuous data as tests become distribution-free. However, the traditional concept of ranks relies on ordering data and is, thus, tied to univariate observations. As a result, it has long remained unclear how one may construct distribution-free yet consistent tests of independence between random vectors. This is the problem addressed in this paper, i…
Testing mutual independence for high-dimensional observations is a fundamental statistical challenge. Popular tests based on linear and simple rank correlations are known to be incapable of detecting non-linear, non-monotone relationships, calling for methods that can account for such dependences. To address this challenge, we propose a family of tests that are constructed using maxima of pairwise rank correlations that permit consistent assessment of pairwise independence. Built upon a newly developed Cramér-type moderate deviation theorem for degenerate U-statistics, our results cover a vari…
We follow up on Shi et al's (2020) and Cao's and my (2020) work on the local power of a new test for independence, Chatterjee (2019), and its relation to the local power properties of classical tests. We show quite generally that for testing independence with local alternatives either Chatterjee's rank test has no power, or it may be misleading: The Blum, Kiefer, Rosenblatt, and other omnibus classical rank tests do have some local power in any direction other than those where significant results may be misleading. We also suggest methods of selective inference in independence testing. Chatter…
Copulas are mathematical objects that fully capture the dependence structure among random variables and hence, offer a great flexibility in building multivariate stochastic models. In statistics, a copula is used as a general way of formulating a multivariate distribution in such a way that various general types of dependence can be represented. In case of bivariate sample, the notion of estimating copula is closely related to that of testing independence in a bivariate sample, as when the components of the bivariate sample are independent the copula becomes simply product of two uniform distr…
In this paper we study the problem of measuring and testing joint independence for a collection of multivariate random variables. Using the emerging theory of optimal transport (OT) based multivariate ranks, we propose a distribution-free test for multivariate joint independence. Towards this we introduce the notion of rank joint distance covariance (RJdCov), the higher-order rank analogue of the celebrated distance covariance measure, that captures the dependencies among all the subsets of the variables. The RJdCov can be easily estimated from the data without any moment assumptions and the a…
We extend to rank-based tests of multivariate independence the Chernoff-Savage and Hodges-Lehmann classical univariate results. More precisely, we show that the Taskinen, Kankainen and Oja (2004) normal-score rank test for multivariate independence uniformly dominates -- in the Pitman sense -- the classical Wilks (1935) test, which establishes the Pitman non-admissibility of the latter, and provide, for any fixed space dimensions $p,q$ of the marginals, the lower bound for the asymptotic relative efficiency, still with respect to Wilks' test, of the Wilcoxon version of the same.
Motivated by extreme value theory, max-linear Bayesian networks have been recently introduced and studied as an alternative to linear structural equation models. However, for max-linear systems the classical independence results for Bayesian networks are far from exhausting valid conditional independence statements. We use tropical linear algebra to derive a compact representation of the conditional distribution given a partial observation, and exploit this to obtain a complete description of all conditional independence relations. In the context-specific case, where conditional independence i…
Chatterjee (2021) introduced a simple new rank correlation coefficient that has attracted much recent attention. The coefficient has the unusual appeal that it not only estimates a population quantity first proposed by Dette et al. (2013) that is zero if and only if the underlying pair of random variables is independent, but also is asymptotically normal under independence. This paper compares Chatterjee's new correlation coefficient to three established rank correlations that also facilitate consistent tests of independence, namely, Hoeffding's $D$, Blum-Kiefer-Rosenblatt's $R$, and Bergsma-D…
It is well known that non-parametric methods suffer from the "curse of dimensionality". We propose here a new estimation method for a multivariate distribution, using sub-sampling and ranks, which seems not to suffer from this "curse". We prove that in case of independence, the uncertainty of the estimated distribution increases almost linearly w.r.t. the dimension, for dimensions around 6. Otherwise, a simulation study shows that if we use this estimation to build an independence test, the number of observations needed to obtain a given power increases linearly with the dimension. Finally, we…
Matrix Schubert varieties are certain varieties in the affine space of square matrices which are determined by specifying rank conditions on submatrices. We study these varieties for generic matrices, symmetric matrices, and upper triangular matrices in view of two applications to algebraic statistics: we observe that special conditional independence models for Gaussian random variables are intersections of matrix Schubert varieties in the symmetric case. Consequently, we obtain a combinatorial primary decomposition algorithm for some conditional independence ideals. We also characterize the v…
We consider strictly stationary heavy tailed time series whose finite-dimensional exponent measures are concentrated on axes, and hence their extremal properties cannot be tackled using classical multivariate regular variation that is suitable for time series with extremal dependence. We recover relevant information about limiting behavior of time series with extremal independence by introducing a sequence of scaling functions and conditional scaling exponent. Both quantities provide more information about joint extremes than a widely used tail dependence coefficient. We calculate the scaling …
We impose rank one constraints on marginalizations of a tensor, given by a simplicial complex. Following work of Kirkup and Sullivant, such marginal independence models can be made toric by a linear change of coordinates. We study their toric ideals, with emphasis on random graph models and independent set polytopes of matroids. We develop the numerical algebra of parameter estimation, using both Euclidean distance and maximum likelihood, and we present a comprehensive database of small models.
Deheuvels [J. Multivariate Anal. 11 (1981) 102--113] and Genest and Rémillard [Test 13 (2004) 335--369] have shown that powerful rank tests of multivariate independence can be based on combinations of asymptotically independent Cramér--von Mises statistics derived from a Möbius decomposition of the empirical copula process. A result on the large-sample behavior of this process under contiguous sequences of alternatives is used here to give a representation of the limiting distribution of such test statistics and to compute their relative local asymptotic efficiency. Local power curves and asym…
The PC algorithm uses conditional independence tests for model selection in graphical modeling with acyclic directed graphs. In Gaussian models, tests of conditional independence are typically based on Pearson correlations, and high-dimensional consistency results have been obtained for the PC algorithm in this setting. We prove that high-dimensional consistency carries over to the broader class of Gaussian copula or \textit{nonparanormal} models when using rank-based measures of correlation. For graphs with bounded degree, our result is as strong as prior Gaussian results. In simulations, the…
The class of Dirichlet random vectors is central in numerous probabilistic and statistical applications. The main result of this paper derives the exact tail asymptotics of the aggregated risk of powers of Dirichlet random vectors when the radial component has df in the Gumbel or the Weibull max-domain of attraction. We present further results for the joint asymptotic independence and the max-sum equivalence.
The varying-coefficient model is an important nonparametric statistical model that allows us to examine how the effects of covariates vary with exposure variables. When the number of covariates is big, the issue of variable selection arrives. In this paper, we propose and investigate marginal nonparametric screening methods to screen variables in ultra-high dimensional sparse varying-coefficient models. The proposed nonparametric independence screening (NIS) selects variables by ranking a measure of the nonparametric marginal contributions of each covariate given the exposure variable. The sur…
Models based on assumptions of multivariate regular variation and hidden regular variation provide ways to describe a broad range of extremal dependence structures when marginal distributions are heavy tailed. Multivariate regular variation provides a rich description of extremal dependence in the case of asymptotic dependence, but fails to distinguish between exact independence and asymptotic independence. Hidden regular variation addresses this problem by requiring components of the random vector to be simultaneously large but on a smaller scale than the scale for the marginal distributions.…
