Permutation procedures are common practice in hypothesis testing when distributional assumptions about the test statistic are not met or unknown. With only few permutations, empirical p-values lie on a coarse grid and may even be zero when the observed test statistic exceeds all permuted values. Such zero p-values are statistically invalid and hinder multiple testing correction. Parametric tail modeling with the Generalized Pareto Distribution (GPD) has been proposed to address this issue, but existing implementations can again yield zero p-values when the estimated shape parameter is negative and the fitted distribution has a finite upper bound.
We introduce a method for accurate and zero-free p-value approximation in permutation testing, embedded in the permApprox workflow and R package. Building on GPD tail modeling, the method enforces a support constraint during parameter estimation to ensure valid extrapolation beyond the observed statistic, thereby strictly avoiding zero p-values. The workflow further integrates robust parameter estimation, data-driven threshold selection, and principled handling of hybrid p-values that are discrete in the bulk and continuous in the extreme tail.
Extensive simulations using two-sample t-tests and Wilcoxon rank-sum tests show that permApprox produces accurate, robust, and zero-free p-value approximations across a wide range of sample and effect sizes. Applications to single-cell RNA-seq and microbiome data demonstrate its practical utility: permApprox yields smooth and interpretable p-value distributions even with few permutations. By resolving the zero-p-value problem while preserving accuracy and computational efficiency, permApprox enables reliable permutation-based inference in high-dimensional and computationally intensive settings.
Multivariate meta-analysis is gaining prominence in evidence synthesis research because it enables simultaneous synthesis of multiple correlated outcome data, and random-effects models have generally been used for addressing between-studies heterogeneities. However, coverage probabilities of confidence regions or intervals for standard inference methods for random-effects models (e.g., restricted maximum likelihood estimation) cannot retain their nominal confidence levels in general, especially when the number of synthesized studies is small because their validities depend on large sample appr…
A number of biomedical problems require performing many hypothesis tests, with an attendant need to apply stringent thresholds. Often the data take the form of a series of predictor vectors, each of which must be compared with a single response vector, perhaps with nuisance covariates. Parametric tests of association are often used, but can result in inaccurate type I error at the extreme thresholds, even for large sample sizes. Furthermore, standard two-sided testing can reduce power compared to the doubled $p$-value, due to asymmetry in the null distribution. Exact (permutation) testing is a…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Stanford University（斯坦福大学）
Databases in domains such as healthcare are routinely released to the public in aggregated form. Unfortunately, naive modeling with aggregated data may significantly diminish the accuracy of inferences at the individual level. This paper addresses the scenario where features are provided at the individual level, but the target variables are only available as histogram aggregates or order statistics. We consider a limiting case of generalized linear modeling when the target variables are only known up to permutation, and explore how this relates to permutation testing; a standard technique for …
Regional frequency analysis is an important tool to properly estimate hydrological characteristics at ungauged or partially gauged sites in order to prevent hydrological disasters. The delineation of homogeneous groups of sites is an important first step in order to transfer information and obtain accurate quantile estimates at the target site. The Hosking-Wallis homogeneity test is usually used to test the homogeneity of the selected sites. Despite its usefulness and good power, it presents some drawbacks including the subjective choice of a parametric distribution for the data and a poorly j…
Federated Independent Component Analysis via Spectral Alignment and Robust Aggregation
Department of Electrical and Computer Engineering, Rutgers University, New Brunswick（罗格斯大学电气与计算机工程系）；Department of Statistical Sciences, University of Toronto（多伦多大学统计学系）
This paper investigates the problem of testing independence of two random vectors of general dimensions. For this, we give for the first time a distribution-free consistent test. Our approach combines distance covariance with the center-outward ranks and signs developed in Hallin (2017). In technical terms, the proposed test is consistent and distribution-free in the family of multivariate distributions with nonvanishing (Lebesgue) probability densities. Exploiting the (degenerate) U-statistic structure of the distance covariance and the combinatorial nature of Hallin's center-outward ranks an…
The original and amplitude permutations are two basic ordinal patterns; however, their relationship has received little attention. This paper compares the original and amplitude permutations used to characterize vector structures. To accurately convey the vector structure, we modify indexes of equal values in the permutations to be the same ones in each group of equalities. Comparative analysis suggests that the amplitude permutation, comprising the positions of the original values in the reordered vector, directly reflects the vector's temporal structure, whereas the original permutation, con…
Finkelstein-Schoenfeld, Buyse, Pocock, and other authors have developed generalizations of the Mann-Whitney test that allow for pairwise patient comparisons to include a hierarchy of measurements. Various authors present either asymptotic or randomized methods for analyzing the wins. We use graph theory concepts to derive exact means and variances for the number of wins, as a replacement for approximate values obtained from bootstrap analysis or random sampling from the permutation distribution. The time complexity of our algorithm is $O(N^2)$, where $N$ is the total number of patients. In any…
I have three goals in this article: (1) To show the enormous potential of bootstrapping and permutation tests to help students understand statistical concepts including sampling distributions, standard errors, bias, confidence intervals, null distributions, and P-values. (2) To dig deeper, understand why these methods work and when they don't, things to watch out for, and how to deal with these issues when teaching. (3) To change statistical practice---by comparing these methods to common t tests and intervals, we see how inaccurate the latter are; we confirm this with asymptotics. n >= 30 isn…
Voxel-based analysis methods localize brain structural differences by performing voxel-wise statistical comparisons on two groups of images aligned to a common space. This procedure requires highly accurate registration as well as a sufficiently large dataset. However, in practice, the registration algorithms are not perfect due to noise, artifacts, and complex structural variations. The sample size is also limited due to low disease prevalence, recruitment difficulties, and demographic matching issues. To address these issues, in this paper, we propose a method, called block-based statistic (…
Throughout the last decade, random forests have established themselves as among the most accurate and popular supervised learning methods. While their black-box nature has made their mathematical analysis difficult, recent work has established important statistical properties like consistency and asymptotic normality by considering subsampling in lieu of bootstrapping. Though such results open the door to traditional inference procedures, all formal methods suggested thus far place severe restrictions on the testing framework and their computational overhead precludes their practical scientifi…
Vecchia's approximate likelihood for Gaussian process parameters depends on how the observations are ordered, which can be viewed as a deficiency because the exact likelihood is permutation-invariant. This article takes the alternative standpoint that the ordering of the observations can be tuned to sharpen the approximations. Advantageously chosen orderings can drastically improve the approximations, and in fact, completely random orderings often produce far more accurate approximations than default coordinate-based orderings do. In addition to the permutation results, automatic methods for g…
Testing the dependency between two random variables is an important inference problem in statistics since many statistical procedures rely on the assumption that the two samples are independent. To test whether two samples are independent, a so-called HSIC (Hilbert--Schmidt Independence Criterion)-based test has been proposed. Its null distribution is approximated either by permutation or a Gamma approximation. In this paper, a new HSIC-based test is proposed. Its asymptotic null and alternative distributions are established. It is shown that the proposed test is root-n consistent. A three-cum…
