In large-scale studies with parallel signal-plus-noise observations, the local false discovery rate is a summary statistic that is often presumed to be equal to the posterior probability that the signal is null. We prefer to call the latter quantity the local null-signal rate to emphasize our view that a null signal and a false discovery are not identical events. The local null-signal rate is commonly estimated through empirical Bayes procedures that build on the `zero density assumption,' which attributes the density of observations near zero entirely to null signals. In this paper, we argue that this strategy does not furnish estimates of the local null-signal rate, but instead of a quantity we call the complementary local activity rate (clar). Although it is likely to be small, an inactive signal is not necessarily zero. The clar dominates both the local null-signal rate and the local false sign rate and is a weakly continuous functional of the signal distribution. As a consequence, it takes on sensible values when the signal is sparse but not exactly zero. Our findings clarify the interpretation of local false discovery rates estimated under the zero density assumption.
Two major research tasks lie at the heart of high dimensional data analysis: accurate parameter estimation and correct support recovery. The existing literature mostly aims for either the best parameter estimation or the best model selection result, however little has been done to understand the potential interaction between the estimation precision and the selection behavior. In this work, our minimax result shows that an estimator's performance of type I error control directly links with its $L_2$ estimation error rate, and reveals a trade-off phenomenon between the rate of convergence and t…
We consider the class of all multiple testing methods controlling tail probabilities of the false discovery proportion, either for one random set or simultaneously for many such sets. This class encompasses methods controlling familywise error rate, generalized familywise error rate, false discovery exceedance, joint error rate, simultaneous control of all false discovery proportions, and others, as well as seemingly unrelated methods such as gene set testing in genomics and cluster inference methods in neuroimaging. We show that all such methods are either equivalent to a closed testing metho…
New procedures controlling the false discovery proportion via Romano-Wolf's heuristic
当同时执行大量$m$测试时，错误发现比例（FDP）是一种解释误报的方便方法。罗曼诺和沃尔夫[安。35（2007）1378-1408]已经提出了一种一般原理，其从$k$ -族式差错率控制过程构建FDP控制过程，同时以适当的方式并入依赖性;参见Korn等人[J. Statist.计划Inference 124（2004）379-398]; Romano and Wolf（2007）.然而，后者的理论有效性在很大程度上仍然是未知的。本文提供了一个仔细的研究，这种启发式：首先，我们扩展这种方法，使用的概念“边界设备”，使我们能够涵盖范围广泛的临界值，包括那些适应$m\_0$，真实零假设的数量。其次，后者的理论有效性进行了调查，非渐近和渐近。第三，我们引入适当的修改，这种启发式提供了新的方法，克服了现有的程序与一个行之有效的FDP控制。
Multiple testing has been a popular topic in statistical research. Although vast works have been done, controlling the false discoveries remains a challenging task when the corresponding test statistics are dependent. Various methods have been proposed to estimate the false discovery proportion (FDP) under arbitrary dependence among the test statistics. One of the main ideas is to reduce arbitrary dependence to weak dependence and then to establish theoretically the strong consistency of the FDP and false discovery rate (FDR) under weak dependence. As a consequence, FDPs share the same asympto…
While traditional multiple testing procedures prohibit adaptive analysis choices made by users, Goeman and Solari (2011) proposed a simultaneous inference framework that allows users such flexibility while preserving high-probability bounds on the false discovery proportion (FDP) of the chosen set. In this paper, we propose a new class of such simultaneous FDP bounds, tailored for nested sequences of rejection sets. While most existing simultaneous FDP bounds are based on closed testing using global null tests based on sorted p-values, we additionally consider the setting where side informatio…
Multiple testing is a fundamental problem in high-dimensional statistical inference. Although many methods have been proposed to control false discoveries, it is still a challenging task when the tests are correlated to each other. To overcome this challenge, various methods have been proposed to estimate the false discovery rate (FDR) and/or the false discovery proportion (FDP) under arbitrary covariance among the test statistics. An interesting finding of these works is that the estimation of FDP and FDR under weak dependence is identical to that under independence. However, Mei et al. (2021…
E-values have gained attention as potential alternatives to p-values as measures of uncertainty, significance and evidence. In brief, e-values are realized by random variables with expectation at most one under the null; examples include betting scores, (point null) Bayes factors, likelihood ratios and stopped supermartingales. We design a natural analog of the Benjamini-Hochberg (BH) procedure for false discovery rate (FDR) control that utilizes e-values, called the e-BH procedure, and compare it with the standard procedure for p-values. One of our central results is that, unlike the usual BH…
The highly influential two-group model in testing a large number of statistical hypotheses assumes that the test statistics are drawn independently from a mixture of a high probability null distribution and a low probability alternative. Optimal control of the marginal false discovery rate (mFDR), in the sense that it provides maximal power (expected true discoveries) subject to mFDR control, is known to be achieved by thresholding the local false discovery rate (locFDR), i.e., the probability of the hypothesis being null given the set of test statistics, with a fixed threshold. We address the…
We propose a general and flexible procedure for testing multiple hypotheses about sequential (or streaming) data that simultaneously controls both the false discovery rate (FDR) and false nondiscovery rate (FNR) under minimal assumptions about the data streams which may differ in distribution, dimension, and be dependent. All that is needed is a test statistic for each data stream that controls the conventional type I and II error probabilities, and no information or assumptions are required about the joint distribution of the statistics or data streams. The procedure can be used with sequenti…
We introduce a new class of methods for finite-sample false discovery rate (FDR) control in multiple testing problems with dependent test statistics where the dependence is fully or partially known. Our approach separately calibrates a data-dependent p-value rejection threshold for each hypothesis, relaxing or tightening the threshold as appropriate to target exact FDR control. In addition to our general framework we propose a concrete algorithm, the dependence-adjusted Benjamini-Hochberg (dBH) procedure, which adaptively thresholds the q-value for each hypothesis. Under positive regression de…
The traditional approaches to false discovery rate (FDR) control in multiple hypothesis testing are usually based on the null distribution of a test statistic. However, all types of null distributions, including the theoretical, permutation-based and empirical ones, have some inherent drawbacks. For example, the theoretical null might fail because of improper assumptions on the sample distribution. Here, we propose a null distribution-free approach to FDR control for multiple hypothesis testing. This approach, named target-decoy procedure, simply builds on the ordering of tests by some statist…
We develop a new class of distribution--free multiple testing rules for false discovery rate (FDR) control under general dependence. A key element in our proposal is a symmetrized data aggregation (SDA) approach to incorporating the dependence structure via sample splitting, data screening and information pooling. The proposed SDA filter first constructs a sequence of ranking statistics that fulfill global symmetry properties, and then chooses a data--driven threshold along the ranking to control the FDR. The SDA filter substantially outperforms the knockoff method in power under moderate to s…
Harvard University（哈佛大学）；University of Pennsylvania（宾夕法尼亚大学）；Google Research（谷歌研究院）
Differential privacy provides a rigorous framework for privacy-preserving data analysis. This paper proposes the first differentially private procedure for controlling the false discovery rate (FDR) in multiple hypothesis testing. Inspired by the Benjamini-Hochberg procedure (BHq), our approach is to first repeatedly add noise to the logarithms of the $p$-values to ensure differential privacy and to select an approximately smallest $p$-value serving as a promising candidate at each iteration; the selected $p$-values are further supplied to the BHq and our private procedure releases only the re…
In this paper, a noisy version of the stochastic block model (NSBM) is introduced and we investigate the three following statistical inferences in this model: estimation of the model parameters, clustering of the nodes and identification of the underlying graph. While the two first inferences are done by using a variational expectation-maximization (VEM) algorithm, the graph inference is done by controlling the false discovery rate (FDR), that is, the average proportion of errors among the edges declared significant, and by maximizing the true discovery rate (TDR), that is, the average proport…
False Discovery Controlled Regions for Excursion Sets
University of California, San Diego（加州大学圣地亚哥分校）；Big Data Institute, Nuffield Department of Population Health, University of Oxford（牛津大学纳菲尔德人口健康学院大数据研究所）；School of Mathematics, University of Bristol（布里斯托尔大学数学学院）；Halıcıoğlu Data Science Institute, University of California, San Diego（加州大学圣地亚哥分校哈奇奥格鲁数据科学研究所）
Georgia Institute of Technology（佐治亚理工学院）；University of Waterloo（滑铁卢大学）
In hypothesis testing, a false discovery occurs when a hypothesis is incorrectly rejected due to noise in the sample. When adaptively testing multiple hypotheses, the probability of a false discovery increases as more tests are performed. Thus the problem of False Discovery Rate (FDR) control is to find a procedure for testing multiple hypotheses that accounts for this effect in determining the set of hypotheses to reject. The goal is to minimize the number (or fraction) of false discoveries, while maintaining a high true positive rate (i.e., correct discoveries). In this work, we study False …
In the multiple testing problem with independent tests, the classical linear step-up procedure controls the false discovery rate (FDR) at level $π_0α$, where $π_0$ is the proportion of true null hypotheses and $α$ is the target FDR level. Adaptive procedures can improve power by incorporating estimates of $π_0$, which typically rely on a tuning parameter. Fixed adaptive procedures set their tuning parameters before seeing the data and can be shown to control the FDR in finite samples. We develop theoretical results for dynamic adaptive procedures whose tuning parameters are determined by the d…
This article proposes novel rules for false discovery rate control (FDRC) geared towards online anomaly detection in time series. Online FDRC rules allow to control the properties of a sequence of statistical tests. In the context of anomaly detection, the null hypothesis is that an observation is normal and the alternative is that it is anomalous. FDRC rules allow users to target a lower bound on precision in unsupervised settings. The methods proposed in this article overcome short-comings of previous FDRC rules in the context of anomaly detection, in particular ensuring that power remains h…
This paper is concerned with false discovery rate (FDR) control in large-scale multiple testing problems. We first propose a new data-driven testing procedure for controlling the FDR in large-scale t-tests for one-sample mean problem. The proposed procedure achieves exact FDR control in finite sample settings when the populations are symmetric no matter the number of tests or sample sizes. Comparing with the existing bootstrap method for FDR control, the proposed procedure is computationally efficient. We show that the proposed method can control the FDR asymptotically for asymmetric populatio…
