Modern statisticians are often presented with hundreds or thousands of hypothesis testing problems to evaluate at the same time, generated from new scientific technologies such as microarrays, medical and satellite imaging devices, or flow cytometry counters. The relevant statistical literature tends to begin with the tacit assumption that a single combined analysis, for instance, a False Discovery Rate assessment, should be applied to the entire set of problems at hand. This can be a dangerous assumption, as the examples in the paper show, leading to overly conservative or overly liberal conclusions within any particular subclass of the cases. A simple Bayesian theory yields a succinct description of the effects of separation or combination on false discovery rate analyses. The theory allows efficient testing within small subclasses, and has applications to ``enrichment,'' the detection of multi-case effects.
We aim to infer bioactivity of each chemical by assay endpoint combination, addressing sparsity of toxicology data. We propose a Bayesian hierarchical framework which borrows information across different chemicals and assay endpoints, facilitates out-of-sample prediction of activity for chemicals not yet assayed, quantifies uncertainty of predicted activity, and adjusts for multiplicity in hypothesis testing. Furthermore, this paper makes a novel attempt in toxicology to simultaneously model heteroscedastic errors and a nonparametric mean function, leading to a broader definition of activity w…
This article introduces a robust hypothesis testing procedure: the Lq-likelihood-ratio-type test (LqRT). By deriving the asymptotic distribution of this test statistic, the authors demonstrate its robustness both analytically and numerically, and they investigate the properties of both its influence function and its breakdown point. A proposed method to select the tuning parameter q offers a good efficiency/robustness trade-off, compared with the traditional likelihood ratio test (LRT) and other robust tests. A simulation and real data analysis provides further evidence of the advantages of th…
Mediation hypothesis testing for a large number of mediators is challenging due to the composite structure of the null hypothesis, H0:alpha*beta=0 (alpha: effect of the exposure on the mediator after adjusting for confounders; beta: effect of the mediator on the outcome after adjusting for exposure and confounders). In this paper, we reviewed three classes of methods for multiple mediation hypothesis testing. In addition to these existing methods, we developed the Sobel-comp method, which uses a corrected mixture reference distribution for Sobel's test statistic. We performed extensive simulat…
Making binary decisions is a common data analytical task in scientific research and industrial applications. In data sciences, there are two related but distinct strategies: hypothesis testing and binary classification. In practice, how to choose between these two strategies can be unclear and rather confusing. Here we summarize key distinctions between these two strategies in three aspects and list five practical guidelines for data analysts to choose the appropriate strategy for specific analysis needs. We demonstrate the use of those guidelines in a cancer driver gene prediction example.
We propose a kernel machine based hypothesis testing procedure in nonlinear function-on-scalar regression model. Our research is motivated by the Newborn Epigenetic Study (NEST) where the question of interest is whether a pre-specified group of toxic metals or methylation at any of 9 differentially methylated regions (DMRs) is associated with child growth. We take the child growth trajectory as the functional response, and model the toxic metal measurements jointly using a nonlinear function. We use a kernel machine approach to model the unknown function and transform the hypothesis of no effe…
We study the portfolio problem of maximizing the outperformance probability over a random benchmark through dynamic trading with a fixed initial capital. Under a general incomplete market framework, this stochastic control problem can be formulated as a composite pure hypothesis testing problem. We analyze the connection between this pure testing problem and its randomized counterpart, and from latter we derive a dual representation for the maximal outperformance probability. Moreover, in a complete market setting, we provide a closed-form solution to the problem of beating a leveraged exchang…
Motivated by gene set enrichment analysis, we investigate the problem of combined hypothesis testing on a graph. We introduce a general framework to effectively use the structural information of the underlying graph when testing multivariate means. A new testing procedure is proposed within this framework. We show that the test is optimal in that it can consistently detect departure from the collective null at a rate that no other test could improve, for almost all graphs. We also provide general performance bounds for the proposed test under any specific graph, and illustrate their utility th…
We develop hypothesis testing for active information -the averaged quantity in the Kullback-Liebler divergence. To our knowledge, this is the first paper to derive exact probabilities of type-I errors for hypothesis testing in the area.
It is observed that for testing between simple hypotheses where the cost of Type I and Type II errors can be quantified, it is better to let the optimization choose the test size.
We provide necessary and sufficient conditions of uniform consistency of nonparametric sets of alternatives of chi-squared test for testing of hypothesis of homogeneity. The number of cells of chi-squared test increases with sample size growth. Nonparametric sets of alternatives can be defined both in terms of densities and distribution functions.
Bayesian hypothesis testing is investigated when the prior probabilities of the hypotheses, taken as a random vector, are quantized. Nearest neighbor and centroid conditions are derived using mean Bayes risk error as a distortion measure for quantization. A high-resolution approximation to the distortion-rate function is also obtained. Human decision making in segregated populations is studied assuming Bayesian hypothesis testing with quantized priors.
We consider the problem of hypothesis testing in the situation where the first hypothesis is simple and the second one is local one-sided composite. We describe the choice of the thresholds and the power functions of different tests when the intensity function of the observed inhomogeneous Poisson process has two different types of singularity: cusp and discontinuity. The asymptotic results are illustrated by numerical simulations.
In this paper, we consider the hypothesis testing of correlation between two $m$-uniform hypergraphs on $n$ unlabelled nodes. Under the null hypothesis, the hypergraphs are independent, while under the alternative hypothesis, the hyperdges have the same marginal distributions as in the null hypothesis but are correlated after some unknown node permutation. We focus on two scenarios: the hypergraphs are generated from the Gaussian-Wigner model and the dense Erdös-Rényi model. We derive the sharp information-theoretic testing threshold. Above the threshold, there exists a powerful test to distin…
We treat quantum counterparts of testing problems whose optimal tests are given by chi-square, t and F tests. These quantum counterparts are formulated as quantum hypothesis testing problems concerning quantum Gaussian states families, and contain disturbance parameters, which have group symmetry. Quantum Hunt-Stein Theorem removes a part of these disturbance parameters, but other types of difficulty still remain. In order to remove them, combining quantum Hunt-Stein theorem and other reduction methods, we establish a general reduction theorem that reduces a complicated quantum hypothesis test…
This letter presents a novel Block Bayesian Hypothesis Testing Algorithm (Block-BHTA) for reconstructing block sparse signals with unknown block structures. The Block-BHTA comprises the detection and recovery of the supports, and the estimation of the amplitudes of the block sparse signal. The support detection and recovery is performed using a Bayesian hypothesis testing. Then, based on the detected and reconstructed supports, the nonzero amplitudes are estimated by linear MMSE. The effectiveness of Block-BHTA is demonstrated by numerical experiments.
We study a variation of Bayesian M-ary hypothesis testing in which the test outputs a list of L candidates out of the M possible upon processing the observation. We study the minimum error probability of list hypothesis testing, where an error is defined as the event where the true hypothesis is not in the list output by the test. We derive two exact expressions of the minimum probability or error. The first is expressed as the error probability of a certain non-Bayesian binary hypothesis test, and is reminiscent of the meta-converse bound. The second, is expressed as the tail probability of t…
Rice University（莱斯大学）；Department of Economics（经济学系）
In this work we use Recurrent Neural Networks and Multilayer Perceptrons to predict NYSE, NASDAQ and AMEX stock prices from historical data. We experiment with different architectures and compare data normalization techniques. Then, we leverage those findings to question the efficient-market hypothesis through a formal statistical test.
This paper surveys some recent developments in fundamental limits and optimal algorithms for network analysis. We focus on minimax optimal rates in three fundamental problems of network analysis: graphon estimation, community detection, and hypothesis testing. For each problem, we review state-of-the-art results in the literature followed by general principles behind the optimal procedures that lead to minimax estimation and testing. This allows us to connect problems in network analysis to other statistical inference problems from a general perspective.
Indian Institute of Science（印度科学学院）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）；MPI-SWS（马克斯·普朗克软件系统研究所）
We study nonzero-sum hypothesis testing games that arise in the context of adversarial classification, in both the Bayesian as well as the Neyman-Pearson frameworks. We first show that these games admit mixed strategy Nash equilibria, and then we examine some interesting concentration phenomena of these equilibria. Our main results are on the exponential rates of convergence of classification errors at equilibrium, which are analogous to the well-known Chernoff-Stein lemma and Chernoff information that describe the error exponents in the classical binary hypothesis testing problem, but with pa…
Hypothesis testing for small-sample scenarios is a practically important problem. In this paper, we investigate the robust hypothesis testing problem in a data-driven manner, where we seek the worst-case detector over distributional uncertainty sets centered around the empirical distribution from samples using Sinkhorn distance. Compared with the Wasserstein robust test, the corresponding least favorable distributions are supported beyond the training samples, which provides a more flexible detector. Various numerical experiments are conducted on both synthetic and real datasets to validate th…
Randomized response is one of the oldest and most well-known methods for analyzing confidential data. However, its utility for differentially private hypothesis testing is limited because it cannot achieve high privacy levels and low type I error rates simultaneously. In this article, we show how to overcome this issue with the subsample and aggregate technique. The result is a general-purpose method that can be used for both frequentist and Bayesian testing. {{We illustrate the performance of our proposal in three scenarios: goodness-of-fit testing for linear regression models, nonparametric …
The likelihood ratio statistic, with its asymptotic $χ^2$ distribution at regular model points, is often used for hypothesis testing. At model singularities and boundaries, however, the asymptotic distribution may not be $χ^2$, as highlighted by recent work of Drton. Indeed, poor behavior of a $χ^2$ for testing near singularities and boundaries is apparent in simulations, and can lead to conservative or anti-conservative tests. Here we develop a new distribution designed for use in hypothesis testing near singularities and boundaries, which asymptotically agrees with that of the likelihood rat…
We propose a simple robust hypothesis test that has the same sample complexity as that of the optimal Neyman-Pearson test up to constants, but robust to distribution perturbations under Hellinger distance. We discuss the applicability of such a robust test for estimating distributions in Hellinger distance. We empirically demonstrate the power of the test on canonical distributions.
The problem of multiple hypothesis testing arises when there are more than one hypothesis to be tested simultaneously for statistical significance. This is a very common situation in many data mining applications. For instance, assessing simultaneously the significance of all frequent itemsets of a single dataset entails a host of hypothesis, one for each itemset. A multiple hypothesis testing method is needed to control the number of false positives (Type I error). Our contribution in this paper is to extend the multiple hypothesis framework to be used with a generic data mining algorithm. We…
The power of multiple testing procedures can be increased by using weighted p-values (Genovese, Roeder and Wasserman 2005). We derive the optimal weights and we show that the power is remarkably robust to misspecification of these weights. We consider two methods for choosing weights in practice. The first, external weighting, is based on prior information. The second, estimated weighting, uses the data to choose weights.
We consider the problem of hypothesis testing in the situation when the first hypothesis is simple and the second one is local one-sided composite. We describe the choice of the thresholds and the power functions of the Score Function test, of the General Likelihood Ratio test, of the Wald test and of two Bayes tests in the situation when the intensity function of the observed inhomogeneous Poisson process is smooth with respect to the parameter. It is shown that almost all these tests are asymptotically uniformly most powerful. The results of numerical simulations are presented.
We study a class of distributed hypothesis testing against conditional independence problems. Under the criterion that stipulates minimization of the Type II error rate subject to a (constant) upper bound $ε$ on the Type I error rate, we characterize the set of encoding rates and exponent for both discrete memoryless and memoryless vector Gaussian settings.
A vast amount of expert and domain knowledge is captured by causal structural priors, yet there has been little research on testing such priors for generalization and data synthesis purposes. We propose a novel model architecture, Causal Structural Hypothesis Testing, that can use nonparametric, structural causal knowledge and approximate a causal model's functional relationships using deep neural networks. We use these architectures for comparing structural priors, akin to hypothesis testing, using a deliberate (non-random) split of training and testing data. Extensive simulations demonstrate…
The paper estimates the Chernoff rate for the efficiency of quantum hypothesis testing. For both joint and separable measurements, approximate bounds for the rate are given if both states are mixed and exact expressions are derived if at least one of the states is pure. The efficiency of tests with separable measurements is found to be close to the efficiency of tests with joint measurements. The results are illustrated by a test of quantum entanglement.
We interpret likelihood-based test functions from a geometric perspective where the Kullback-Leibler (KL) divergence is adopted to quantify the distance from a distribution to another. Such a test function can be seen as a sub-Gaussian random variable, and we propose a principled way to calculate its corresponding sub-Gaussian norm. Then an error bound for binary hypothesis testing can be obtained in terms of the sub-Gaussian norm and the KL divergence, which is more informative than Pinsker's bound when the significance level is prescribed. For $M$-ary hypothesis testing, we also derive an er…
We discuss an "operational" approach to testing convex composite hypotheses when the underlying distributions are heavy-tailed. It relies upon Euclidean separation of convex sets and can be seen as an extension of the approach to testing by convex optimization developed in [8, 12]. In particular, we show how one can construct quasi-optimal testing procedures for families of distributions which are majorated, in a certain precise sense, by a sub-spherical symmetric one and study the relationship between tests based on Euclidean separation and "potential-based tests." We apply the promoted metho…
In this study, we propose a two-stage procedure for hypothesis testing, where the first stage is conventional hypothesis testing and the second is an equivalence testing procedure using an introduced Empirical Equivalence Bound. In 2016, the American Statistical Association released a policy statement on P-values to clarify the proper use and interpretation in response to the criticism of reproducibility and replicability in scientific findings. A recent solution to improve reproducibility and transparency in statistical hypothesis testing is to integrate P-values (or confidence intervals) wit…
Topological data analysis involves the statistical characterization of the shape of data. Persistent homology is a primary tool of topological data analysis, which can be used to analyze topological features and perform statistical inference. In this paper, we present a two-stage hypothesis test for vectorized persistence diagrams. The first stage filters vector elements in the vectorized persistence diagrams to enhance the power of the test. The second stage consists of multiple hypothesis tests, with false positives controlled by false discovery rates. We demonstrate the flexibility of our m…
We consider a hypothesis testing problem where a part of data cannot be observed. Our helper observes the missed data and can send us a limited amount of information about them. What kind of this limited information will allow us to make the best statistical inference? In particular, what is the minimum information sufficient to obtain the same results as if we directly observed all the data? We derive estimates for this minimum information and some other similar results.
We address the following question in this paper: "What are the most robust statistical methods for social choice?'' By leveraging the theory of uniformly least favorable distributions in the Neyman-Pearson framework to finite models and randomized tests, we characterize uniformly most powerful (UMP) tests, which is a well-accepted statistical optimality w.r.t. robustness, for testing whether a given alternative is the winner under Mallows' model and under Condorcet's model, respectively.
Bayesian supervised predictive classifiers, hypothesis testing, and parametric estimation under Partition Exchangeability are implemented. The two classifiers presented are the marginal classifier (that assumes test data is i.i.d.) next to a more computationally costly but accurate simultaneous classifier (that finds a labelling for the entire test dataset at once based on simultanous use of all the test data to predict each label). We also provide the Maximum Likelihood Estimation (MLE) of the only underlying parameter of the partition exchangeability generative model as well as hypothesis te…
The classical binary hypothesis testing problem is revisited. We notice that when one of the hypotheses is composite, there is an inherent difficulty in defining an optimality criterion that is both informative and well-justified. For testing in the simple normal location problem (that is, testing for the mean of multivariate Gaussians), we overcome the difficulty as follows. In this problem there exists a natural hardness order between parameters as for different parameters the error-probailities curves (when the parameter is known) are either identical, or one dominates the other. We can thu…
The two-sample hypothesis testing problem is studied for the challenging scenario of high dimensional data sets with small sample sizes. We show that the two-sample hypothesis testing problem can be posed as a one-class set classification problem. In the set classification problem the goal is to classify a set of data points that are assumed to have a common class. We prove that the average probability of error given a set is less than or equal to the Bayes error and decreases as a power of $n$ number of sample data points in the set. We use the positive definite Set Kernel for directly mappin…
Hypothesis testing is a useful statistical tool in determining whether a given model should be rejected based on a sample from the population. Sample data may contain sensitive information about individuals, such as medical information. Thus it is important to design statistical tests that guarantee the privacy of subjects in the data. In this work, we study hypothesis testing subject to differential privacy, specifically chi-squared tests for goodness of fit for multinomial data and independence between two categorical variables. We propose new tests for goodness of fit and independence testi…
Two variational formulas for the power of the binary hypothesis testing problem are derived. The first is given as the Legendre transform of a certain function and the second, induced from the first, is given in terms of the Cumulative Distribution Function (CDF) of the log-likelihood ratio. One application of the first formula is an upper bound on the power of the binary hypothesis testing problem in terms of the Re'nyi divergence. The second formula provide a general framework for proving asymptotic and non-asymptotic expressions for the power of the test utilizing corresponding expressions …
Neural networks are powerful predictive models, but they provide little insight into the nature of relationships between predictors and outcomes. Although numerous methods have been proposed to quantify the relative contributions of input features, statistical inference and hypothesis testing of feature associations remain largely unexplored. We propose a permutation-based approach to testing that uses the partial derivatives of the network output with respect to specific inputs to assess both the significance of input features and whether significant features are linearly associated with the …
In this article, we study the hypothesis testing of the blip / net effects of treatments in a treatment sequence. We illustrate that the likelihood ratio test and the score test may suffer from the curse of dimensionality, the null paradox and the high-dimensional constraint on standard parameters under the null hypothesis. On the other hand, we construct the Wald test via a small number of point effects of treatments in single-point causal inference. We show that the Wald test can avoid these problems under the same assumptions as the Wald test for testing the point effect of treatment. The s…
School of Mathematical and Computer Sciences, Heriot-Watt University（赫瑞斯泰学院数学与计算机科学系，赫瑞瓦特大学）；School of Mathematics, University of Edinburgh（爱丁堡大学数学学院）；Maxwell Institute for Mathematical Sciences（麦克斯韦数学科学研究所）
Multiple hypothesis testing often involves composite nulls, i.e., nulls that are associated with two or more distributions. In many cases, it is reasonable to assume that there is a prior distribution on the distributions despite it is unknown. When the number of distributions under true nulls is finite, we show that under the above assumption, the false discover rate (FDR) can be controlled using $p$-values computed under constraints imposed by the empirical distribution of the observations. Comparing to FDR control using $p$-values defined as maximum significance level over all null distribu…
Harvard John A. Paulson School Of Engineering And Applied Sciences（哈佛大学约翰·A·保尔森工程与应用科学学院）；Georgia Institute of Technology（佐治亚理工学院）；H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology（佐治亚理工学院H·米尔顿·斯图尔特工业与系统工程学院）；Department of Industrial Engineering and Operations Research, Columbia University（哥伦比亚大学工业工程与运筹学系）
The sequential hypothesis testing problem is a class of statistical analyses where the sample size is not fixed in advance. Instead, the decision-process takes in new observations sequentially to make real-time decisions for testing an alternative hypothesis against a null hypothesis until some stopping criterion is satisfied. In many common applications of sequential hypothesis testing, the data can be highly sensitive and may require privacy protection; for example, sequential hypothesis testing is used in clinical trials, where doctors sequentially collect data from patients and must determ…
