A strong converse bound for multiple hypothesis testing, with applications to high-dimensional estimation
Ramji Venkataramanan, Oliver Johnson
备注：
In the latest version, the value of $λ$ in the statements of Lemma 4.1 and Proposition 4.2 is restricted to the interval $(0,1]$. This is the correct condition, rather than $λ>0$ stated in the journal version below
发表信息：
Electronic Journal of Statistics, Vol. 12, No. 1, pp. 1126-1149, 2018
In statistical inference problems, we wish to obtain lower bounds on the minimax risk, that is to bound the performance of any possible estimator. A standard technique to obtain risk lower bounds involves the use of Fano's inequality. In an information-theoretic setting, it is known that Fano's inequality typically does not give a sharp converse result (error lower bound) for channel coding problems. Moreover, recent work has shown that an argument based on binary hypothesis testing gives tighter results. We adapt this technique to the statistical setting, and argue that Fano's inequality can always be replaced by this approach to obtain tighter lower bounds that can be easily computed and are asymptotically sharp. We illustrate our technique in three applications: density estimation, active learning of a binary classifier, and compressed sensing, obtaining tighter risk lower bounds in each case.
A distributed binary hypothesis testing (HT) problem over a noisy (discrete and memoryless) channel studied previously by the authors is investigated from the perspective of the strong converse property. It was shown by Ahlswede and Csiszár that a strong converse holds in the above setting when the channel is rate-limited and noiseless. Motivated by this observation, we show that the strong converse continues to hold in the noisy channel setting for a special case of HT known as testing against independence (TAI). The proof utilizes the blowing up lemma and the recent change of measure techniq…
We develop hypothesis testing for active information -the averaged quantity in the Kullback-Liebler divergence. To our knowledge, this is the first paper to derive exact probabilities of type-I errors for hypothesis testing in the area.
Bayesian hypothesis testing is investigated when the prior probabilities of the hypotheses, taken as a random vector, are quantized. Nearest neighbor and centroid conditions are derived using mean Bayes risk error as a distortion measure for quantization. A high-resolution approximation to the distortion-rate function is also obtained. Human decision making in segregated populations is studied assuming Bayesian hypothesis testing with quantized priors.
We study a variation of Bayesian M-ary hypothesis testing in which the test outputs a list of L candidates out of the M possible upon processing the observation. We study the minimum error probability of list hypothesis testing, where an error is defined as the event where the true hypothesis is not in the list output by the test. We derive two exact expressions of the minimum probability or error. The first is expressed as the error probability of a certain non-Bayesian binary hypothesis test, and is reminiscent of the meta-converse bound. The second, is expressed as the tail probability of t…
We propose a simple robust hypothesis test that has the same sample complexity as that of the optimal Neyman-Pearson test up to constants, but robust to distribution perturbations under Hellinger distance. We discuss the applicability of such a robust test for estimating distributions in Hellinger distance. We empirically demonstrate the power of the test on canonical distributions.
We study a class of distributed hypothesis testing against conditional independence problems. Under the criterion that stipulates minimization of the Type II error rate subject to a (constant) upper bound $ε$ on the Type I error rate, we characterize the set of encoding rates and exponent for both discrete memoryless and memoryless vector Gaussian settings.
We interpret likelihood-based test functions from a geometric perspective where the Kullback-Leibler (KL) divergence is adopted to quantify the distance from a distribution to another. Such a test function can be seen as a sub-Gaussian random variable, and we propose a principled way to calculate its corresponding sub-Gaussian norm. Then an error bound for binary hypothesis testing can be obtained in terms of the sub-Gaussian norm and the KL divergence, which is more informative than Pinsker's bound when the significance level is prescribed. For $M$-ary hypothesis testing, we also derive an er…
We consider a hypothesis testing problem where a part of data cannot be observed. Our helper observes the missed data and can send us a limited amount of information about them. What kind of this limited information will allow us to make the best statistical inference? In particular, what is the minimum information sufficient to obtain the same results as if we directly observed all the data? We derive estimates for this minimum information and some other similar results.
The classical binary hypothesis testing problem is revisited. We notice that when one of the hypotheses is composite, there is an inherent difficulty in defining an optimality criterion that is both informative and well-justified. For testing in the simple normal location problem (that is, testing for the mean of multivariate Gaussians), we overcome the difficulty as follows. In this problem there exists a natural hardness order between parameters as for different parameters the error-probailities curves (when the parameter is known) are either identical, or one dominates the other. We can thu…
In this paper, we consider the hypothesis testing of correlation between two $m$-uniform hypergraphs on $n$ unlabelled nodes. Under the null hypothesis, the hypergraphs are independent, while under the alternative hypothesis, the hyperdges have the same marginal distributions as in the null hypothesis but are correlated after some unknown node permutation. We focus on two scenarios: the hypergraphs are generated from the Gaussian-Wigner model and the dense Erdös-Rényi model. We derive the sharp information-theoretic testing threshold. Above the threshold, there exists a powerful test to distin…
Two variational formulas for the power of the binary hypothesis testing problem are derived. The first is given as the Legendre transform of a certain function and the second, induced from the first, is given in terms of the Cumulative Distribution Function (CDF) of the log-likelihood ratio. One application of the first formula is an upper bound on the power of the binary hypothesis testing problem in terms of the Re'nyi divergence. The second formula provide a general framework for proving asymptotic and non-asymptotic expressions for the power of the test utilizing corresponding expressions …
This letter presents a novel Block Bayesian Hypothesis Testing Algorithm (Block-BHTA) for reconstructing block sparse signals with unknown block structures. The Block-BHTA comprises the detection and recovery of the supports, and the estimation of the amplitudes of the block sparse signal. The support detection and recovery is performed using a Bayesian hypothesis testing. Then, based on the detected and reconstructed supports, the nonzero amplitudes are estimated by linear MMSE. The effectiveness of Block-BHTA is demonstrated by numerical experiments.
It is observed that for testing between simple hypotheses where the cost of Type I and Type II errors can be quantified, it is better to let the optimization choose the test size.
This paper surveys some recent developments in fundamental limits and optimal algorithms for network analysis. We focus on minimax optimal rates in three fundamental problems of network analysis: graphon estimation, community detection, and hypothesis testing. For each problem, we review state-of-the-art results in the literature followed by general principles behind the optimal procedures that lead to minimax estimation and testing. This allows us to connect problems in network analysis to other statistical inference problems from a general perspective.
We provide necessary and sufficient conditions of uniform consistency of nonparametric sets of alternatives of chi-squared test for testing of hypothesis of homogeneity. The number of cells of chi-squared test increases with sample size growth. Nonparametric sets of alternatives can be defined both in terms of densities and distribution functions.
We consider the problem of hypothesis testing in the situation where the first hypothesis is simple and the second one is local one-sided composite. We describe the choice of the thresholds and the power functions of different tests when the intensity function of the observed inhomogeneous Poisson process has two different types of singularity: cusp and discontinuity. The asymptotic results are illustrated by numerical simulations.
The power of multiple testing procedures can be increased by using weighted p-values (Genovese, Roeder and Wasserman 2005). We derive the optimal weights and we show that the power is remarkably robust to misspecification of these weights. We consider two methods for choosing weights in practice. The first, external weighting, is based on prior information. The second, estimated weighting, uses the data to choose weights.
We treat quantum counterparts of testing problems whose optimal tests are given by chi-square, t and F tests. These quantum counterparts are formulated as quantum hypothesis testing problems concerning quantum Gaussian states families, and contain disturbance parameters, which have group symmetry. Quantum Hunt-Stein Theorem removes a part of these disturbance parameters, but other types of difficulty still remain. In order to remove them, combining quantum Hunt-Stein theorem and other reduction methods, we establish a general reduction theorem that reduces a complicated quantum hypothesis test…
Harvard John A. Paulson School Of Engineering And Applied Sciences（哈佛大学约翰·A·保尔森工程与应用科学学院）；Georgia Institute of Technology（佐治亚理工学院）；H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology（佐治亚理工学院H·米尔顿·斯图尔特工业与系统工程学院）；Department of Industrial Engineering and Operations Research, Columbia University（哥伦比亚大学工业工程与运筹学系）
The sequential hypothesis testing problem is a class of statistical analyses where the sample size is not fixed in advance. Instead, the decision-process takes in new observations sequentially to make real-time decisions for testing an alternative hypothesis against a null hypothesis until some stopping criterion is satisfied. In many common applications of sequential hypothesis testing, the data can be highly sensitive and may require privacy protection; for example, sequential hypothesis testing is used in clinical trials, where doctors sequentially collect data from patients and must determ…
We consider the problem of hypothesis testing in the situation when the first hypothesis is simple and the second one is local one-sided composite. We describe the choice of the thresholds and the power functions of the Score Function test, of the General Likelihood Ratio test, of the Wald test and of two Bayes tests in the situation when the intensity function of the observed inhomogeneous Poisson process is smooth with respect to the parameter. It is shown that almost all these tests are asymptotically uniformly most powerful. The results of numerical simulations are presented.
We discuss an "operational" approach to testing convex composite hypotheses when the underlying distributions are heavy-tailed. It relies upon Euclidean separation of convex sets and can be seen as an extension of the approach to testing by convex optimization developed in [8, 12]. In particular, we show how one can construct quasi-optimal testing procedures for families of distributions which are majorated, in a certain precise sense, by a sub-spherical symmetric one and study the relationship between tests based on Euclidean separation and "potential-based tests." We apply the promoted metho…
The likelihood ratio statistic, with its asymptotic $χ^2$ distribution at regular model points, is often used for hypothesis testing. At model singularities and boundaries, however, the asymptotic distribution may not be $χ^2$, as highlighted by recent work of Drton. Indeed, poor behavior of a $χ^2$ for testing near singularities and boundaries is apparent in simulations, and can lead to conservative or anti-conservative tests. Here we develop a new distribution designed for use in hypothesis testing near singularities and boundaries, which asymptotically agrees with that of the likelihood rat…
Although there are many ideas for the formulations of statistical hypothesis testing, we consider that the likelihood ratio test is the most reasonable and orthodox. However, it is not handy, and thus, it is not usual in elementary books. That is, the statistical hypothesis testing written in elementary books is different from the likelihood ratio test. Thus, from the theoretical point of view, we have the following question: "What is the statistical hypothesis testing written in elementary books?" For example, we consider that even the difference between "one sided test" and "two sided test" …
This study derives a new property of the Wishart distribution when the degree-of-freedom and the size of the matrix parameter of the distribution grow simultaneoulsy. Particularly, the asymptotic normality of the product of four independent Wishart matrices is shown under a high dimensional asymptotic regime. As an application of the result, a statistical test procedure for the common principal components hypothesis is proposed. For this problem, the proposed test statistic is asymptotically normal under the null hypothesis. In addition, the proposed test statistic diverges to positive infinit…
We discuss a general approach to handling "multiple hypotheses" testing in the case when a particular hypothesis states that the vector of parameters identifying the distribution of observations belongs to a convex compact set associated with the hypothesis. With our approach, this problem reduces to testing the hypotheses pairwise. Our central result is a test for a pair of hypotheses of the outlined type which, under appropriate assumptions, is provably nearly optimal. The test is yielded by a solution to a convex programming problem, so that our construction admits computationally efficient…
Indian Institute of Science（印度科学学院）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）；MPI-SWS（马克斯·普朗克软件系统研究所）
We study nonzero-sum hypothesis testing games that arise in the context of adversarial classification, in both the Bayesian as well as the Neyman-Pearson frameworks. We first show that these games admit mixed strategy Nash equilibria, and then we examine some interesting concentration phenomena of these equilibria. Our main results are on the exponential rates of convergence of classification errors at equilibrium, which are analogous to the well-known Chernoff-Stein lemma and Chernoff information that describe the error exponents in the classical binary hypothesis testing problem, but with pa…
In the classical two-sample problem, the conventional approach for testing distributions equality is based on the difference between the two marginal empirical distribution functions, whereas a test for independence is based on the contrast between the bivariate and the product of the marginal empirical distribution functions. In this article we consider the problem of testing independence and distributions equality when the observer is "colour blind" so he cannot distinguish the distribution which has generated each of the two measurements. Within a nonparametric framework, we propose an empi…
In this paper, we further develop the approach, originating in [GJN], to "computation-friendly" hypothesis testing via Convex Programming. Most of the existing results on hypothesis testing aim to quantify in a closed analytic form separation between sets of distributions allowing for reliable decision in precisely stated observation models. In contrast to this descriptive (and highly instructive) traditional framework, the approach we promote here can be qualified as operational -- the testing routines and their risks are yielded by an efficient computation. All we know in advance is that, un…
Hypothesis testing is a useful statistical tool in determining whether a given model should be rejected based on a sample from the population. Sample data may contain sensitive information about individuals, such as medical information. Thus it is important to design statistical tests that guarantee the privacy of subjects in the data. In this work, we study hypothesis testing subject to differential privacy, specifically chi-squared tests for goodness of fit for multinomial data and independence between two categorical variables. We propose new tests for goodness of fit and independence testi…
Motivated by gene set enrichment analysis, we investigate the problem of combined hypothesis testing on a graph. We introduce a general framework to effectively use the structural information of the underlying graph when testing multivariate means. A new testing procedure is proposed within this framework. We show that the test is optimal in that it can consistently detect departure from the collective null at a rate that no other test could improve, for almost all graphs. We also provide general performance bounds for the proposed test under any specific graph, and illustrate their utility th…
Multiple hypothesis testing often involves composite nulls, i.e., nulls that are associated with two or more distributions. In many cases, it is reasonable to assume that there is a prior distribution on the distributions despite it is unknown. When the number of distributions under true nulls is finite, we show that under the above assumption, the false discover rate (FDR) can be controlled using $p$-values computed under constraints imposed by the empirical distribution of the observations. Comparing to FDR control using $p$-values defined as maximum significance level over all null distribu…
The paper estimates the Chernoff rate for the efficiency of quantum hypothesis testing. For both joint and separable measurements, approximate bounds for the rate are given if both states are mixed and exact expressions are derived if at least one of the states is pure. The efficiency of tests with separable measurements is found to be close to the efficiency of tests with joint measurements. The results are illustrated by a test of quantum entanglement.
We address the following question in this paper: "What are the most robust statistical methods for social choice?'' By leveraging the theory of uniformly least favorable distributions in the Neyman-Pearson framework to finite models and randomized tests, we characterize uniformly most powerful (UMP) tests, which is a well-accepted statistical optimality w.r.t. robustness, for testing whether a given alternative is the winner under Mallows' model and under Condorcet's model, respectively.
We present a test for the problem of decentralized sequential hypothesis testing, which is asymptotically optimum. By selecting a suitable sampling mechanism at each sensor, communication between sensors and fusion center is asynchronous and limited to 1-bit data. The proposed SPRT-like test turns out to be order-2 asymptotically optimum in the case of continuous time and continuous path signals, while in discrete time this strong asymptotic optimality property is preserved under proper conditions. If these conditions do not hold, then we can show optimality of order-1. Simulations corroborate…
High-dimensional logistic regression is widely used in analyzing data with binary outcomes. In this paper, global testing and large-scale multiple testing for the regression coefficients are considered in both single- and two-regression settings. A test statistic for testing the global null hypothesis is constructed using a generalized low-dimensional projection for bias correction and its asymptotic null distribution is derived. A lower bound for the global testing is established, which shows that the proposed test is asymptotically minimax optimal over some sparsity range. For testing the in…
This paper focuses on hypothesis testing for the input of a Lévy-driven storage system by sampling of the storage level. As the likelihood is not explicit we propose two tests that rely on transformation of the data. The first approach uses i.i.d. `quasi-busy-periods' between observations of zero workload. The distribution of the duration of quasi-busy-periods is determined. The second method is a conditional likelihood ratio test based on the Bernoulli events of observing a zero or positive workload, conditional on the previous workload. Performance analysis is presented for both tests along …
We study the simple hypothesis testing problem for the drift coefficient for stochastic fractional heat equation driven by additive noise. We introduce the notion of asymptotically the most powerful test, and find explicit forms of such tests in two asymptotic regimes: large time asymptotics, and increasing number of Fourier modes. The proposed statistics are derived based on Maximum Likelihood Ratio. Additionally, we obtain a series of important technical results of independent interest: we find the cumulant generating function of the log-likelihood ratio; obtain sharp large deviation type re…
We consider a data-driven robust hypothesis test where the optimal test will minimize the worst-case performance regarding distributions that are close to the empirical distributions with respect to the Wasserstein distance. This leads to a new non-parametric hypothesis testing framework based on distributionally robust optimization, which is more robust when there are limited samples for one or both hypotheses. Such a scenario often arises from applications such as health care, online change-point detection, and anomaly detection. We study the computational and statistical properties of the p…
Hypothesis testing for small-sample scenarios is a practically important problem. In this paper, we investigate the robust hypothesis testing problem in a data-driven manner, where we seek the worst-case detector over distributional uncertainty sets centered around the empirical distribution from samples using Sinkhorn distance. Compared with the Wasserstein robust test, the corresponding least favorable distributions are supported beyond the training samples, which provides a more flexible detector. Various numerical experiments are conducted on both synthetic and real datasets to validate th…
The local model for differential privacy is emerging as the reference model for practical applications collecting and sharing sensitive information while satisfying strong privacy guarantees. In the local model, there is no trusted entity which is allowed to have each individual's raw data as is assumed in the traditional curator model for differential privacy. So, individuals' data are usually perturbed before sharing them. We explore the design of private hypothesis tests in the local model, where each data entry is perturbed to ensure the privacy of each participant. Specifically, we analyz…
The problem of multiple hypothesis testing arises when there are more than one hypothesis to be tested simultaneously for statistical significance. This is a very common situation in many data mining applications. For instance, assessing simultaneously the significance of all frequent itemsets of a single dataset entails a host of hypothesis, one for each itemset. A multiple hypothesis testing method is needed to control the number of false positives (Type I error). Our contribution in this paper is to extend the multiple hypothesis framework to be used with a generic data mining algorithm. We…
Rice University（莱斯大学）；Department of Economics（经济学系）
In this work we use Recurrent Neural Networks and Multilayer Perceptrons to predict NYSE, NASDAQ and AMEX stock prices from historical data. We experiment with different architectures and compare data normalization techniques. Then, we leverage those findings to question the efficient-market hypothesis through a formal statistical test.
The two-sample hypothesis testing problem is studied for the challenging scenario of high dimensional data sets with small sample sizes. We show that the two-sample hypothesis testing problem can be posed as a one-class set classification problem. In the set classification problem the goal is to classify a set of data points that are assumed to have a common class. We prove that the average probability of error given a set is less than or equal to the Bayes error and decreases as a power of $n$ number of sample data points in the set. We use the positive definite Set Kernel for directly mappin…
A vast amount of expert and domain knowledge is captured by causal structural priors, yet there has been little research on testing such priors for generalization and data synthesis purposes. We propose a novel model architecture, Causal Structural Hypothesis Testing, that can use nonparametric, structural causal knowledge and approximate a causal model's functional relationships using deep neural networks. We use these architectures for comparing structural priors, akin to hypothesis testing, using a deliberate (non-random) split of training and testing data. Extensive simulations demonstrate…
School of Mathematical and Computer Sciences, Heriot-Watt University（赫瑞斯泰学院数学与计算机科学系，赫瑞瓦特大学）；School of Mathematics, University of Edinburgh（爱丁堡大学数学学院）；Maxwell Institute for Mathematical Sciences（麦克斯韦数学科学研究所）
