Hypothesis testing is a useful statistical tool in determining whether a given model should be rejected based on a sample from the population. Sample data may contain sensitive information about individuals, such as medical information. Thus it is important to design statistical tests that guarantee the privacy of subjects in the data. In this work, we study hypothesis testing subject to differential privacy, specifically chi-squared tests for goodness of fit for multinomial data and independence between two categorical variables.
We propose new tests for goodness of fit and independence testing that like the classical versions can be used to determine whether a given model should be rejected or not, and that additionally can ensure differential privacy. We give both Monte Carlo based hypothesis tests as well as hypothesis tests that more closely follow the classical chi-squared goodness of fit test and the Pearson chi-squared test for independence. Crucially, our tests account for the distribution of the noise that is injected to ensure privacy in determining significance.
We show that these tests can be used to achieve desired significance levels, in sharp contrast to direct applications of classical tests to differentially private contingency tables which can result in wildly varying significance levels. Moreover, we study the statistical power of these tests. We empirically show that to achieve the same level of power as the classical non-private tests our new tests need only a relatively modest increase in sample size.
The problem of testing the goodness of fit of an absolutely continuous distribution to a set of observations grouped into equal probability intervals, against to a family of sequences of alternatives approaching the hypothesis is considered. We focus our study to the asymptotic intermediate efficiency (AIE) due to Inglot (1999) of the chi-squared and log-likelihood ratio tests. It is assumed that the number of groups increases together with the number of observations.
The Bayesian nonparametric inference and Dirichlet process are popular tools in statistical methodologies. In this paper, we employ the Dirichlet process in hypothesis testing to propose a Bayesian nonparametric chi-squared goodness-of-fit test. In our Bayesian nonparametric approach, we consider the Dirichlet process as the prior for the distribution of data and carry out the test based on the Kullback-Leibler distance between the updated Dirichlet process and the hypothesized distribution F0. We prove that this distance asymptotically converges to the same chi-squared distribution as the chi…
Randomized response is one of the oldest and most well-known methods for analyzing confidential data. However, its utility for differentially private hypothesis testing is limited because it cannot achieve high privacy levels and low type I error rates simultaneously. In this article, we show how to overcome this issue with the subsample and aggregate technique. The result is a general-purpose method that can be used for both frequentist and Bayesian testing. {{We illustrate the performance of our proposal in three scenarios: goodness-of-fit testing for linear regression models, nonparametric …
A self-contained Fortran-77 program for goodness of fit tests for histograms with weighted entries as well as with unweighted entries is presented. The code calculates test statistics for case of histogram with normalized weights of events and in case of unnormalized weights of events.
It is well known that the approximate distribution of the usual test statistic of a goodness-of-fit test is chi-square, with degrees of freedom equal to the number of categories minus 1 (assuming that no parameters are to be estimated -- something we do throughout this article). Here we show how to improve this approximation by including two correction terms, each of them inversely proportional to the total number of observations.
Weighted histograms are used for the estimation of probability density functions. Computer simulation is the main domain of application of this type of histogram. A review of chi-square goodness of fit tests for weighted histograms is presented in this paper. Improvements are proposed to these tests that have size more close to its nominal value. Numerical examples are presented in this paper for evaluation of tests and to demonstrate various applications of tests.
Taking the goodness of fit test (Chi test) as an example, this paper attempts to calculate the Bayesian factor BF10 of n-fold Bernoulli test by the Excel software (using JASP software as the evidence). The results showed that in the range of 0.15-0.55 (the rate of samples which are all "true"), the calculated results of Excel were more accurate, and the differences between the two (Excel and JASP) were not statistically significant (P>0.3).
Motivated by recent studies of big samples, this work aims at constructing a parametric model which is characterized by the following features: (i) a "local" reinforcement, i.e. a reinforcement mechanism mainly based on the last observations, (ii) a random persistent fluctuation of the predictive mean, and (iii) a long-term convergence of the empirical mean to a deterministic limit, together with a chi-squared goodness of fit result. This triple purpose has been achieved by the introduction of a new variant of the Eggenberger-Polya urn, that we call the "Rescaled" Polya urn. We provide a compl…
In this paper, we consider methods for performing hypothesis tests on data protected by a statistical disclosure control technology known as differential privacy. Previous approaches to differentially private hypothesis testing either perturbed the test statistic with random noise having large variance (and resulted in a significant loss of power) or added smaller amounts of noise directly to the data but failed to adjust the test in response to the added noise (resulting in biased, unreliable $p$-values). In this paper, we develop a variety of practical hypothesis tests that address these pro…
The chi square goodness-of-fit test is among the oldest known statistical tests, first proposed by Pearson in 1900 for the multinomial distribution. It has been in use in many fields ever since. However, various studies have shown that when applied to data from a continuous distribution it is generally inferior to other methods such as the Kolmogorov-Smirnov or Anderson-Darling tests. However, the performance, that is the power, of the chi square test depends crucially on the way the data is binned. In this paper we describe a method that automatically finds a binning that is very good against…
This paper is first devoted to study an adaptive wavelet based estimator of the long memory parameter for linear processes in a general semi-parametric frame. This is an extension of Bardet {\it et al.} (2008) which only concerned Gaussian processes. Moreover, the definition of the long memory parameter estimator is modified and asymptotic results are improved even in the Gaussian case. Finally an adaptive goodness-of-fit test is also built and easy to be employed: it is a chi-square type test. Simulations confirm the interesting properties of consistency and robustness of the adaptive estimat…
In this paper, we compare the powers of several discrete goodness-of-fit test statistics considered by Steele and Chaseling [10] under the null hypothesis of a 'zig-zag' distribution. The results suggest that the Discrete Kolmogorov-Smirnov test statistic is generally more powerful for the decreasing trend alternative. The Pearson Chi-Square statistic is generally more powerful for the increasing, unimodal, leptokurtic, platykurtic and bath-tub shaped alternatives. Finally, both the Nominal Kolmogorov- Smirnov and the Pearson Chi-Square test statistic are generally more powerful for the bimoda…
Pearson's chi-squared test is widely used to test the goodness of fit between categorical data and a given discrete distribution function. When the number of sets of the categorical data, say $k$, is a fixed integer, Pearson's chi-squared test statistic converges in distribution to a chi-squared distribution with $k-1$ degrees of freedom when the sample size $n$ goes to infinity. In real applications, the number $k$ often changes with $n$ and may be even much larger than $n$. By using the martingale techniques, we prove that Pearson's chi-squared test statistic converges to the normal under qu…
We consider a multivariable functional errors-in-variables model $AX\approx B$, where the data matrices $A$ and $B$ are observed with errors, and a matrix parameter $X$ is to be estimated. A goodness-of-fit test is constructed based on the total least squares estimator. The proposed test is asymptotically chi-squared under null hypothesis. The power of the test under local alternatives is discussed.
We consider the goodness of fit testing problem for stochastic differential equation with small diffiusion coefficient. The basic hypothesis is always simple and it is described by the known trend coefficient. We propose several tests of the type of Cramer-von Mises, Kolmogorov-Smirnov and Chi-Square. The power functions of these tests we study for a special classes of close alternatives. We discuss the construction of the goodness of fit test based on the local time and the possibility of the construction of asymptotically distribution free tests in the case of composite basic hypothesis.
We consider goodness-of-fit tests for uniformity of a multinomial distribution by means of tests based on a class of symmetric statistics, defined as the sum of some function of cell-frequencies. We are dealing with an asymptotic regime, where the number of cells grows with the sample size. Most attention is focused on the class of power divergence statistics. The aim of this article is to study the intermediate asymptotic relative efficiency of two tests, where the powers of the tests are asymptotically non-degenerate and the sequences of alternatives converge to the hypothesis, but not too f…
We consider multinomial goodness-of-fit tests in the high-dimensional regime where the number of bins increases with the sample size. In this regime, Pearson's chi-squared test can suffer from low power due to the substantial bias as well as high variance of its statistic. To resolve these issues, we introduce a family of U-statistic for multinomial goodness-of-fit and study their asymptotic behaviors in high-dimensions. Specifically, we establish conditions under which the considered U-statistic is asymptotically Poisson or Gaussian, and investigate its power function under each asymptotic re…
Despite a substantial literature on nonparametric two-sample goodness-of-fit testing in arbitrary dimensions spanning decades, there is no mention there of any curse of dimensionality. Only more recently Ramdas et al. (2015) have discussed this issue in the context of kernel methods by showing that their performance degrades with the dimension even when the underlying distributions are isotropic Gaussians. We take a minimax perspective and follow in the footsteps of Ingster (1987) to derive the minimax rate in arbitrary dimension when the discrepancy is measured in the L2 metric. That rate is …
In this paper, we develop new test statistics for private hypothesis testing. These statistics are designed specifically so that their asymptotic distributions, after accounting for noise added for privacy concerns, match the asymptotics of the classical (non-private) chi-square tests for testing if the multinomial data parameters lie in lower dimensional manifolds (examples include goodness of fit and independence testing). Empirically, these new test statistics outperform prior work, which focused on noisy versions of existing statistics.
Statistical data is often analyzed as a contingency table, sometimes with empty cells called zeros. Such sparse tables can be due to scarse observations classified in numerous categories, as for example in genetic association studies. Thus, classical independence tests involving Pearson's chi-square statistic Q or Kullback's minimum discrimination information statistic G cannot be applied because some of the expected frequencies are too small. More generally, we consider goodness of fit tests with composite hypotheses for sparse multinomial vectors and suggest simple corrections for Q and G th…
Statistical data is often analyzed as a contingency table, sometimes with empty cells called zeros. Such sparse tables can be due to scarse observations classified in numerous categories, as for example in genetic association studies. Thus, classical independence tests involving Pearson's chi-square statistic Q or Kullback's minimum discrimination information statistic G cannot be applied because some of the expected frequencies are too small. More generally, we consider goodness of fit tests with composite hypotheses for sparse multinomial vectors and suggest simple corrections for Q and G th…
The object of study is the problem of testing for uniformity of the multinomial distribution. We consider tests based on symmetric statistics, defined as the sum of some function of cell-frequencies. Mainly, attention is focused on the class of power divergence statistics, in particular, on the chi-square and log-likelihood ratio statistics. The main issue of the article is to study the asymptotic properties of tests at the concept of an intermediate setting in terms of so called -intermediate asymptotic efficiency due to Ivchenko and Mirakhmedov (1995), when the asymptotic power of tests are …
The Newcomb-Benford probability distribution is becoming very popular in many areas using statistics, notably in fraud detection. In such contexts, it is important to be able to determine if a data set arises from this distribution while controlling the risk of a Type 1 error, i.e. falsely identifying a fraud, and a Type 2 error, i.e. not detecting that a fraud occurred. The statistical tool to do this work is a goodness-of-fit test. For the Newcomb-Benford distribution, the most popular such test is Pearson's chi-square test whose power, related to the Type 2 error, is known to be weak. Conse…
We consider goodness-of-fit tests with i.i.d. samples generated from a categorical distribution $(p_1,...,p_k)$. For a given $(q_1,...,q_k)$, we test the null hypothesis whether $p_j=q_{π(j)}$ for some label permutation $π$. The uncertainty of label permutation implies that the null hypothesis is composite instead of being singular. In this paper, we construct a testing procedure using statistics that are defined as indefinite integrals of some symmetric polynomials. This method is aimed directly at the invariance of the problem, and avoids the need of matching the unknown labels. The asymptot…
We find asymptotic formulas for error probabilities of two-fold Pearson goodness-of-fit test as functions of two critical levels. These results may be reformulated in terms of tails of two-dimensional distributions of the Bessel process. Necessary properties of the Infeld function are obtained.
The problem of curve registration appears in many different areas of applications ranging from neuroscience to road traffic modeling. In the present work, we propose a nonparametric testing framework in which we develop a generalized likelihood ratio test to perform curve registration. We first prove that, under the null hypothesis, the resulting test statistic is asymptotically distributed as a chi-squared random variable. This result, often referred to as Wilks' phenomenon, provides a natural threshold for the test of a prescribed asymptotic significance level and a natural measure of lack-o…
The local model for differential privacy is emerging as the reference model for practical applications collecting and sharing sensitive information while satisfying strong privacy guarantees. In the local model, there is no trusted entity which is allowed to have each individual's raw data as is assumed in the traditional curator model for differential privacy. So, individuals' data are usually perturbed before sharing them. We explore the design of private hypothesis tests in the local model, where each data entry is perturbed to ensure the privacy of each participant. Specifically, we analyz…
Chi-squared tests for lack of fit are traditionally employed to find evidence against a hypothesized model, with the model accepted if the Karl Pearson statistic comparing observed and expected numbers of observations falling within cells is not significantly large. However, if one really wants evidence for goodness of fit, it is better to adopt an equivalence testing approach in which small values of the chi-squared statistic are evidence for the desired model. This method requires one to define what is meant by equivalence to the desired model, and guidelines are proposed. Then a simple exte…
This paper is an extension of the work about the exponential increase of the power of two non-parametric tests: the $ Z $-test and the chi-square goodness-of-fit test. Subject to having auxiliary information, it is possible to improve exponentially relative to the size of the sample the power of the famous chi-square tests of independence and homogeneity. Improving the power of these statistical tests by using auxiliary information makes it possible either to reduce the probability of accepting the null hypothesis under the alternative hypothesis, or to reduce the size of the sample necessary …
Coarse Structural Nested Mean Models (SNMMs) provide useful tools to estimate treatment effects from longitudinal observational data with time-dependent confounders. Coarse SNMMs lead to a large class of estimators,within which an optimal estimator can be derived under the conditions of well-specified models for the treatment effect, for treatment initiation, and for nuisance regression outcomes (Lok & Griner, 2015). The key assumption lies in a well-specified model for the treatment effect; however, there is no existing guidance to specify the treatment effect model, and model misspecificatio…
Density-based directed distances -- particularly known as divergences -- between probability distributions are widely used in statistics as well as in the adjacent research fields of information theory, artificial intelligence and machine learning. Prominent examples are the Kullback-Leibler information distance (relative entropy) which e.g. is closely connected to the omnipresent maximum likelihood estimation method, and Pearson's chisquare-distance which e.g. is used for the celebrated chisquare goodness-of-fit test. Another line of statistical inference is built upon distribution-function-b…
We propose a kernel machine based hypothesis testing procedure in nonlinear function-on-scalar regression model. Our research is motivated by the Newborn Epigenetic Study (NEST) where the question of interest is whether a pre-specified group of toxic metals or methylation at any of 9 differentially methylated regions (DMRs) is associated with child growth. We take the child growth trajectory as the functional response, and model the toxic metal measurements jointly using a nonlinear function. We use a kernel machine approach to model the unknown function and transform the hypothesis of no effe…
We propose a new class of goodness-of-fit tests for the logistic distribution based on a characterisation related to the density approach in the context of Stein's method. This characterisation based test is a first of its kind for the logistic distribution. The asymptotic null distribution of the test statistic is derived and it is shown that the test is consistent against fixed alternatives. The finite sample power performance of the newly proposed class of tests is compared to various existing tests by means of a Monte Carlo study. It is found that this new class of tests are especially pow…
NIST SP800-22 is one of the most widely used statistical testing tools for pseudorandom number generators (PRNGs). This tool consists of 15 tests (one-level tests) and two additional tests (two-level tests). Each one-level test provides one or more $p$-values. The two-level tests measure the uniformity of the obtained $p$-values for a fixed one-level test. One of the two-level tests categorizes the $p$-values into ten intervals of equal length, and apply a chi-squared goodness-of-fit test. This two-level test is often more powerful than one-level tests, but sometimes it rejects even good PRNGs…
We propose a goodness-of-fit test for degree-corrected stochastic block models (DCSBM). The test is based on an adjusted chi-square statistic for measuring equality of means among groups of $n$ multinomial distributions with $d_1,\dots,d_n$ observations. In the context of network models, the number of multinomials, $n$, grows much faster than the number of observations, $d_i$, corresponding to the degree of node $i$, hence the setting deviates from classical asymptotics. We show that a simple adjustment allows the statistic to converge in distribution, under null, as long as the harmonic mean …
The raking-ratio method is a statistical and computational method which adjusts the empirical measure to match the true probability of sets of a finite partition. We study the asymptotic behavior of the raking-ratio empirical process indexed by a class of functions when the auxiliary information is given by estimates. We suppose that these estimates result from the learning of the probability of sets of partitions from another sample larger than the sample of the statistician, as in the case of two-stage sampling surveys. Under some metric entropy hypothesis and conditions on the size of the i…
A multivariate distribution function F is in the max-domain of attraction of an extreme value distribution if and only if this is true for the copula corresponding to F and its univariate margins. Aulbach et al. (2012a) have shown that a copula satisfies the extreme value condition if and only if the copula is tail equivalent to a generalized Pareto copula (GPC). In this paper we propose a chi-square goodness-of-fit test in arbitrary dimension for testing whether a copula is in a certain neighborhood of a GPC. The test can be applied to stochastic processes as well to check whether the corresp…
Local differential privacy has recently received increasing attention from the statistics community as a valuable tool to protect the privacy of individual data owners without the need of a trusted third party. Similar to the classical notion of randomized response, the idea is that data owners randomize their true information locally and only release the perturbed data. Many different protocols for such local perturbation procedures can be designed. In most estimation problems studied in the literature so far, however, no significant difference in terms of minimax risk between purely non-inte…
If a discrete probability distribution in a model being tested for goodness-of-fit is not close to uniform, then forming the Pearson chi-square statistic can involve division by nearly zero. This often leads to serious trouble in practice -- even in the absence of round-off errors -- as the present article illustrates via numerous examples. Fortunately, with the now widespread availability of computers, avoiding all the trouble is simple and easy: without the problematic division by nearly zero, the actual values taken by goodness-of-fit statistics are not humanly interpretable, but black-box …
We consider the goodness-of-fit testing problem of distinguishing whether the data are drawn from a specified distribution, versus a composite alternative separated from the null in the total variation metric. In the discrete case, we consider goodness-of-fit testing when the null distribution has a possibly growing or unbounded number of categories. In the continuous case, we consider testing a Lipschitz density, with possibly unbounded support, in the low-smoothness regime where the Lipschitz parameter is not assumed to be constant. In contrast to existing results, we show that the minimax r…
We provide evidence that a root-mean-square test of goodness-of-fit can be significantly more powerful than state-of-the-art exact tests in detecting deviations from Hardy-Weinberg equilibrium. Unlike Pearson's chi-square test, the log--likelihood-ratio test, and Fisher's exact test, which are sensitive to relative discrepancies between genotypic frequencies, the root-mean-square test is sensitive to absolute discrepancies. This can increase statistical power, as we demonstrate using benchmark datasets and through asymptotic analysis. With the aid of computers, exact P-values for the root-mean…
In this article, the weighted empirical likelihood is applied to a general setting of two-sample semiparametric models, which includes biased sampling models and case-control logistic regression models as special cases. For various types of censored data, such as right censored data, doubly censored data, interval censored data and partly interval-censored data, the weighted empirical likelihood-based semiparametric maximum likelihood estimator $(\tildeθ_n,\tilde{F}_n)$ for the underlying parameter $θ_0$ and distribution $F_0$ is derived, and the strong consistency of $(\tildeθ_n,\tilde{F}_n)$…
This paper focuses on inhomogeneous quadratic tests, which involve the sum of a dependent non-central chi-square with a Gaussian random variable. Unfortunately, no closed-form expression is available for the statistical distribution of the resulting random variable, thus hindering the analytical characterization of these tests in terms of probability of detection and probability of false alarm. In order to circumvent this limitation, two closed-form approximations are proposed in this work based on results from Edgeworth series expansions and Extreme Value Theory (EVT). The use of these approx…
We carry out a study of the statistical distribution of rainfall precipitation data for 20 cites in India. We have determined the best-fit probability distribution for these cities from the monthly precipitation data spanning 100 years of observations from 1901 to 2002. To fit the observed data, we considered 10 different distributions. The efficacy of the fits for these distributions was evaluated using four empirical non-parametric goodness-of-fit tests namely Kolmogorov-Smirnov, Anderson-Darling, Chi-Square, Akaike information criterion, and Bayesian Information criterion. Finally, the best…
