Experimental research on behavior and cognition frequently rests on stimulus or subject selection where not all characteristics can be fully controlled, even when attempting strict matching. For example, when contrasting patients to controls, variables such as intelligence or socioeconomic status are often correlated with patient status. Similarly, when presenting word stimuli, variables such as word frequency are often correlated with primary variables of interest. One procedure very commonly employed to control for such nuisance effects is conducting inferential tests on confounding stimulus or subject characteristics. For example, if word length is not significantly different for two stimulus sets, they are considered as matched for word length. Such a test has high error rates and is conceptually misguided. It reflects a common misunderstanding of statistical tests: interpreting significance not to refer to inference about a particular population parameter, but about 1. the sample in question, 2. the practical relevance of a sample difference (so that a nonsignificant test is taken to indicate evidence for the absence of relevant differences). We show inferential testing for assessing nuisance effects to be inappropriate both pragmatically and philosophically, present a survey showing its high prevalence, and briefly discuss an alternative in the form of regression including nuisance variables.
A framework for estimation and hypothesis testing of functional restrictions against general alternatives is proposed. The parameter space is a reproducing kernel Hilbert space (RKHS). The null hypothesis does not necessarily define a parametric model. The test allows us to deal with infinite dimensional nuisance parameters. The methodology is based on a moment equation similar in spirit to the construction of the efficient score in semiparametric statistics. The feasible version of such moment equation requires to consistently estimate projections in the space of RKHS and it is shown that thi…
Bootstrap test procedure for variance components in nonlinear mixed effects models in the presence of nuisance parameters and a singular Fisher Information Matrix
Hypothesis testing results often rely on simple, yet important assumptions about the behaviour of the distribution of p-values under the null and the alternative. We examine tests for one dimensional parameters of interest that converge to a normal distribution, possibly in the presence of nuisance parameters, and characterize the distribution of the p-values using techniques from the higher order asymptotics literature. We show that commonly held beliefs regarding the distribution of p-values are misleading when the variance and location of the test statistic are not well-calibrated or when t…
Testing for point-impact effects in a spatial semi-functional linear regression model
Laboratoire de Probabilités, Statistique et Informatique（概率、统计与信息实验室）；Unité de Recherche en Mathématiques et Informatique（数学与信息技术研究单位）；Université des Sciences et Techniques de Masuku（马苏库科学技术大学）
As a specific proportional hazard rates model, sequential order statistics can be used to describe the lifetimes of load-sharing systems. Inference for these systems needs to account for small sample sizes, which are prevalent in reliability applications. By exploiting the probabilistic structure of sequential order statistics, we derive exact finite sample inference procedures to test for the load-sharing parameters and for the nonparametrically specified baseline distribution, treating the respective other part as a nuisance quantity. This improves upon previous approaches for the model, whi…
We present a new test when there is a nuisance parameter under the alternative hypothesis. The test exploits the p-value occupation time [PVOT], the measure of the nuisance parameter subset on which a p-value test based on a a test statistic rejects the null hypothesis. Key contributions are: (i) An asymptotic critical value upper bound for our test is the significance level α, making inference easy. (ii) We only require the test statistic to have a known or bootstrappable limit distribution, hence we do not require root(n)-Gaussian asymptotics, allowing for weak or non-identification, boundar…
Importance sampling is a common technique for Monte Carlo approximation, including Monte Carlo approximation of p-values. Here it is shown that a simple correction of the usual importance sampling p-values creates valid p-values, meaning that a hypothesis test created by rejecting the null when the p-value is <= alpha will also have a type I error rate <= alpha. This correction uses the importance weight of the original observation, which gives valuable diagnostic information under the null hypothesis. Using the corrected p-values can be crucial for multiple testing and also in problems where …
The goal of the paper is to develop a specific application of the convex optimization based hypothesis testing techniques developed in A. Juditsky, A. Nemirovski, "Hypothesis testing via affine detectors," Electronic Journal of Statistics 10:2204--2242, 2016. Namely, we consider the Change Detection problem as follows: given an evolving in time noisy observations of outputs of a discrete-time linear dynamical system, we intend to decide, in a sequential fashion, on the null hypothesis stating that the input to the system is a nuisance, vs. the alternative stating that the input is a "nontrivia…
A Transferability Criterion for Null-Optimized Variance Reduction in Cumulant-Based Error-Independence Testing
Cherkasy State Business College（切尔卡瑟州立商业学院）；State Scientific Research Institute of Armament and Military Equipment Testing and Certification（国家武器装备测试与认证科学研究院）；Uzhhorod National University（乌日霍罗德国立大学）
A number of biomedical problems require performing many hypothesis tests, with an attendant need to apply stringent thresholds. Often the data take the form of a series of predictor vectors, each of which must be compared with a single response vector, perhaps with nuisance covariates. Parametric tests of association are often used, but can result in inaccurate type I error at the extreme thresholds, even for large sample sizes. Furthermore, standard two-sided testing can reduce power compared to the doubled $p$-value, due to asymmetry in the null distribution. Exact (permutation) testing is a…
After variable selection, standard inferential procedures for regression parameters may not be uniformly valid; there is no finite-sample size at which a standard test is guaranteed to approximately attain its nominal size. This problem is exacerbated in high-dimensional settings, where variable selection becomes unavoidable. This has prompted a flurry of activity in developing uniformly valid hypothesis tests for a low-dimensional regression parameter (e.g. the causal effect of an exposure A on an outcome Y) in high-dimensional models. So far there has been limited focus on model misspecifica…
Linear mixed-effects models are widely used in analyzing repeated measures data, including clustered and longitudinal data, where inferences of both fixed effects and variance components are of importance. Unlike the fixed effect inference that has been well studied, inference on the variance components is more challenging due to null value being on the boundary and the nuisance parameters of the fixed effects. Existing methods often require strong distributional assumptions on the random effects and random errors. In this paper, we develop empirical likelihood-based methods for the inference …
This research creates a general class of "perturbation models" which are described by an underlying "null" model that accounts for most of the structure in data and a perturbation that accounts for possible small localized departures. The perturbation models encompass finite mixture models and spatial scan process. In this article, (1) we propose a new test statistic to detect the presence of perturbation, including the case where the null model contains a set of nuisance parameters, and show that it is equivalent to the likelihood ratio test; (2) we establish that the asymptotic distribution …
A CUSUM type test for constant correlation that goes beyond a previously suggested correlation constancy test by considering Spearman's rho in arbitrary dimensions is proposed. Since the new test does not require the existence of any moments, the applicability on usually heavy-tailed financial data is greatly improved. The asymptotic null distribution is calculated using an invariance principle for the sequential empirical copula process. The limit distribution is free of nuisance parameters and critical values can be obtained without bootstrap techniques. A local power result and an analysis …
We propose a likelihood ratio statistic for forming hypothesis tests and confidence intervals for a nonparametrically estimated univariate regression function, based on the shape restriction of concavity (alternatively, convexity). Dealing with the likelihood ratio statistic requires studying an estimator satisfying a null hypothesis, that is, studying a concave least-squares estimator satisfying a further equality constraint. We study this null hypothesis least-squares estimator (NLSE) here, and use it to study our likelihood ratio statistic. The NLSE is the solution to a convex program, and …
A storm of favorable or critical publications regarding p-values-based procedures has been observed in both the theoretical and applied literature. We focus on valid definitions of p-values in the scenarios when composite null models are in effect. A valid p-value (VpV) statistic can be used to make a prefixed level-decision. In this context, Kolmogorov Smirnov goodness-of-fit tests and the normal two sample problem are considered. In particular, we examine an issue regarding the goodness-of-fit testability based on a single observation. This article exemplifies constructions of new test proce…
We consider large-scale studies in which thousands of significance tests are performed simultaneously. In some of these studies, the multiple testing procedure can be severely biased by latent confounding factors such as batch effects and unmeasured covariates that correlate with both primary variable(s) of interest (e.g. treatment variable, phenotype) and the outcome. Over the past decade, many statistical methods have been proposed to adjust for the confounders in hypothesis testing. We unify these methods in the same framework, generalize them to include multiple primary variables and multi…
In this paper, we propose a test for the equality of multiple distributions based on kernel mean embeddings. Our framework provides a flexible way to handle multivariate or even high-dimensional data by virtue of kernel methods and allows the number of distributions to increase with the sample size. This is in contrast to previous studies that have been mostly restricted to classical low-dimensional settings with a fixed number of distributions. By building on Cramer-type moderate deviation for degenerate two-sample V-statistics, we derive the limiting null distribution of the test statistic a…
Harvard T. H. Chan School of Public Health（哈佛大学陈曾熙公共卫生学院）
For many causal effect parameters of interest, doubly robust machine learning (DRML) estimators $\hatψ_{1}$ are the state-of-the-art, incorporating the good prediction performance of machine learning; the decreased bias of doubly robust estimators; and the analytic tractability and bias reduction of sample splitting with cross fitting. Nonetheless, even in the absence of confounding by unmeasured factors, the nominal $(1 - α)$ Wald confidence interval $\hatψ_{1} \pm z_{α/ 2} \widehat{\mathsf{se}} [\hatψ_{1}]$ may still undercover even in large samples, because the bias of $\hatψ_{1}$ may be of…
I propose two U-statistics to test coefficients in generalized linear models. One of them is used to deal with global hypothesis and the other one to test with the nuisance parameter. Both the statistics proposed are within high-dimensional setting which means the number of coefficients is much larger than the sample size. The statistics are based on quasi-likelihood function so that they have wilder applications. I theoretically analyze the asymptotic distribution of the statistics under the null hypothesis and the power functions under the local and fixed alternatives. To serve as a comparis…
We investigate hypothesis testing in nonparametric additive models estimated using simplified smooth backfitting (Huang and Yu, Journal of Computational and Graphical Statistics, \textbf{28(2)}, 386--400, 2019). Simplified smooth backfitting achieves oracle properties under regularity conditions and provides closed-form expressions of the estimators that are useful for deriving asymptotic properties. We develop a generalized likelihood ratio (GLR) and a loss function (LF) based testing framework for inference. Under the null hypothesis, both the GLR and LF tests have asymptotically rescaled ch…
We consider the problem of uncertainty assessment for low dimensional components in high dimensional models. Specifically, we propose a decorrelated score function to handle the impact of high dimensional nuisance parameters. We consider both hypothesis tests and confidence regions for generic penalized M-estimators. Unlike most existing inferential methods which are tailored for individual models, our approach provides a general framework for high dimensional inference and is applicable to a wide range of applications. From the testing perspective, we develop general theorems to characterize …
This paper develops a test for homogeneity in finite mixture models where the mixing proportions are known a priori (taken to be 0.5) and a common nuisance parameter is present. Statistical tests based on the notion of Projected Likelihood Contrasts (PLC) are considered. The PLC is a slight modification of the usual likelihood ratio statistic or the Wilk's $Λ$ and is similar in spirit to the Rao's score test. Theoretical investigations have been carried out to understand the large sample statistical properties of these tests. Simulation studies have been carried out to understand the behavior …
In many scientific studies, it is of interest to determine whether an exposure has a causal effect on an outcome. In observational studies, this is a challenging task due to the presence of confounding variables that affect both the exposure and the outcome. Many methods have been developed to test for the presence of a causal effect when all such confounding variables are observed and when the exposure of interest is discrete. In this article, we propose a class of nonparametric tests of the null hypothesis that there is no average causal effect of an arbitrary univariate exposure on an outco…
Leiden University（莱顿大学）；Vrije Universiteit（阿姆斯特丹自由大学）；University of Twente（特文特大学）；Centrum Wiskunde & Informatica（荷兰国家数学与计算机科学研究所）
We develop the theory of hypothesis testing based on the e-value, a notion of evidence that, unlike the p-value, allows for effortlessly combining results from several studies in the common scenario where the decision to perform a new study may depend on previous outcomes. Tests based on e-values are safe, i.e. they preserve Type-I error guarantees, under such optional continuation. We define growth-rate optimality (GRO) as an analogue of power in an optional continuation context, and we show how to construct GRO e-variables for general testing problems with composite null and alternative, emp…
We consider tests of hypotheses when the parameters are not identifiable under the null in semiparametric models, where regularity conditions for profile likelihood theory fail. Exponential average tests based on integrated profile likelihood are constructed and shown to be asymptotically optimal under a weighted average power criterion with respect to a prior on the nonidentifiable aspect of the model. These results extend existing results for parametric models, which involve more restrictive assumptions on the form of the alternative than do our results. Moreover, the proposed tests accommod…
This work is motivated by learning the individualized minimal clinically important difference, a vital concept to assess clinical importance in various biomedical studies. We formulate the scientific question into a high-dimensional statistical problem where the parameter of interest lies in an individualized linear threshold. The goal is to develop a hypothesis testing procedure for the significance of a single element in this parameter as well as of a linear combination of this parameter. The difficulty dues to the high-dimensional nuisance in developing such a testing procedure, and also st…
Multiple imputation (MI) inference handles missing data by imputing the missing values $m$ times, and then combining the results from the $m$ complete-data analyses. However, the existing method for combining likelihood ratio tests (LRTs) has multiple defects: (i) the combined test statistic can be negative, but its null distribution is approximated by an $F$-distribution; (ii) it is not invariant to re-parametrization; (iii) it fails to ensure monotonic power owing to its use of an inconsistent estimator of the fraction of missing information (FMI) under the alternative hypothesis; and (iv) i…
Exact Conditional Confidence Intervals for Cramér's V: Near-Nominal and Tight Where the Guaranteed Interval Is Wide and the Software Interval Does Not Cover
We study a likelihood ratio test for the location of the mode of a log-concave density. Our test is based on comparison of the log-likelihoods corresponding to the unconstrained maximum likelihood estimator of a log-concave density and the constrained maximum likelihood estimator where the constraint is that the mode of the density is fixed, say at $m$. The constrained estimation problem is studied in detail in Doss and Wellner [2018]. Here the results of that paper are used to show that, under the null hypothesis (and strict curvature of $-\log f$ at the mode), the likelihood ratio statistic …
It is well known that correlations in microarray data represent a serious nuisance deteriorating the performance of gene selection procedures. This paper is intended to demonstrate that the correlation structure of microarray data provides a rich source of useful information. We discuss distinct correlation substructures revealed in microarray gene expression data by an appropriate ordering of genes. These substructures include stochastic proportionality of expression signals in a large percentage of all gene pairs, negative correlations hidden in ordered gene triples, and a long sequence of w…
In this paper, we are concerned with how to select significant variables in semiparametric modeling. Variable selection for semiparametric regression models consists of two components: model selection for nonparametric components and selection of significant variables for the parametric portion. Thus, semiparametric variable selection is much more challenging than parametric variable selection (e.g., linear and generalized linear models) because traditional variable selection procedures including stepwise regression and the best subset selection now require separate model selection for the non…
In applied settings, tests of hypothesis where a nuisance parameter is only identifiable under the alternative often reduces into one of Testing One Hypothesis Multiple times (TOHM). Specifically, a fine discretization of the space of the non-identifiable parameter is specified, and the null hypothesis is tested against a set of sub-alternative hypothesis, one for each point of the discretization. The resulting sub-test statistics are then combined to obtain a global p-value. In this paper, we discuss a computationally efficient inferential tool to perform TOHM under stringent significance req…
We study the distribution of Durbin-Wu-Hausman (DWH) and Revankar-Hartley (RH) tests for exogeneity from a finite-sample viewpoint, under the null and alternative hypotheses. We consider linear structural models with possibly non-Gaussian errors, where structural parameters may not be identified and where reduced forms can be incompletely specified (or nonparametric). On level control, we characterize the null distributions of all the test statistics. Through conditioning and invariance arguments, we show that these distributions do not involve nuisance parameters. In particular, this applies …
The Jeffreys-Lindley paradox displays how the use of a p-value (or number of standard deviations z) in a frequentist hypothesis test can lead to an inference that is radically different from that of a Bayesian hypothesis test in the form advocated by Harold Jeffreys in the 1930s and common today. The setting is the test of a well-specified null hypothesis (such as the Standard Model of elementary particle physics, possibly with "nuisance parameters") versus a composite alternative (such as the Standard Model plus a new force of nature of unknown strength). The p-value, as well as the ratio of …
