This work develops central limit theorems for cross-validation and consistent estimators of its asymptotic variance under weak stability conditions on the learning algorithm. Together, these results provide practical, asymptotically-exact confidence intervals for $k$-fold test error and valid, powerful hypothesis tests of whether one learning algorithm has smaller $k$-fold test error than another. These results are also the first of their kind for the popular choice of leave-one-out cross-validation. In our real-data experiments with diverse learning algorithms, the resulting intervals and tests outperform the most popular alternative methods from the literature.
Institut de Mathématiques de Toulouse（图卢兹数学研究所）；Universidad de Valladolid（巴利亚多利德大学）
We provide the asymptotic distribution of the major indexes used in the statistical literature to quantify disparate treatment in machine learning. We aim at promoting the use of confidence intervals when testing the so-called group disparate impact. We illustrate on some examples the importance of using confidence intervals and not a single value.
MediaTek Research（联发科研究）；National University of Singapore（新加坡国立大学）；University of California, San Diego（加州大学圣地亚哥分校）
Confidence intervals are a crucial building block in the analysis of various online learning problems. The analysis of kernel based bandit and reinforcement learning problems utilize confidence intervals applicable to the elements of a reproducing kernel Hilbert space (RKHS). However, the existing confidence bounds do not appear to be tight, resulting in suboptimal regret bounds. In fact, the existing regret bounds for several kernelized bandit algorithms (e.g., GP-UCB, GP-TS, and their variants) may fail to even be sublinear. It is unclear whether the suboptimal regret bound is a fundamental …
Suppose that X_1,X_2,...,X_n are independent and identically Bernoulli(theta) distributed. Also suppose that our aim is to find an exact confidence interval for theta that is the intersection of a 1-α/2 upper confidence interval and a 1-α/2 lower confidence interval. The Clopper-Pearson interval is the standard such confidence interval for theta, which is widely used in practice. We consider the randomized confidence interval of Stevens, 1950 and present some extensions, including pseudorandomized confidence intervals. We also consider the "data-randomized" confidence interval of Korn, 1987 an…
College of Information and Computer Sciences（信息与计算机科学学院）；University of Massachusetts（马萨诸塞大学）
We present a new method for constructing a confidence interval for the mean of a bounded random variable from samples of the random variable. We conjecture that the confidence interval has guaranteed coverage, i.e., that it contains the mean with high probability for all distributions on a bounded interval, for all samples sizes, and for all confidence levels. This new method provides confidence intervals that are competitive with those produced using Student's t-statistic, but does not rely on normality assumptions. In particular, its only requirement is that the distribution be bounded on a …
This paper proposes a decorrelation-based approach to test hypotheses and construct confidence intervals for the low dimensional component of high dimensional proportional hazards models. Motivated by the geometric projection principle, we propose new decorrelated score, Wald and partial likelihood ratio statistics. Without assuming model selection consistency, we prove the asymptotic normality of these test statistics, establish their semiparametric optimality. We also develop new procedures for constructing pointwise confidence intervals for the baseline hazard function and baseline survival…
Well-recommended methods of forming `confidence intervals' for a binomial proportion give interval estimates that do not actually meet the definition of a confidence interval, in that their coverages are sometimes lower than the nominal confidence level. The methods are favoured because their intervals have a shorter average length than the Clopper-Pearson (gold-standard) method, whose intervals really are confidence intervals. Comparison of such methods is tricky -- the best method should perhaps be the one that gives the shortest intervals (on average), but when is the coverage of a method s…
We discuss a new way of constructing pointwise confidence intervals for the distribution function in the current status model. The confidence intervals are based on the smoothed maximum likelihood estimator (SMLE) and constructed using bootstrap methods. Other methods to construct confidence intervals, using the non-standard limit distribution of the (restricted) MLE, are compared to our approach via simulations and real data applications.
We introduce a general method, named the h-function method, to unify the constructions of level-alpha exact test and 1-alpha exact confidence interval. Using this method, any confidence interval is improved as follows: i) an approximate interval, including a point estimator, is modified to an exact interval; ii) an exact interval is refined to be an interval that is a subset of the previous one. Two real datasets are used to illustrate the method.
A novel confidence interval estimator is proposed for the risk difference in noninferiority binomial trials. The confidence interval is consistent with an exact unconditional test that preserves the type-I error, and has improved power, particularly for smaller sample sizes, compared to the confidence interval by Chan & Zhang (1999). The improved performance of the proposed confidence interval is theoretically justified and demonstrated with simulations and examples. An R package is also distributed that implements the proposed methods along with other confidence interval estimators.
We exploit the asymptotic normality of the extreme value theory (EVT) based estimators of the parameters of a symmetric Lévy-stable distribution, to construct confidence intervals. The accuracy of these intervals is evaluated through a simulation study.
One-sided confidence intervals are presented for the average of non-identical Bernoulli parameters. These confidence intervals are expressed as analytical functions of the total number of Bernoulli games won, the number of rounds and the confidence level. Tightness of these bounds in the sense of Buehler, i.e. as the strictest possible monotonic intervals, is demonstrated for all confidence levels. A simple interval valid for all confidence levels is also provided with a tightness guarantee. Finally, an application of the proposed confidence intervals to sequential sampling is discussed.
This note points out how confidence interval estimates for standard deviation transform into confidence interval estimates for the power of F-tests at fixed alternative means. An application is shown for the test of a two-sided hypothesis for the mean of a normal distribution.
It is proved that the average length of standard confidence intervals for parameters of gamma and normal distributions monotonically decrease with the sample size. The proofs are based on fine properties of the classical gamma function.
Interval estimation of the probability of success in a Binomial model is considered. Zieliński (2018) showed that the confidence interval which uses information about non-homogeneity of the sample is better than the classical one. In the following paper the shortest confidence interval for non-homogenous sample is constructed.
Consider a linear regression model with independent and identically normally distributed random errors. Suppose that the parameter of interest is a specified linear combination of the regression parameters. We prove that the usual confidence interval for this parameter is admissible within a broad class of confidence intervals.
We study nonparametric isotonic confidence intervals for monotone functions. In Banerjee and Wellner (2001) pointwise confidence intervals, based on likelihood ratio tests for the restricted and unrestricted MLE in the current status model, are introduced. We extend the method to the treatment of other models with monotone functions, and demonstrate our method by a new proof of the results in Banerjee and Wellner (2001) and also by constructing confidence intervals for monotone densities, for which still theory had to be developed. For the latter model we prove that the limit distribution of t…
Rating systems are ubiquitous, with applications ranging from product recommendation to teaching evaluations. Confidence intervals for functionals of rating data such as empirical means or quantiles are critical to decision-making in various applications including recommendation/ranking algorithms. Confidence intervals derived from standard Hoeffding and Bernstein bounds can be quite loose, especially in small sample regimes, since these bounds do not exploit the geometric structure of the probability simplex. We propose a new approach to deriving confidence intervals that are tailored to the …
We study exact confidence intervals and two-sided hypothesis tests for univariate parameters of stochastically increasing discrete distributions, such as the binomial and Poisson distributions. It is shown that several popular methods for constructing short intervals lack strict nestedness, meaning that accepting a lower confidence level not always will lead to a shorter confidence interval. These intervals correspond to a class of tests that are shown to assign differing $p$-values to indistinguishable models. Finally, we show that among strictly nested intervals, fiducial intervals, includin…
We compare the following two sources of poor coverage of post-model-selection confidence intervals: the preliminary data-based model selection sometimes chooses the wrong model and the data used to choose the model is re-used for the construction of the confidence interval.
This paper presents two results concerning uniform confidence intervals for the tail index and the extreme quantile. First, we show that it is impossible to construct a length-optimal confidence interval satisfying the correct uniform coverage over a local non-parametric family of tail distributions. Second, in light of the impossibility result, we construct honest confidence intervals that are uniformly valid by incorporating the worst-case bias in the local non-parametric family. The proposed method is applied to simulated data and a real data set of National Vital Statistics from National C…
Confidence sets play a fundamental role in statistical inference. In this paper, we consider confidence intervals for high dimensional linear regression with random design. We first establish the convergence rates of the minimax expected length for confidence intervals in the oracle setting where the sparsity parameter is given. The focus is then on the problem of adaptation to sparsity for the construction of confidence intervals. Ideally, an adaptive confidence interval should have its length automatically adjusted to the sparsity of the unknown regression vector, while maintaining a prespec…
In this paper, we develop an approach for optimizing the explicit binomial confidence interval recently derived by Chen et al. The optimization reduces conservativeness while guaranteeing prescribed coverage probability.
In this short note I apply the methodology of game-theoretic probability to calculating non-asymptotic confidence intervals for the coefficient of a simple first order scalar autoregressive model. The most distinctive feature of the proposed procedure is that with high probability it produces confidence intervals that always cover the true parameter value when applied sequentially.
An empirical Bayes confidence interval has high user demand in many applications. In particular, the second-order empirical Bayes confidence interval, the coverage error of which is of the third order for large number of areas, is widely used in small area estimation when the sample size within each area is not large enough to make reliable direct estimates based on a design-based approach. Yoshimori and Lahiri (2014a) proposed a new type of confidence interval, called the second-order efficient empirical Bayes confidence interval, whose length is less than that of the direct confidence interv…
University of Alberta（阿尔伯塔大学）；DeepMind（深度思维）；Google Research（谷歌研究院）
We study high-confidence behavior-agnostic off-policy evaluation in reinforcement learning, where the goal is to estimate a confidence interval on a target policy's value, given only access to a static experience dataset collected by unknown behavior policies. Starting from a function space embedding of the linear program formulation of the $Q$-function, we obtain an optimization problem with generalized estimating equation constraints. By applying the generalized empirical likelihood method to the resulting Lagrangian, we propose CoinDICE, a novel and efficient algorithm for computing confide…
Practical or scientific considerations often lead to selecting a subset of parameters as ``important.'' Inferences about those parameters often are based on the same data used to select them in the first place. That can make the reported uncertainties deceptively optimistic: confidence intervals that ignore selection generally have less than their nominal coverage probability. Controlling the probability that one or more intervals for selected parameters do not cover---the ``simultaneous over the selected'' (SoS) error rate---is crucial in many scientific problems. Intervals that control the S…
We consider the classic problem of interval estimation of a proportion $p$ based on binomial sampling. The "exact" Clopper-Pearson confidence interval for $p$ is known to be unnecessarily conservative. We propose coverage-adjustments of the Clopper-Pearson interval using prior and posterior distributions of $p$. The adjusted intervals have improved coverage and are often shorter than competing intervals found in the literature. Using new heatmap-type plots for comparing confidence intervals, we find that the coverage-adjusted intervals are particularly suitable for $p$ close to 0 or 1.
In this short note, we show how to use concentration inequalities in order to build exact confidence intervals for the Hurst parameter associated with a one-dimensional fractional Brownian motion
In this article, we consider the problem of constructing the confidence interval and testing hypothesis for the common coefficient of variation (CV) of several normal populations. A new method is suggested using the concepts of generalized p-value and generalized confidence interval. Using this new method and a method proposed by Tian (2005), we obtain a shorter confidence interval for the common CV. This combination method has good properties in terms of length and coverage probability compared to other methods. A simulation study is performed to illustrate properties.. Finally, these methods…
We provide Buehler-optimal one-sided and some valid two-sided confidence intervals for the average success probability of a possibly inhomogeneous fixed length Bernoulli chain, based on the number of observed successes. Contrary to some claims in the literature, the one-sided Clopper-Pearson intervals for the homogeneous case are not completely robust here, not even if applied to hypergeometric estimation problems.
In this article, we derive an explicit formula for computing confidence interval for the mean of a bounded random variable. Moreover, we have developed multistage point estimation methods for estimating the mean value with prescribed precision and confidence level based on the proposed confidence interval.
For any class of one-sided $1-α$ confidence intervals with a certain monotonicity ordering on the random confidence limit, the smallest interval, in the sense of the set inclusion for the difference of two proportions of two independent binomial random variables, is constructed based on a direct analysis of coverage probability function. A special ordering on the confidence limit is developed and the corresponding smallest confidence interval is derived. This interval is then applied to identify the minimum effective dose (MED) for binary data in dose-response studies, and a multiple test proc…
Recently we proposed quantum language(or, measurement theory), which is characterized as the linguistic turn of the Copenhagen interpretation of quantum mechanics. Also, we consider that this is a kind of system theory such that it is applicable to both classical and quantum systems. As far as classical systems, it should be noted that quantum language is similar to statistics. In this paper, we discuss the usual confidence interval methods in terms of quantum language. And we assert that three concepts (i.e.,"estimator" and "quantity" and "semi-distance)are indispensable for the theoretical u…
Portnoy (2019) considered the problem of constructing an optimal confidence interval for the mean based on a single observation $\, X \sim {\cal{N}}(μ, \, σ^2) \,$. Here we extend this result to obtaining 1-sample confidence intervals for $\, σ\,$ and to cases of symmetric unimodal distributions and of distributions with compact support. Finally, we extend the multivariate result in Portnoy (2019) to allow a sample of size $\, m \,$ from a multivariate normal distribution where $m$ may be less than the dimension.
Valid inference after model selection is currently a very active area of research. The polyhedral method, pioneered by Lee, et al. (2016), allows for valid inference after model selection if the model selection event can be described by polyhedral constraints. In that reference, the method is exemplified by constructing two valid confidence intervals when the Lasso estimator is used to select a model. We here study the length of these intervals. For one of these confidence intervals, which is easier to compute, we find that its expected length is always infinite. For the other of these confide…
Empirical-likelihood-based confidence intervals for a mean were introduced by Owen [Biometrika 75 (1988) 237-249], where at least a finite second moment is required. This excludes some important distributions, for example, those in the domain of attraction of a stable law with index between 1 and 2. In this article we use a method similar to Qin and Wong [Scand. J. Statist. 23 (1996) 209-219] to derive an empirical-likelihood-based confidence interval for the mean when the underlying distribution has heavy tails. Our method can easily be extended to obtain a confidence interval for any order o…
We introduce a new adjusted residual maximum likelihood method (REML) in the context of producing an empirical Bayes (EB) confidence interval for a normal mean, a problem of great interest in different small area applications. Like other rival empirical Bayes confidence intervals such as the well-known parametric bootstrap empirical Bayes method, the proposed interval is second-order correct, that is, the proposed interval has a coverage error of order $O(m^{-{3}/{2}})$. Moreover, the proposed interval is carefully constructed so that it always produces an interval shorter than the correspondi…
We present an efficient method of calculating exact confidence intervals for the hypergeometric parameter representing the number of "successes," or "special items," in the population. The method inverts minimum-width acceptance intervals after shifting them to make their endpoints nondecreasing while preserving their level. The resulting set of confidence intervals achieves minimum possible average size, and even in comparison with confidence sets not required to be intervals it attains the minimum possible cardinality most of the time, and always within $1$. The method compares favorably wit…
We study the problem of constructing honest and adaptive confidence intervals for the tail coefficient in the second order Pareto model, when the second order coefficient is unknown. This problem is translated into a testing problem on the second order parameter. By constructing an appropriate model and an associated test statistic, we provide a uniform and adaptive confidence interval for the first order parameter. We also provide an almost matching lower bound, which proves that the result is minimax optimal up to a logarithmic factor.
In this paper, we show how concentration inequalities for Gaussian quadratic form can be used to propose exact confidence intervals of the Hurst index parametrizing a fractional Brownian motion. Both cases where the scaling parameter of the fractional Brownian motion is known or unknown are investigated. These intervals are obtained by observing a single discretized sample path of a fractional Brownian motion and without any assumption on the parameter $H$.
A nonparametric adaptation theory is developed for the construction of confidence intervals for linear functionals. A between class modulus of continuity captures the expected length of adaptive confidence intervals. Sharp lower bounds are given for the expected length and an ordered modulus of continuity is used to construct adaptive confidence procedures which are within a constant factor of the lower bounds. In addition, minimax theory over nonconvex parameter spaces is developed.
Consider X_1,X_2,...,X_n that are independent and identically N(mu,sigma^2) distributed. Suppose that we have uncertain prior information that mu = 0. We answer the question: to what extent can a frequentist 1-alpha confidence interval for mu utilize this prior information?
I propose a new type of confidence interval for correct asymptotic inference after using data to select a model of interest without assuming any model is correctly specified. This hybrid confidence interval is constructed by combining techniques from the selective inference and post-selection inference literatures to yield a short confidence interval across a wide range of data realizations. I show that hybrid confidence intervals have correct asymptotic coverage, uniformly over a large class of probability distributions that do not bound scaled model parameters. I illustrate the use of these …
Consider a linear regression model with n-dimensional response vector, p-dimensional regression parameter beta and independent normally distributed errors. Suppose that the parameter of interest is theta = a^T beta where a is a specified vector. Define the s-dimensional parameter vector tau=C^T beta-t where C and t are specified. Also suppose that we have uncertain prior information that tau=0. Part of our evaluation of a frequentist confidence interval for theta is the ratio (expected length of this confidence interval)/(expected length of standard 1-alpha confidence interval), the scaled exp…
The goal of this paper is to develop a practical and general-purpose approach to construct confidence intervals for differentially private parametric estimation. We find that the parametric bootstrap is a simple and effective solution. It cleanly reasons about variability of both the data sample and the randomized privacy mechanism and applies "out of the box" to a wide class of private estimation routines. It can also help correct bias caused by clipping data to limit sensitivity. We prove that the parametric bootstrap gives consistent confidence intervals in two broadly relevant settings, in…
Standard confidence intervals employed in applied statistical analysis are usually based on asymptotic approximations. Such approximations can be considerably inaccurate in small and moderate sized samples. We derive accurate confidence intervals based on higher-order approximate quantiles of the score function. The coverage approximation error is $O(n^{-3/2})$ while the approximation error of confidence intervals based on the asymptotic normality of MLEs is $O(n^{-1/2})$. Monte Carlo simulations confirm the theoretical findings. An implementation for regression models and real data applicatio…
An important statistical inference problem in sequential analysis is the construction of confidence intervals following sequential tests, to which Michael Woodroofe has made fundamental contributions. This paper reviews Woodroofe's method and other approaches in the literature. In particular it shows how a bias-corrected pivot originally introduced by Woodroofe can be used as an improved root for sequential bootstrap confidence intervals.
We study an AMOC time series model with an abrupt change in the mean and dependent errors that fulfill certain mixing conditions. We obtain confidence intervals for the unknown change-point via bootstrapping methods. Precisely we use a block bootstrap of the estimated centered error sequence. Then we reconstruct a sequence with a change in the mean using the same estimators as before. The difference between the change-point estimator of the resampled sequence and the one for the original sequence can be use as an approximation of the difference between the real change-point and its estimator. …
The recent paper "Simple confidence intervals for MCMC without CLTs" by J.S. Rosenthal, showed the derivation of a simple MCMC confidence interval using only Chebyshev's inequality, not CLT. That result required certain assumptions about how the estimator bias and variance grow with the number of iterations $n$. In particular, the bias is $o(1/\sqrt{n})$. This assumption seemed mild. It is generally believed that the estimator bias will be $O(1/n)$ and hence $o(1/\sqrt{n})$. However, questions were raised by researchers about how to verify this assumption. Indeed, we show that this assumption …
For a reversible and ergodic Markov chain $\{X_n,n\geq0\}$ with invariant distribution $π$, we show that a valid confidence interval for $π(h)$ can be constructed whenever the asymptotic variance $σ^2_P(h)$ is finite and positive. We do not impose any additional condition on the convergence rate of the Markov chain. The confidence interval is derived using the so-called fixed-b lag-window estimator of $σ_P^2(h)$. We also derive a result that suggests that the proposed confidence interval procedure converges faster than classical confidence interval procedures based on the Gaussian distribution…
Adaptive confidence intervals for regression functions are constructed under shape constraints of monotonicity and convexity. A natural benchmark is established for the minimum expected length of confidence intervals at a given function in terms of an analytic quantity, the local modulus of continuity. This bound depends not only on the function but also the assumed function class. These benchmarks show that the constructed confidence intervals have near minimum expected length for each individual function, while maintaining a given coverage probability for functions within the class. Such ada…
Letter to the Editor: Some comments on "On construction of the smallest one-sided confidence interval for the difference of two proportions" by Weizhen Wang [arXiv:1002.4945].
In this paper we consider the problem of constructing confidence intervals for coefficients of martingale regression models (in particular, time series models) after variable selection. Although constructing confidence intervals are common practice in statistical analysis, it is challenging in our framework due to the data-dependence of the selected model and the correlation among the variables being selected and not selected. We first introduce estimators for the selected coefficients and show that it is consistent under martingale regression model, in which the observations can be dependent …
We derive the exponential non improvable Grand Lebesgue Space norm decreasing estimations for tail of distribution for exact normed deviation for the famous recursive Wolverton-Wagner multivariate statistical density estimation. We consider pointwise as well as Lebesgue-Riesz norm error of statistical density of measurement.
The purpose of this paper is to construct confidence intervals for the regression coefficients in high-dimensional Cox proportional hazards regression models where the number of covariates may be larger than the sample size. Our debiased estimator construction is similar to those in Zhang and Zhang (2014) and van de Geer et al. (2014), but the time-dependent covariates and censored risk sets introduce considerable additional challenges. Our theoretical results, which provide conditions under which our confidence intervals are asymptotically valid, are supported by extensive numerical experimen…
