We construct robust empirical Bayes confidence intervals (EBCIs) in a normal means problem. The intervals are centered at the usual linear empirical Bayes estimator, but use a critical value accounting for shrinkage. Parametric EBCIs that assume a normal distribution for the means (Morris, 1983b) may substantially undercover when this assumption is violated. In contrast, our EBCIs control coverage regardless of the means distribution, while remaining close in length to the parametric EBCIs when the means are indeed Gaussian. If the means are treated as fixed, our EBCIs have an average coverage guarantee: the coverage probability is at least $1 - α$ on average across the $n$ EBCIs for each of the means. Our empirical application considers the effects of U.S. neighborhoods on intergenerational mobility.
We consider the problem of interval estimation of the odds ratio. An asymptotic confidence interval is widely applied in medical research. Unfortunately that confidence interval has a poor coverage probability: it is significantly smaller than the nominal confidence level. In this paper a new confidence interval is proposed. The construction needs only information on sample sizes and sample odds ratio. The coverage probability of the proposed confidence interval is at least the nominal confidence level.
We consider interval estimation of the difference between two binomial proportions. Several methods of constructing such an interval are known. Unfortunately those confidence intervals have poor coverage probability: it is significantly smaller than the nominal confidence level. In this paper a new confidence interval is proposed. The construction needs only information on sample sizes and sample difference between proportions. The coverage probability of the proposed confidence interval is at least the nominal confidence level. The new confidence interval is illustrated by a medical example.
Jędrzejczak et al. (2018) constructed a confidence interval for a ratio of quantiles coming from the Dagum distribution, which is frequently applied as a theoretical model in numerous income distribution analyses. The proposed interval is symmetric with respect to the ratio of sample quantiles, which result may be unsatisfactory in many practical applications. The search for a confidence interval with a smaller length led to the derivation of the shortest interval with the ends being asymmetric relative to the ratio of sample quantiles. In the paper, the existence of the shortest confidence in…
We evaluate the model averaged profile likelihood confidence intervals proposed by Fletcher and Turek (2011) in a simple situation in which there are two linear regression models over which we average. We obtain exact expressions for the coverage and the scaled expected length of the intervals and use these to compute these quantities in particular situations. We show that the Fletcher-Turek confidence intervals can have coverage well below the nominal coverage and expected length greater than that of the standard confidence interval with coverage equal to the same minimum coverage. In these s…
Out-of-bag error is commonly used as an estimate of generalisation error in ensemble-based learning models such as random forests. We present confidence intervals for this quantity using the delta-method-after-bootstrap and the jackknife-after-bootstrap techniques. These methods do not require growing any additional trees. We show that these new confidence intervals have improved coverage properties over the naive confidence interval, in real and simulated examples.
We propose simple formulas of confidence intervals for the Wald statistic, likelihood ratio statistic, and score statistic for a network meta-analysis. In addition, we consider resolutions for concerns that network meta-analyses with a small number of studies cannot hold a nominal confidence level. For a bias adjustment in analyses with a small number of studies, a Bartlett-type adjustment is a well-known method. Many Bartlett-type adjustment-type methods are based on maximum likelihood estimators. However, the network meta-analysis often uses the restricted maximum likelihood estimators that …
The current standard for confidence interval construction in the context of a possibly misspecified model is to use an interval based on the sandwich estimate of variance. These intervals provide asymptotically correct coverage, but small-sample coverage is known to be poor. By eliminating a plug-in assumption, we derive a pivot-based method for confidence interval construction under possibly misspecified models. When compared against confidence intervals generated by the sandwich estimate of variance, this method provides more accurate coverage of the pseudo-true parameter at small sample siz…
We introduce a new framework for creating point-wise confidence intervals for the distribution of event times for current status data. Existing methods are based on asymptotics. Our framework is based on binomial properties and motivates confidence intervals that are very simple to apply and are valid, i.e., guarantee nominal coverage. Although these confidence intervals are necessarily conservative for small sample sizes, asymptotically their coverage rate approaches the nominal one. This binomial framework also motivates approximately valid confidence intervals, and simulations show that the…
Contribution to the discussion of the paper "Causal inference using invariant prediction: identification and confidence intervals" by Peters, Bühlmann and Meinshausen, to appear in the Journal of the Royal Statistical Society, Series B.
Confidence interval procedures used in low dimensional settings are often inappropriate for high dimensional applications. When a large number of parameters are estimated, marginal confidence intervals associated with the most significant estimates have very low coverage rates: They are too small and centered at biased estimates. The problem of forming confidence intervals in high dimensional settings has previously been studied through the lens of selection adjustment. In this framework, the goal is to control the proportion of non-covering intervals formed for selected parameters. In this pa…
In economic research inequality measures based on ratios of quantiles are frequently applied to the analysis of income distributions. In the paper, we construct a confidence interval for such measures under the Dagum distribution which has been widely assumed as a~model for income distributions in empirical analyses Its properties are investigated on the basis of computer simulations. The constructed confidence interval is applied to the analysis of inequality income in Poland in 2015.
The factor estimation process is a really challenging task for non-linear models. Even whether researchers manage to successfully estimate model factors, they still must estimate their confidence intervals, which could require a high computational cost to turn them into informative measures. Some methods in the literature attempt to estimate regions within the estimation search space where factors may jointly exist and fit the real data (confidence contours), however, its estimation process raises several issues as the number of factors increases. Hence, in this paper, we focus on the estimati…
Several scientific fields including psychology are undergoing a replication crisis. There are many reasons for this problem, one of which is a misuse of p-values. There are several alternatives to p-values, and in this paper we describe a complement that is geared towards replication. In particular, we focus on confidence intervals for the probability that a parameter estimate will exceed a specified value in an exact replication study. These intervals convey uncertainty in a way that p-values and standard confidence intervals do not, and can help researchers to draw sounder scientific conclus…
When analyzing incomplete data, is it better to use multiple imputation (MI) or full information maximum likelihood (ML)? In large samples ML is clearly better, but in small samples ML's usefulness has been limited because ML commonly uses normal test statistics and confidence intervals that require large samples. We propose small-sample t-based ML confidence intervals that have good coverage and are shorter than t-based confidence intervals under MI. We also show that ML point estimates are less biased and more efficient than MI point estimates in small samples of bivariate normal data. With …
Hard thresholding, LASSO , adaptive LASSO and SCAD point estimators have been suggested for use in the linear regression context when most of the components of the regression parameter vector are believed to be zero, a sparsity type of assumption. Potscher and Schneider, 2010, Electronic Journal of Statistics, have considered the properties of fixed-width confidence intervals that include one of these point estimators (for all possible data values). They consider a normal linear regression model with orthogonal regressors and show that these confidence intervals are longer than the standard co…
Bootstrap smoothed (bagged) parameter estimators have been proposed as an improvement on estimators found after preliminary data-based model selection. The key result of Efron (2014) is a very convenient and widely applicable formula for a delta method approximation to the standard deviation of the bootstrap smoothed estimator. This approximation provides an easily computed guide to the accuracy of this estimator. In addition, Efron (2014) proposed a confidence interval centered on the bootstrap smoothed estimator, with width proportional to the estimate of this approximation to the standard d…
This short note argues that 95% confidence intervals for MCMC estimates can be obtained even without establishing a CLT, by multiplying their widths by 2.3.
A novel confidence interval estimator is proposed for the risk difference in noninferiority binomial trials. The confidence interval is consistent with an exact unconditional test that preserves the type-I error, and has improved power, particularly for smaller sample sizes, compared to the confidence interval by Chan & Zhang (1999). The improved performance of the proposed confidence interval is theoretically justified and demonstrated with simulations and examples. An R package is also distributed that implements the proposed methods along with other confidence interval estimators.
We consider a general regression model, without a scale parameter. Our aim is to construct a confidence interval for a scalar parameter of interest $θ$ that utilizes the uncertain prior information that a distinct scalar parameter $τ$ takes the specified value $t$. This confidence interval should have good coverage properties. It should also have scaled expected length, where the scaling is with respect to the usual confidence interval, that (a) is substantially less than 1 when the prior information is correct, (b) has a maximum value that is not too large and (c) is close to 1 when the data …
We compare two ways of constructing confidence intervals for the moments-matching parameter estimates of a Gaussian spatio-temporal Ornstein-Uhlenbeck process. It was found that those obtained via pairwise likelihood approximations had lower coverages and were more prone to the curse of dimensionality as opposed to those from a parametric bootstrap procedure.
In an empirical Bayes analysis, we use data from repeated sampling to imitate inferences made by an oracle Bayesian with extensive knowledge of the data-generating distribution. Existing results provide a comprehensive characterization of when and why empirical Bayes point estimates accurately recover oracle Bayes behavior. In this paper, we develop flexible and practical confidence intervals that provide asymptotic frequentist coverage of empirical Bayes estimands, such as the posterior mean or the local false sign rate. The coverage statements hold even when the estimands are only partially …
Consider panel data modelled by a linear random intercept model that includes a time-varying covariate. Suppose that we have uncertain prior information that this covariate is exogenous. We present a new confidence interval for the slope parameter that utilizes this uncertain prior information. This interval has minimum coverage probability very close to its nominal coverage. Let the scaled expected length of this new confidence interval be its expected length divided by the expected length of the confidence interval, with the same minimum coverage, constructed using the fixed effects model. T…
Penalized spline regression is a popular method for scatterplot smoothing, but there has long been a debate on how to construct confidence intervals for penalized spline fits. Due to the penalty, the fitted smooth curve is a biased estimate of the target function. Many methods, including Bayesian intervals and the simple-shift bias-reduction, have been proposed to upgrade the coverage of the confidence intervals, but these methods usually fail to adequately improve the situation at predictor values where the function is sharply curved. In this paper, we develop a novel approach to improving th…
Introductory texts on statistics typically only cover the classical "two sigma" confidence interval for the mean value and do not describe methods to obtain confidence intervals for other estimators. The present technical report fills this gap by first defining different methods for the construction of confidence intervals, and then by their application to a binomial proportion, the mean value, and to arbitrary estimators. Beside the frequentist approach, the likelihood ratio and the highest posterior density approach are explained. Two methods to estimate the variance of general maximum likel…
Given $m$ unknown parameters with corresponding independent estimators, the Benjamini-Hochberg (BH) procedure can be used to classify the sign of parameters such that the expected proportion of erroneous directional decisions (directional FDR) is controlled at a preset level $q$. More ambitiously, our goal is to construct sign-determining confidence intervals---instead of only classifying the sign---such that the expected proportion of non-covering constructed intervals (FCR) is controlled. We suggest a valid procedure which adjusts a marginal confidence interval in order to construct a maximu…
This study aims to evaluate the performance of power in the likelihood ratio test for changepoint detection by bootstrap sampling, and proposes a hypothesis test based on bootstrapped confidence interval lengths. Assuming i.i.d normally distributed errors, and using the bootstrap method, the changepoint sampling distribution is estimated. Furthermore, this study describes a method to estimate a data set with no changepoint to form the null sampling distribution. With the null sampling distribution, and the distribution of the estimated changepoint, critical values and power calculations can be…
The goal of any estimation study is an interval estimation of a the parameter(s) of interest. These estimations are mostly expressed using empirical confidence intervals that are based on sample point estimates of the corresponding parameter(s). In contrast, calculations of the necessary sample size usually use expected confidence intervals that are based on the expected value of the parameter(s). The approach that guarantees the required probability of the required width of empirical confidence interval is known at least since 1989. However, till now, this approach is not implemented for most…
We consider the problem of constructing confidence intervals for nonparametric functional data analysis using empirical likelihood. In this doubly infinite-dimensional context, we demonstrate the Wilks's phenomenon and propose a bias-corrected construction that requires neither undersmoothing nor direct bias estimation. We also extend our results to partially linear regression involving functional data. Our numerical results demonstrated the improved performance of empirical likelihood over approximation based on asymptotic normality.
In the analysis of clustered and longitudinal data, which includes a covariate that varies both between and within clusters (e.g. time-varying covariate in longitudinal data), a Hausman pretest is commonly used to decide whether subsequent inference is made using the linear random intercept model or the fixed effects model. We assess the effect of this pretest on the coverage probability and expected length of a confidence interval for the slope parameter. Our results show that for the small levels of significance of the Hausman pretest commonly used in applications, the minimum coverage proba…
Interval estimation of the probability of success in a Binomial model is considered. Zieliński (2018) showed that the confidence interval which uses information about non-homogeneity of the sample is better than the classical one. In the following paper the shortest confidence interval for non-homogenous sample is constructed.
This report compares two tests of second-order stationarity through simulation. It also provides several examples of localised autocovariances and their approximate confidence intervals on different real and simulated data sets. An empirical verification of an asymptotic Gaussianity result is also demonstrated. The commands use to produce figures in a companion paper are also described.
We congratulate the authors on their exciting paper, which introduces a novel idea for assessing the estimation bias in causal estimates. Doubly robust estimators are now part of the standard set of tools in causal inference, but a typical analysis stops with an estimate and a confidence interval. The authors give an approach for a unique type of model-checking that allows the user to check whether the bias is sufficiently small with respect to the standard error, which is generally required for confidence intervals to be reliable.
Split sample methods have recently been put forward as a way to reduce the coverage oscillations that haunt confidence intervals for parameters of lattice distributions, such as the binomial and Poisson distributions. We study split sample intervals in the binomial setting, showing that these intervals can be viewed as being based on adding discrete random noise to the data. It is shown that they can be improved upon by using noise with a continuous distribution instead, regardless of whether the randomization is determined by the data or an external source of randomness. We compare split samp…
It is in general challenging to provide confidence intervals for individual variables in high-dimensional regression without making strict or unverifiable assumptions on the design matrix. We show here that a "group-bound" confidence interval can be derived without making any assumptions on the design matrix. The lower bound for the regression coefficient of individual variables can be derived via linear programming. The idea also generalises naturally to groups of variables, where we can derive a one-sided confidence interval for the joint effect of a group. While the confidence intervals of …
Consider a linear regression model with n-dimensional response vector, regression parameter β= (β_1, ..., β_p) and independent and identically N(0, σ^2) distributed errors. Suppose that the parameter of interest is θ= a^T βwhere a is a specified vector. Define the parameter τ= c^T β- t where c and t are specified. Also suppose that we have uncertain prior information that τ= 0. Part of our evaluation of a frequentist confidence interval for θis the ratio (expected length of this confidence interval)/(expected length of standard 1-αconfidence interval), which we call the scaled expected length …
We compare the following two sources of poor coverage of post-model-selection confidence intervals: the preliminary data-based model selection sometimes chooses the wrong model and the data used to choose the model is re-used for the construction of the confidence interval.
Confidence interval (CI) methods for stratified bilateral studies use intraclass correlation to avoid misleading results. In this article, we propose four CI methods (sample-size weighted global MLE-based Wald-type CI, complete MLE-based Wald-type CI, profile likelihood CI, and complete MLE-based score CI) to investigate CIs of proportion ratios to clinical trial design with stratified bilateral data under Dallal's intraclass model. Monte Carlo simulations are performed, and the complete MLE-based score confidence interval (CS) method yields a robust outcome. Lastly, a real data example is con…
This is the rejoinder to the discussion by Kennedy, Balakrishnan and Wasserman on the paper "On nearly assumption-free tests of nominal confidence interval coverage for causal parameters estimated by machine learning" published in Statistical Science.
In this short note I apply the methodology of game-theoretic probability to calculating non-asymptotic confidence intervals for the coefficient of a simple first order scalar autoregressive model. The most distinctive feature of the proposed procedure is that with high probability it produces confidence intervals that always cover the true parameter value when applied sequentially.
A new method is proposed for the correction of confidence intervals when the original interval does not have the correct nominal coverage probabilities in the frequentist sense. The proposed method is general and does not require any distributional assumptions. It can be applied to both frequentist and Bayesian inference where interval estimates are desired. We provide theoretical results for the consistency of the proposed estimator, and give two complex examples, on confidence interval correction for composite likelihood estimators and in approximate Bayesian computation (ABC), to demonstrat…
Linear combinations of multinomial probabilities, such as those resulting from contingency tables, are of use when evaluating classification system performance. While large sample inference methods for these combinations exist, small sample methods exist only for regions on the multinomial parameter space instead of the linear combinations. However, in medical classification problems it is common to have small samples necessitating a small sample confidence interval on linear combinations of multinomial probabilities. Therefore, in this paper we derive an exact confidence interval, through the…
We consider the classic problem of interval estimation of a proportion $p$ based on binomial sampling. The "exact" Clopper-Pearson confidence interval for $p$ is known to be unnecessarily conservative. We propose coverage-adjustments of the Clopper-Pearson interval using prior and posterior distributions of $p$. The adjusted intervals have improved coverage and are often shorter than competing intervals found in the literature. Using new heatmap-type plots for comparing confidence intervals, we find that the coverage-adjusted intervals are particularly suitable for $p$ close to 0 or 1.
Null hypothesis significance tests and p values are widely used despite very strong arguments against their use in many contexts. Confidence intervals are often recommended as an alternative, but these do not achieve the objective of assessing the credibility of a hypothesis, and the distinction between confidence and probability is an unnecessary confusion. This paper proposes a more straightforward (probabilistic) definition of confidence, and suggests how the idea can be applied to whatever hypotheses are of interest to researchers. The relative merits of the different approaches are discus…
The purpose of this paper is to propose methodologies for statistical inference of low-dimensional parameters with high-dimensional data. We focus on constructing confidence intervals for individual coefficients and linear combinations of several of them in a linear regression model, although our ideas are applicable in a much broad context. The theoretical results presented here provide sufficient conditions for the asymptotic normality of the proposed estimators along with a consistent estimator for their finite-dimensional covariance matrices. These sufficient conditions allow the number of…
The problem of quantifying uncertainty about the locations of multiple change points by means of confidence intervals is addressed. The asymptotic distribution of the change point estimators obtained as the local maximisers of moving sum statistics is derived, where the limit distributions differ depending on whether the corresponding size of changes is local, i.e. tends to zero as the sample size increases, or fixed. A bootstrap procedure for confidence interval generation is proposed which adapts to the unknown magnitude of changes and guarantees asymptotic validity both for local and fixed …
In many statistical problems, several estimators are usually available for interval estimation of a parameter of interest, and hence, the selection of an appropriate estimator is important. The criterion for a good estimator is to have a high coverage probability close to the nominal level and a shorter interval length. However, these two concepts are in opposition to each other: high and low coverages are associated with longer and shorter interval lengths respectively. Some methods, such as bootstrap calibration, modify the nominal level to improve the coverage and thereby allow the selectio…
Confidence intervals for the means of multiple normal populations are often based on a hierarchical normal model. While commonly used interval procedures based on such a model have the nominal coverage rate on average across a population of groups, their actual coverage rate for a given group will be above or below the nominal rate, depending on the value of the group mean. Alternatively, a coverage rate that is constant as a function of a group's mean can be simply achieved by using a standard $t$-interval, based on data only from that group. The standard $t$-interval, however, fails to share…
We extend the method of Balister, Bollobás and Walters for determining rigorous confidence intervals for the critical threshold of two dimensional lattices to three (and higher) dimensional lattices. We describe a method for determining a full confidence interval and apply it to show that the critical threshold for bond percolation on the simple cubic lattice is between 0.2485 and 0.2490 with 99.9999% confidence, and the critical threshold for site percolation on the same lattice is between 0.3110 and 0.3118 with 99.9999% confidence.
We present several related methods for creating confidence intervals to assess disease prevalence in variety of survey sampling settings. These include simple random samples with imperfect tests, weighted sampling with perfect tests, and weighted sampling with imperfect tests, with the first two settings considered special cases of the third. Our methods use survey results and measurements of test sensitivity and specificity to construct melded confidence intervals. We demonstrate that our methods appear to guarantee coverage in simulated settings, while competing methods are shown to achieve …
Suppose that X_1,X_2,...,X_n are independent and identically Bernoulli(theta) distributed. Also suppose that our aim is to find an exact confidence interval for theta that is the intersection of a 1-α/2 upper confidence interval and a 1-α/2 lower confidence interval. The Clopper-Pearson interval is the standard such confidence interval for theta, which is widely used in practice. We consider the randomized confidence interval of Stevens, 1950 and present some extensions, including pseudorandomized confidence intervals. We also consider the "data-randomized" confidence interval of Korn, 1987 an…
The purpose of the present paper is to assess the efficacy of confidence intervals for Rosenthal's fail-safe number. Although Rosenthal's estimator is highly used by researchers, its statistical properties are largely unexplored. First of all, we developed statistical theory which allowed us to produce confidence intervals for Rosenthal's fail-safe number.This was produced by discerning whether the number of studies analysed in a meta-analysis is fixed or random. Each case produces different variance estimators. For a given number of studies and a given distribution, we provided five variance …
We present an efficient method of calculating exact confidence intervals for the hypergeometric parameter representing the number of "successes," or "special items," in the population. The method inverts minimum-width acceptance intervals after shifting them to make their endpoints nondecreasing while preserving their level. The resulting set of confidence intervals achieves minimum possible average size, and even in comparison with confidence sets not required to be intervals it attains the minimum possible cardinality most of the time, and always within $1$. The method compares favorably wit…
