Measuring performance & quantifying a performance change are core evaluation techniques in programming language and systems research. Of 122 recent scientific papers, as many as 65 included experimental evaluation that quantified a performance change using a ratio of execution times. Few of these papers evaluated their results with the level of rigour that has come to be expected in other experimental sciences. The uncertainty of measured results was largely ignored. Scarcely any of the papers mentioned uncertainty in the ratio of the mean execution times, and most did not even mention uncertainty in the two means themselves. Most of the papers failed to address the non-deterministic execution of computer programs (caused by factors such as memory placement, for example), and none addressed non-deterministic compilation. It turns out that the statistical methods presented in the computer systems performance evaluation literature for the design and summary of experiments do not readily allow this either. This poses a hazard to the repeatability, reproducibility and even validity of quantitative results.
Inspired by statistical methods used in other fields of science, and building on results in statistics that did not make it to introductory textbooks, we present a statistical model that allows us both to quantify uncertainty in the ratio of (execution time) means and to design experiments with a rigorous treatment of those multiple sources of non-determinism that might impact measured performance. Better still, under our framework summaries can be as simple as "system A is faster than system B by 5.5% $\pm$ 2.5%, with 95% confidence", a more natural statement than those derived from typical current practice, which are often misinterpreted.
November 2013
We consider the problem of interval estimation of the odds ratio. An asymptotic confidence interval is widely applied in medical research. Unfortunately that confidence interval has a poor coverage probability: it is significantly smaller than the nominal confidence level. In this paper a new confidence interval is proposed. The construction needs only information on sample sizes and sample odds ratio. The coverage probability of the proposed confidence interval is at least the nominal confidence level.
We consider interval estimation of the difference between two binomial proportions. Several methods of constructing such an interval are known. Unfortunately those confidence intervals have poor coverage probability: it is significantly smaller than the nominal confidence level. In this paper a new confidence interval is proposed. The construction needs only information on sample sizes and sample difference between proportions. The coverage probability of the proposed confidence interval is at least the nominal confidence level. The new confidence interval is illustrated by a medical example.
Jędrzejczak et al. (2018) constructed a confidence interval for a ratio of quantiles coming from the Dagum distribution, which is frequently applied as a theoretical model in numerous income distribution analyses. The proposed interval is symmetric with respect to the ratio of sample quantiles, which result may be unsatisfactory in many practical applications. The search for a confidence interval with a smaller length led to the derivation of the shortest interval with the ends being asymmetric relative to the ratio of sample quantiles. In the paper, the existence of the shortest confidence in…
Out-of-bag error is commonly used as an estimate of generalisation error in ensemble-based learning models such as random forests. We present confidence intervals for this quantity using the delta-method-after-bootstrap and the jackknife-after-bootstrap techniques. These methods do not require growing any additional trees. We show that these new confidence intervals have improved coverage properties over the naive confidence interval, in real and simulated examples.
We propose simple formulas of confidence intervals for the Wald statistic, likelihood ratio statistic, and score statistic for a network meta-analysis. In addition, we consider resolutions for concerns that network meta-analyses with a small number of studies cannot hold a nominal confidence level. For a bias adjustment in analyses with a small number of studies, a Bartlett-type adjustment is a well-known method. Many Bartlett-type adjustment-type methods are based on maximum likelihood estimators. However, the network meta-analysis often uses the restricted maximum likelihood estimators that …
We introduce a new framework for creating point-wise confidence intervals for the distribution of event times for current status data. Existing methods are based on asymptotics. Our framework is based on binomial properties and motivates confidence intervals that are very simple to apply and are valid, i.e., guarantee nominal coverage. Although these confidence intervals are necessarily conservative for small sample sizes, asymptotically their coverage rate approaches the nominal one. This binomial framework also motivates approximately valid confidence intervals, and simulations show that the…
In economic research inequality measures based on ratios of quantiles are frequently applied to the analysis of income distributions. In the paper, we construct a confidence interval for such measures under the Dagum distribution which has been widely assumed as a~model for income distributions in empirical analyses Its properties are investigated on the basis of computer simulations. The constructed confidence interval is applied to the analysis of inequality income in Poland in 2015.
The factor estimation process is a really challenging task for non-linear models. Even whether researchers manage to successfully estimate model factors, they still must estimate their confidence intervals, which could require a high computational cost to turn them into informative measures. Some methods in the literature attempt to estimate regions within the estimation search space where factors may jointly exist and fit the real data (confidence contours), however, its estimation process raises several issues as the number of factors increases. Hence, in this paper, we focus on the estimati…
Several scientific fields including psychology are undergoing a replication crisis. There are many reasons for this problem, one of which is a misuse of p-values. There are several alternatives to p-values, and in this paper we describe a complement that is geared towards replication. In particular, we focus on confidence intervals for the probability that a parameter estimate will exceed a specified value in an exact replication study. These intervals convey uncertainty in a way that p-values and standard confidence intervals do not, and can help researchers to draw sounder scientific conclus…
Variable-width confidence intervals in Gaussian regression and penalized maximum likelihood estimators
硬阈值、LASSO、自适应LASSO和SCAD点估计器已经被建议用于当回归参数向量的大多数分量被认为是零（稀疏型假设）时的线性回归上下文中。Potscher和Schneider，2010年，Electronic Journal of Statistics，考虑了固定宽度置信区间的属性，其中包括这些点估计量之一（对于所有可能的数据值）。他们认为一个正常的线性回归模型与正交回归，并表明，这些置信区间是长于标准置信区间（基于极大似然估计）时，这些点估计的调整参数选择，导致保守或一致的模型选择。我们将这种分析扩展到可变宽度置信区间的情况，其中包括这些点估计（对于所有可能的数据值）。与Potscher和Schneider的这些发现一致，我们发现这些置信区间与标准置信区间相比表现不佳，当选择这些点估计的调谐参数以导致一致的模型选择时。然而，当这些点估计的调整参数被选择，导致保守的模型选择，我们的结论不同于Potscher和Schneider。我们考虑Farchione和Kabaila，2008，Statistics & Probability Letters中的可变宽度置信区间，在稀疏性假设中存在信念的情况下，它比标准置信区间具有优势。这些可变宽度的置信区间，包括硬阈值，LASSO，自适应LASSO和SCAD估计（所有可能的数据值），这些估计的调谐参数被选择属于一个适当的区间。
Bootstrap smoothed (bagged) parameter estimators have been proposed as an improvement on estimators found after preliminary data-based model selection. The key result of Efron (2014) is a very convenient and widely applicable formula for a delta method approximation to the standard deviation of the bootstrap smoothed estimator. This approximation provides an easily computed guide to the accuracy of this estimator. In addition, Efron (2014) proposed a confidence interval centered on the bootstrap smoothed estimator, with width proportional to the estimate of this approximation to the standard d…
This short note argues that 95% confidence intervals for MCMC estimates can be obtained even without establishing a CLT, by multiplying their widths by 2.3.
A novel confidence interval estimator is proposed for the risk difference in noninferiority binomial trials. The confidence interval is consistent with an exact unconditional test that preserves the type-I error, and has improved power, particularly for smaller sample sizes, compared to the confidence interval by Chan & Zhang (1999). The improved performance of the proposed confidence interval is theoretically justified and demonstrated with simulations and examples. An R package is also distributed that implements the proposed methods along with other confidence interval estimators.
We consider a general regression model, without a scale parameter. Our aim is to construct a confidence interval for a scalar parameter of interest $θ$ that utilizes the uncertain prior information that a distinct scalar parameter $τ$ takes the specified value $t$. This confidence interval should have good coverage properties. It should also have scaled expected length, where the scaling is with respect to the usual confidence interval, that (a) is substantially less than 1 when the prior information is correct, (b) has a maximum value that is not too large and (c) is close to 1 when the data …
Introductory texts on statistics typically only cover the classical "two sigma" confidence interval for the mean value and do not describe methods to obtain confidence intervals for other estimators. The present technical report fills this gap by first defining different methods for the construction of confidence intervals, and then by their application to a binomial proportion, the mean value, and to arbitrary estimators. Beside the frequentist approach, the likelihood ratio and the highest posterior density approach are explained. Two methods to estimate the variance of general maximum likel…
Given $m$ unknown parameters with corresponding independent estimators, the Benjamini-Hochberg (BH) procedure can be used to classify the sign of parameters such that the expected proportion of erroneous directional decisions (directional FDR) is controlled at a preset level $q$. More ambitiously, our goal is to construct sign-determining confidence intervals---instead of only classifying the sign---such that the expected proportion of non-covering constructed intervals (FCR) is controlled. We suggest a valid procedure which adjusts a marginal confidence interval in order to construct a maximu…
This study aims to evaluate the performance of power in the likelihood ratio test for changepoint detection by bootstrap sampling, and proposes a hypothesis test based on bootstrapped confidence interval lengths. Assuming i.i.d normally distributed errors, and using the bootstrap method, the changepoint sampling distribution is estimated. Furthermore, this study describes a method to estimate a data set with no changepoint to form the null sampling distribution. With the null sampling distribution, and the distribution of the estimated changepoint, critical values and power calculations can be…
The goal of any estimation study is an interval estimation of a the parameter(s) of interest. These estimations are mostly expressed using empirical confidence intervals that are based on sample point estimates of the corresponding parameter(s). In contrast, calculations of the necessary sample size usually use expected confidence intervals that are based on the expected value of the parameter(s). The approach that guarantees the required probability of the required width of empirical confidence interval is known at least since 1989. However, till now, this approach is not implemented for most…
Interval estimation of the probability of success in a Binomial model is considered. Zieliński (2018) showed that the confidence interval which uses information about non-homogeneity of the sample is better than the classical one. In the following paper the shortest confidence interval for non-homogenous sample is constructed.
We congratulate the authors on their exciting paper, which introduces a novel idea for assessing the estimation bias in causal estimates. Doubly robust estimators are now part of the standard set of tools in causal inference, but a typical analysis stops with an estimate and a confidence interval. The authors give an approach for a unique type of model-checking that allows the user to check whether the bias is sufficiently small with respect to the standard error, which is generally required for confidence intervals to be reliable.
Confidence interval (CI) methods for stratified bilateral studies use intraclass correlation to avoid misleading results. In this article, we propose four CI methods (sample-size weighted global MLE-based Wald-type CI, complete MLE-based Wald-type CI, profile likelihood CI, and complete MLE-based score CI) to investigate CIs of proportion ratios to clinical trial design with stratified bilateral data under Dallal's intraclass model. Monte Carlo simulations are performed, and the complete MLE-based score confidence interval (CS) method yields a robust outcome. Lastly, a real data example is con…
This is the rejoinder to the discussion by Kennedy, Balakrishnan and Wasserman on the paper "On nearly assumption-free tests of nominal confidence interval coverage for causal parameters estimated by machine learning" published in Statistical Science.
Linear combinations of multinomial probabilities, such as those resulting from contingency tables, are of use when evaluating classification system performance. While large sample inference methods for these combinations exist, small sample methods exist only for regions on the multinomial parameter space instead of the linear combinations. However, in medical classification problems it is common to have small samples necessitating a small sample confidence interval on linear combinations of multinomial probabilities. Therefore, in this paper we derive an exact confidence interval, through the…
The problem of quantifying uncertainty about the locations of multiple change points by means of confidence intervals is addressed. The asymptotic distribution of the change point estimators obtained as the local maximisers of moving sum statistics is derived, where the limit distributions differ depending on whether the corresponding size of changes is local, i.e. tends to zero as the sample size increases, or fixed. A bootstrap procedure for confidence interval generation is proposed which adapts to the unknown magnitude of changes and guarantees asymptotic validity both for local and fixed …
In many statistical problems, several estimators are usually available for interval estimation of a parameter of interest, and hence, the selection of an appropriate estimator is important. The criterion for a good estimator is to have a high coverage probability close to the nominal level and a shorter interval length. However, these two concepts are in opposition to each other: high and low coverages are associated with longer and shorter interval lengths respectively. Some methods, such as bootstrap calibration, modify the nominal level to improve the coverage and thereby allow the selectio…
We present an efficient method of calculating exact confidence intervals for the hypergeometric parameter representing the number of "successes," or "special items," in the population. The method inverts minimum-width acceptance intervals after shifting them to make their endpoints nondecreasing while preserving their level. The resulting set of confidence intervals achieves minimum possible average size, and even in comparison with confidence sets not required to be intervals it attains the minimum possible cardinality most of the time, and always within $1$. The method compares favorably wit…
Confidence intervals for the population mean of normally distributed data are some of the most standard statistical outputs one might want from a database. In this work we give practical differentially private algorithms for this task. We provide five algorithms and then compare them to each other and to prior work. We give concrete, experimental analysis of their accuracy and find that our algorithms provide much more accurate confidence intervals than prior work. For example, in one setting (with ε = 0.1 and n = 2782) our algorithm yields an interval that is only 1/15th the size of the stand…
I propose a new type of confidence interval for correct asymptotic inference after using data to select a model of interest without assuming any model is correctly specified. This hybrid confidence interval is constructed by combining techniques from the selective inference and post-selection inference literatures to yield a short confidence interval across a wide range of data realizations. I show that hybrid confidence intervals have correct asymptotic coverage, uniformly over a large class of probability distributions that do not bound scaled model parameters. I illustrate the use of these …
