We discuss some issues arising in the evaluation of confidence intervals in the presence of nuisance parameters (systematic uncertainties) by means of direct Neyman construction in multi-dimensional space. While this kind of procedure provides rigorous coverage, it may be affected by large overcoverage, and/or produce results with counterintuitive behavior with respect to the uncertainty on the nuisance parameters, or other undesirable properties. We describe a choice of ordering algorithm that provides results with good general properties, the correct behavior for small uncertainties, and limited overcoverage.
We study the frequentist properties of confidence intervals computed by the method known to statisticians as the Profile Likelihood. It is seen that the coverage of these intervals is surprisingly good over a wide range of possible parameter values for important classes of problems, in particular whenever there are additional nuisance parameters with statistical or systematic errors. Programs are available for calculating these intervals.
In this note we consider coverage of confidence intervals calculated with and without systematic uncertainties. These calculations follow the prescription originally proposed by Cousins & Highland but here extended to account for different shapes, size and types of systematic uncertainties. Also two different ordering schemes are considered: the Feldman & Cousins ordering and its variant where conditioning on the background expectation is applied as proposed by Roe & Woodroofe. Without uncertainties Feldman & Cousins method over-covers as expected because of the discreteness of the Poisson dis…
Inspired by the recent proposal by Feldman and Cousins of a ``unified approach to the classical statistical analysis of small signals'' based on a choice of ordering in Neyman's construction of classical confidence intervals, I propose a new ordering principle for the classical statistical analysis of Poisson processes with background which minimizes the effect on the resulting confidence intervals of the observation of less background events than expected. The new ordering principle is applied to the calculation of the confidence region implied by the recent null result of the KARMEN neutrino…
In high energy physics, a widely used method to treat systematic uncertainties in confidence interval calculations is based on combining a frequentist construction of confidence belts with a Bayesian treatment of systematic uncertainties. In this note we present a study of the coverage of this method for the standard Likelihood Ratio (aka Feldman & Cousins) construction for a Poisson process with known background and Gaussian or log-Normal distributed uncertainties in the background or signal efficiency. For uncertainties in the signal efficiency of upto 40 % we find over-coverage on the level…
It is shown that all the Frequentist methods are equivalent from a statistical point of view, but the physical significance of the confidence intervals depends on the method. The Bayesian Ordering method is presented and confronted with the Unified Approach in the case of a Poisson process with background. Some criticisms to both methods are answered. It is also argued that a general Frequentist method is not needed.
The behaviors of various confidence/credible interval constructions are explored, particularly in the region of low statistics where methods diverge most. We highlight a number of challenges, such as the treatment of nuisance parameters, and common misconceptions associated with such constructions. An informal survey of the literature suggests that confidence intervals are not always defined in relevant ways and are too often misinterpreted and/or misapplied. This can lead to seemingly paradoxical behaviours and flawed comparisons regarding the relevance of experimental results. We therefore c…
The paper comments on properties of the so-called "Unified approach to the construction of classical confidence intervals", in which confidence intervals are computed in a Neyman construction using the likelihood ratio as ordering quantity. In particular, two of the main results of a paper by Feldman and Cousins (F&C) are discussed. It is shown that in the case of central intervals the so-called flip-flopping problem, occuring in the specific scenario where the experimenter decides to quote a standard upper limit or a confidence interval depending on the measurement, can be easily avoided by c…
One way to incorporate systematic uncertainties into the calculation of confidence intervals is by integrating over probability density functions parametrizing the uncertainties. In this note we present a development of this method which takes into account uncertainties in the prediction of background processes, uncertainties in the signal detection efficiency and background efficiency and allows for a correlation between the signal and background detection efficiencies. We implement this method with the Feldman & Cousins unified approach with and without conditioning. We present studies of co…
The new Belle phi_3/gamma measurement arXiv:hep-ex/0604054, based on Dalitz analysis of D -> Kshort pi+ pi- in B+- -> D(*) K(*)+- decays, uses likelihood ratio ordering to set confidence intervals in phi_3 and the r,delta parameters. This is different to the choice made by BaBar in PRL 95, 121802 (2005) and arXiv:hep-ex/0507101, and requires additional computation. This Note explains Belle's choice using a related but simpler example: the averaging of two numbers. We find that intervals calculated with likelihood ratio ordering reproduce the analytic solution to this problem, whereas intervals…
INFN - Sezione di Padova（意大利国家核物理研究所帕多瓦分所 (INFN - Sezione di Padova)）
Complex computer simulations are commonly required for accurate data modelling in many scientific disciplines, making statistical inference challenging due to the intractability of the likelihood evaluation for the observed data. Furthermore, sometimes one is interested on inference drawn over a subset of the generative model parameters while taking into account model uncertainty or misspecification on the remaining nuisance parameters. In this work, we show how non-linear summary statistics can be constructed by minimising inference-motivated losses via stochastic gradient descent such they p…
Parameter estimation via unbinned maximum likelihood fits is central for many analyses performed in high energy physics. Unbinned maximum likelihood fits using event weights, for example to statistically subtract background contributions via the sPlot formalism, or to correct for acceptance effects, have recently seen increasing use in the community. However, it is well known that the naive approach to the estimation of parameter uncertainties via the second derivative of the logarithmic likelihood does not yield confidence intervals with the correct coverage in the presence of event weights. …
In this note we present studies of coverage and power for confidence intervals for a Poisson process with known background calculated using the Likelihood ratio (aka Feldman & Cousins) ordering with Bayesian treatment of uncertainties in nuisance parameters. We consider both the variant where the Bayesian integration is done in both the numerator and the denominator and the modification where the integration is done only in the numerator whereas in the denominator the likelihood is taken at the maximum likelihood estimate of the parameters. Furthermore we discuss how measurements can be combin…
The evaluation of the error to be attributed to cut efficiencies is a common question in the practice of experimental particle physics. Specifically, the need to evaluate the efficiency of the cuts for background removal, when they are tested in a signal-free-background-only energy window, originates a statistical problem which finds its natural framework in the ample family of solutions for two classical, and closely related, questions, i.e. the determination of confidence intervals for the parameter of a binomial proportion and for the ratio of Poisson means. In this paper the problem is fir…
The incorporation of systematic uncertainties into confidence interval calculations has been addressed recently in a paper by Conrad et al. (Physical Review D 67 (2003) 012002). In their work, systematic uncertainities in detector efficiencies and background flux predictions were incorporated following the hybrid frequentist-Bayesian prescription of Cousins and Highland, but using the likelihood ratio ordering of Feldman and Cousins in order to produce "unified" confidence intervals. In general, the resulting intervals behaved as one would intuitively expect, i.e. increased with increasing unc…
RooStats is a project to create advanced statistical tools required for the analysis of LHC data, with emphasis on discoveries, confidence intervals, and combined measurements. The idea is to provide the major statistical techniques as a set of C++ classes with coherent interfaces, so that can be used on arbitrary model and datasets in a common way. The classes are built on top of the RooFit package, which provides functionality for easily creating probability models, for analysis combinations and for digital publications of the results. We will present in detail the design and the implementat…
