In high energy physics, a widely used method to treat systematic uncertainties in confidence interval calculations is based on combining a frequentist construction of confidence belts with a Bayesian treatment of systematic uncertainties. In this note we present a study of the coverage of this method for the standard Likelihood Ratio (aka Feldman & Cousins) construction for a Poisson process with known background and Gaussian or log-Normal distributed uncertainties in the background or signal efficiency. For uncertainties in the signal efficiency of upto 40 % we find over-coverage on the level of 2 to 4 % depending on the size of uncertainties and the region in signal space. Uncertainties in the background generally have smaller effect on the coverage. A considerable smoothing of the coverage curves is observed. A software package is presented which allows fast calculation of the confidence intervals for a variety of assumptions on shape and size of systematic uncertainties for different nuisance parameters. The calculation speed allows experimenters to test the coverage for their specific conditions.
We study the frequentist properties of confidence intervals computed by the method known to statisticians as the Profile Likelihood. It is seen that the coverage of these intervals is surprisingly good over a wide range of possible parameter values for important classes of problems, in particular whenever there are additional nuisance parameters with statistical or systematic errors. Programs are available for calculating these intervals.
The following questions are discussed: ``Why confidence intervals are a hot topic?''; ``Are confidence intervals objective?''; ``What is the usefulness of coverage?''; ``How to obtain useful information from experiment?''; ``The confidence level must be chosen independently from the knowledge of the data?''.
One way to incorporate systematic uncertainties into the calculation of confidence intervals is by integrating over probability density functions parametrizing the uncertainties. In this note we present a development of this method which takes into account uncertainties in the prediction of background processes, uncertainties in the signal detection efficiency and background efficiency and allows for a correlation between the signal and background detection efficiencies. We implement this method with the Feldman & Cousins unified approach with and without conditioning. We present studies of co…
The paper comments on properties of the so-called "Unified approach to the construction of classical confidence intervals", in which confidence intervals are computed in a Neyman construction using the likelihood ratio as ordering quantity. In particular, two of the main results of a paper by Feldman and Cousins (F&C) are discussed. It is shown that in the case of central intervals the so-called flip-flopping problem, occuring in the specific scenario where the experimenter decides to quote a standard upper limit or a confidence interval depending on the measurement, can be easily avoided by c…
Results of numerical procedure of constructing confidence intervals for parameter of the Poisson distribution of signal events in the presence of background events with known value of parameter of Poisson distribution are presented. It is shown that the used procedure has both the Bayesian and frequentist interpretations. Also the possibility to construct a continuous analogue of the Poisson distribution to search the bounds of confidence intervals for the parameter of the Poisson distribution is discussed.
In this paper we propose a procedure to evaluate Bayesian confidence intervals in counting experiments where both signal and background fluctuations are described by the Poisson statistics. The results obtained when the method is applied to the calculation of upper limits will also be illustrated.
When searching for new physics effects, collaborations will often wish to publish upper limits and intervals with a lower confidence level than the threshold they would set to claim an excess or a discovery. However, confidence intervals are typically constructed to provide constant coverage, or probability to contain the true value, with possible overcoverage if the random parameter is discrete. In particular, that means that the confidence interval will contain the $0$-signal case with the same frequency as the confidence level. This paper details a modification to the Feldman-Cousins method…
In this note we consider coverage of confidence intervals calculated with and without systematic uncertainties. These calculations follow the prescription originally proposed by Cousins & Highland but here extended to account for different shapes, size and types of systematic uncertainties. Also two different ordering schemes are considered: the Feldman & Cousins ordering and its variant where conditioning on the background expectation is applied as proposed by Roe & Woodroofe. Without uncertainties Feldman & Cousins method over-covers as expected because of the discreteness of the Poisson dis…
We present an algorithm which allows a fast numerical computation of Feldman-Cousins confidence intervals for Poisson processes, even when the number of background events is relatively large. This algorithm incorporates an appropriate treatment of the singularities that arise as a consequence of the discreteness of the variable.
We have investigated how uncertainties in the estimation of the detection efficiency affect the 90% confidence intervals in the unified approach for constructing confidence intervals. The study has been conducted for experiments where the number of detected events is large and can be described by a Gaussian probability density function. We also assume the detection efficiency has a Gaussian probability density and study the range of the relative uncertainties $σ_ε$ between 0 and 30%. We find that the confidence intervals provide proper coverage over a wide signal range and increase smoothly an…
It is shown that all the Frequentist methods are equivalent from a statistical point of view, but the physical significance of the confidence intervals depends on the method. The Bayesian Ordering method is presented and confronted with the Unified Approach in the case of a Poisson process with background. Some criticisms to both methods are answered. It is also argued that a general Frequentist method is not needed.
We consider the power to reject false values of the parameter in Frequentist methods for the calculation of confidence intervals. We connect the power with the physical significance (reliability) of confidence intervals for a parameter bounded to be non-negative. We show that the confidence intervals (upper limits) obtained with a (biased) method that near the boundary has large power in testing the parameter against larger alternatives and small power in testing the parameter against smaller alternatives are physically more significant. Considering the recently proposed methods with correct c…
Robert Machol's surprising result, that from a single observation it is possible to have finite length confidence intervals for the parameters of location-scale models, is re-produced and extended. Two previously unpublished modifications are included. First, Herbert Robbins nonparametric confidence interval is obtained. Second, I introduce a technique for obtaining confidence intervals for the scale parameter of finite length in the logarithmic metric. Keywords: Theory/Foundations , Estimation, Prior Distributions, Non-parametrics & Semi-parametrics Geometry of Inference, Confidence Intervals…
Comparisons are carried out of the confidence intervals constructed with Neyman's frequentist method and with the ΔL=1/2 likelihood method, using the example of low-statistics life time estimates.
1/f noise is very common but is difficult to handle in a metrological way. After having recalled the main characteristics of stongly correlated noise, this paper will determine relationships giving confidence intervals over the arithmetic mean and the linear drift parameters. A complete example of processing of an actual measurement sequence affected by 1/f noise will be given.
The possibility of construction of continuous analogue of Poisson distribution with the search of bounds of confidence intervals for parameter of Poisson distribution is discussed. Also, in the article is shown that the true value of a parameter of Poisson distribution for the observed value $\hat x$ has Gamma distribution with the scale parameter, which is equal to one, and the shape parameter, which is equal to $\hat x$. The results of numerical construction of confidence intervals are presented.
A priori bound for the parameter to be estimated is incorporated into confidence intervals within frequentistic approach in a straightforward and optimal fashion, ensuring the best resolution of non-boundary values as well as robustness for non-physical values of the estimator.
A results of numerical procedure for construction of confidence intervals for parameter of Poisson distribution for signal in the presence of background which has Poisson distribution with known value of parameter are presented. It is shown that the described procedure has both Bayesian and frequentist interpretations.
We investigate the relation between frequentist and Bayesian approaches. Namely, we find the "frequentist" Bayes prior π_{f}(λ,x_{obs}) = -\frac{\int_{-\infty}^{x_{obs}}\frac{\partial f(x,λ)}{\partial λ}dx}{f(x_{obs},λ)} (here f(x,λ) is the probability density) for which the results of frequentist and Bayes approaches to the determination of confidence intervals coincide. In many cases (but not always) the "frequentist" prior which reproduces frequentist results coincides with the Jeffreys prior.
Roe and Woodroofe (RW) have suggested that certain conditional probabilities be incorporated into the ``unified approach'' for constructing confidence intervals, previously described by Feldman and Cousins (FC). RW illustrated this conditioning technique using one of the two prototype problems in the FC paper, that of Poisson processes with background. The main effect was on the upper curve in the confidence belt. In this paper, we attempt to apply this style of conditioning to the other prototype problem, that of Gaussian errors with a bounded physical region. We find that the lower curve on …
In this note I go through the `proof' of frequentistic confidence intervals and show what it logically implies concerning the value of a physical quantity given an experimental observation (nothing).
Calculating the expected number of misclassified outcomes is a standard problem of particular interest for rare-event searches. The Clopper-Pearson method allows calculation of classical confidence intervals on the amount of misclassification if data are all drawn from the same binomial probability distribution. However, data is often better described by breaking it up into several bins, each represented by a different binomial distribution. We describe and provide an algorithm for calculating a classical confidence interval on the expected total number of misclassified events from several bin…
We review the methods of constructing confidence intervals that account for a priori information about one-sided constraints on the parameter being estimated. We show that the so-called method of sensitivity limit yields a correct solution of the problem. Derived are the solutions for the cases of a continuous distribution with non-negative estimated parameter and a discrete distribution, specifically a Poisson process with background. For both cases, the best upper limit is constructed that accounts for the a priori information. A table is provided with the confidence intervals for the parame…
The incorporation of systematic uncertainties into confidence interval calculations has been addressed recently in a paper by Conrad et al. (Physical Review D 67 (2003) 012002). In their work, systematic uncertainities in detector efficiencies and background flux predictions were incorporated following the hybrid frequentist-Bayesian prescription of Cousins and Highland, but using the likelihood ratio ordering of Feldman and Cousins in order to produce "unified" confidence intervals. In general, the resulting intervals behaved as one would intuitively expect, i.e. increased with increasing unc…
The evaluation of the error to be attributed to cut efficiencies is a common question in the practice of experimental particle physics. Specifically, the need to evaluate the efficiency of the cuts for background removal, when they are tested in a signal-free-background-only energy window, originates a statistical problem which finds its natural framework in the ample family of solutions for two classical, and closely related, questions, i.e. the determination of confidence intervals for the parameter of a binomial proportion and for the ratio of Poisson means. In this paper the problem is fir…
We consider the Roe-Woodroofe construction of confidence intervals for the case of a Poisson distributed variate where the mean is the sum of a known background and an unknown non-negative signal. We point out that the intervals do not have coverage in the usual sense but can be made to have such with a modification that does not affect the believability and other desirable features of this attractive construction. A similar modification can be used to provide coverage to the construction recently proposed by Cousins for the Gaussian-with-boundary problem.
