We study the frequentist properties of confidence intervals computed by the method known to statisticians as the Profile Likelihood. It is seen that the coverage of these intervals is surprisingly good over a wide range of possible parameter values for important classes of problems, in particular whenever there are additional nuisance parameters with statistical or systematic errors. Programs are available for calculating these intervals.
We discuss some issues arising in the evaluation of confidence intervals in the presence of nuisance parameters (systematic uncertainties) by means of direct Neyman construction in multi-dimensional space. While this kind of procedure provides rigorous coverage, it may be affected by large overcoverage, and/or produce results with counterintuitive behavior with respect to the uncertainty on the nuisance parameters, or other undesirable properties. We describe a choice of ordering algorithm that provides results with good general properties, the correct behavior for small uncertainties, and lim…
We address the common problem of calculating intervals in the presence of systematic uncertainties. We aim to investigate several approaches, but here describe just a Bayesian technique for setting upper limits. The particular example we study is that of inferring the rate of a Poisson process when there are uncertainties on the acceptance and the background. Limit calculating software associated with this work is available in the form of C functions.
In this note we consider coverage of confidence intervals calculated with and without systematic uncertainties. These calculations follow the prescription originally proposed by Cousins & Highland but here extended to account for different shapes, size and types of systematic uncertainties. Also two different ordering schemes are considered: the Feldman & Cousins ordering and its variant where conditioning on the background expectation is applied as proposed by Roe & Woodroofe. Without uncertainties Feldman & Cousins method over-covers as expected because of the discreteness of the Poisson dis…
The following questions are discussed: ``Why confidence intervals are a hot topic?''; ``Are confidence intervals objective?''; ``What is the usefulness of coverage?''; ``How to obtain useful information from experiment?''; ``The confidence level must be chosen independently from the knowledge of the data?''.
The paper comments on properties of the so-called "Unified approach to the construction of classical confidence intervals", in which confidence intervals are computed in a Neyman construction using the likelihood ratio as ordering quantity. In particular, two of the main results of a paper by Feldman and Cousins (F&C) are discussed. It is shown that in the case of central intervals the so-called flip-flopping problem, occuring in the specific scenario where the experimenter decides to quote a standard upper limit or a confidence interval depending on the measurement, can be easily avoided by c…
Results of numerical procedure of constructing confidence intervals for parameter of the Poisson distribution of signal events in the presence of background events with known value of parameter of Poisson distribution are presented. It is shown that the used procedure has both the Bayesian and frequentist interpretations. Also the possibility to construct a continuous analogue of the Poisson distribution to search the bounds of confidence intervals for the parameter of the Poisson distribution is discussed.
In this paper we propose a procedure to evaluate Bayesian confidence intervals in counting experiments where both signal and background fluctuations are described by the Poisson statistics. The results obtained when the method is applied to the calculation of upper limits will also be illustrated.
When searching for new physics effects, collaborations will often wish to publish upper limits and intervals with a lower confidence level than the threshold they would set to claim an excess or a discovery. However, confidence intervals are typically constructed to provide constant coverage, or probability to contain the true value, with possible overcoverage if the random parameter is discrete. In particular, that means that the confidence interval will contain the $0$-signal case with the same frequency as the confidence level. This paper details a modification to the Feldman-Cousins method…
We present an algorithm which allows a fast numerical computation of Feldman-Cousins confidence intervals for Poisson processes, even when the number of background events is relatively large. This algorithm incorporates an appropriate treatment of the singularities that arise as a consequence of the discreteness of the variable.
It is shown that all the Frequentist methods are equivalent from a statistical point of view, but the physical significance of the confidence intervals depends on the method. The Bayesian Ordering method is presented and confronted with the Unified Approach in the case of a Poisson process with background. Some criticisms to both methods are answered. It is also argued that a general Frequentist method is not needed.
We consider the power to reject false values of the parameter in Frequentist methods for the calculation of confidence intervals. We connect the power with the physical significance (reliability) of confidence intervals for a parameter bounded to be non-negative. We show that the confidence intervals (upper limits) obtained with a (biased) method that near the boundary has large power in testing the parameter against larger alternatives and small power in testing the parameter against smaller alternatives are physically more significant. Considering the recently proposed methods with correct c…
Robert Machol's surprising result, that from a single observation it is possible to have finite length confidence intervals for the parameters of location-scale models, is re-produced and extended. Two previously unpublished modifications are included. First, Herbert Robbins nonparametric confidence interval is obtained. Second, I introduce a technique for obtaining confidence intervals for the scale parameter of finite length in the logarithmic metric. Keywords: Theory/Foundations , Estimation, Prior Distributions, Non-parametrics & Semi-parametrics Geometry of Inference, Confidence Intervals…
Comparisons are carried out of the confidence intervals constructed with Neyman's frequentist method and with the ΔL=1/2 likelihood method, using the example of low-statistics life time estimates.
1/f noise is very common but is difficult to handle in a metrological way. After having recalled the main characteristics of stongly correlated noise, this paper will determine relationships giving confidence intervals over the arithmetic mean and the linear drift parameters. A complete example of processing of an actual measurement sequence affected by 1/f noise will be given.
The possibility of construction of continuous analogue of Poisson distribution with the search of bounds of confidence intervals for parameter of Poisson distribution is discussed. Also, in the article is shown that the true value of a parameter of Poisson distribution for the observed value $\hat x$ has Gamma distribution with the scale parameter, which is equal to one, and the shape parameter, which is equal to $\hat x$. The results of numerical construction of confidence intervals are presented.
A priori bound for the parameter to be estimated is incorporated into confidence intervals within frequentistic approach in a straightforward and optimal fashion, ensuring the best resolution of non-boundary values as well as robustness for non-physical values of the estimator.
We have investigated how uncertainties in the estimation of the detection efficiency affect the 90% confidence intervals in the unified approach for constructing confidence intervals. The study has been conducted for experiments where the number of detected events is large and can be described by a Gaussian probability density function. We also assume the detection efficiency has a Gaussian probability density and study the range of the relative uncertainties $σ_ε$ between 0 and 30%. We find that the confidence intervals provide proper coverage over a wide signal range and increase smoothly an…
A results of numerical procedure for construction of confidence intervals for parameter of Poisson distribution for signal in the presence of background which has Poisson distribution with known value of parameter are presented. It is shown that the described procedure has both Bayesian and frequentist interpretations.
Roe and Woodroofe (RW) have suggested that certain conditional probabilities be incorporated into the ``unified approach'' for constructing confidence intervals, previously described by Feldman and Cousins (FC). RW illustrated this conditioning technique using one of the two prototype problems in the FC paper, that of Poisson processes with background. The main effect was on the upper curve in the confidence belt. In this paper, we attempt to apply this style of conditioning to the other prototype problem, that of Gaussian errors with a bounded physical region. We find that the lower curve on …
Calculating the expected number of misclassified outcomes is a standard problem of particular interest for rare-event searches. The Clopper-Pearson method allows calculation of classical confidence intervals on the amount of misclassification if data are all drawn from the same binomial probability distribution. However, data is often better described by breaking it up into several bins, each represented by a different binomial distribution. We describe and provide an algorithm for calculating a classical confidence interval on the expected total number of misclassified events from several bin…
We investigate the relation between frequentist and Bayesian approaches. Namely, we find the "frequentist" Bayes prior π_{f}(λ,x_{obs}) = -\frac{\int_{-\infty}^{x_{obs}}\frac{\partial f(x,λ)}{\partial λ}dx}{f(x_{obs},λ)} (here f(x,λ) is the probability density) for which the results of frequentist and Bayes approaches to the determination of confidence intervals coincide. In many cases (but not always) the "frequentist" prior which reproduces frequentist results coincides with the Jeffreys prior.
In this note I go through the `proof' of frequentistic confidence intervals and show what it logically implies concerning the value of a physical quantity given an experimental observation (nothing).
We consider the Roe-Woodroofe construction of confidence intervals for the case of a Poisson distributed variate where the mean is the sum of a known background and an unknown non-negative signal. We point out that the intervals do not have coverage in the usual sense but can be made to have such with a modification that does not affect the believability and other desirable features of this attractive construction. A similar modification can be used to provide coverage to the construction recently proposed by Cousins for the Gaussian-with-boundary problem.
