Maximum likelihood fits to data can be performed using binned data and unbinned data. The likelihood fits in either case produce only the fitted quantities but not the goodness of fit. With binned data, one can obtain a measure of the goodness of fit by using the $χ^2$ method, after the maximum likelihood fitting is performed. With unbinned data, currently, the fitted parameters are obtained but no measure of goodness of fit is available. This remains, to date, an unsolved problem in statistics. By considering the transformation properties of likelihood functions with respect to change of variable, we conclude that the likelihood ratio of the theoretically predicted probability density to that of {\it the data density} is invariant under change of variable and provides the goodness of fit. We show how to apply this likelihood ratio for binned as well as unbinned likelihoods and show that even the $χ^2$ test is a special case of this general theory. In order to calculate errors in the fitted quantities, we need to solve the problem of inverse probabilities. We use Bayes' theorem to do this, using the data density obtained in the goodness of fit. This permits one to invert the probabilities without the use of a Bayesian prior. The resulting statistics is consistent with frequentist ideas.
Maximum likelihood fits to data can be done using binned data (histograms) and unbinned data. With binned data, one gets not only the fitted parameters but also a measure of the goodness of fit. With unbinned data, currently, the fitted parameters are obtained but no measure of goodness of fit is available. This remains, to date, an unsolved problem in statistics. Using Bayes theorem and likelihood ratios, we provide a method by which both the fitted quantities and a measure of the goodness of fit are obtained for unbinned likelihood fits, as well as errors in the fitted quantities. We provide…
Maximum likelihood fits to data can be done using binned data (histograms) and unbinned data. With binned data, one gets not only the fitted parameters but also a measure of the goodness of fit. With unbinned data, currently, the fitted parameters are obtained but no measure of goodness of fit is available. This remains, to date, an unsolved problem in statistics. Using Bayes' theorem and likelihood ratios, we provide a method by which both the fitted quantities and a measure of the goodness of fit are obtained for unbinned likelihood fits, as well as errors in the fitted quantities. The quant…
The value of the likelihood is occasionally used by high energy physicists as a statistic to measure goodness-of-fit in unbinned maximum likelihood fits. Simple examples are presented that illustrate why this (seemingly intuitive) method fails in practice to achieve the desired goal.
Multivariate analyses play an important role in high energy physics. Such analyses often involve performing an unbinned maximum likelihood fit of a probability density function (p.d.f.) to the data. This paper explores a variety of unbinned methods for determining the goodness of fit of the p.d.f. to the data. The application and performance of each method is discussed in the context of a real-life high energy physics analysis (a Dalitz-plot analysis). Several of the methods presented in this paper can also be used for the non-parametric determination of whether two samples originate from the …
The coefficient of determination, known as $R^2$, is commonly used as a goodness-of-fit criterion for fitting linear models. $R^2$ is somewhat controversial when fitting nonlinear models, although it may be generalised on a case-by-case basis to deal with specific models such as the logistic model. Assume we are fitting a parametric distribution to a data set using, say, the maximum likelihood estimation method. A general approach to measure the goodness-of-fit of the fitted parameters, which is advocated herein, is to use a nonparametric measure for comparison between the empirical distributi…
A new method for calculation of goodness of multidimensional fits in particle physics experiments is proposed. This method finds the smallest and largest clusters of nearest neighbors for observed data points. The cluster size is used to estimate the goodness-of-fit and the cluster location provides clues about possible problems with data modeling. The performance of the new method is compared to that of the likelihood method and Kolmogorov-Smirnov test using toy Monte Carlo studies. The new method is applied to estimate the goodness-of-fit in a B->Kll analysis at BaBar.
Unbinned likelihood fits are frequent in Physics, and often involve complex functions with several components. We discuss the potential pitfalls of situations where the templates used in the fit are not fixed but depend on the event observables, as it happens when the resolution of the measurement is event--dependent, and the procedure to avoid them.
Parameter estimation via unbinned maximum likelihood fits is central for many analyses performed in high energy physics. Unbinned maximum likelihood fits using event weights, for example to statistically subtract background contributions via the sPlot formalism, or to correct for acceptance effects, have recently seen increasing use in the community. However, it is well known that the naive approach to the estimation of parameter uncertainties via the second derivative of the logarithmic likelihood does not yield confidence intervals with the correct coverage in the presence of event weights. …
A common goal in an experimental physics analysis is to extract information from a reaction with multi-dimensional kinematics. The preferred method for such a task is typically the unbinned maximum likelihood method. In fits using this method, the likelihood is a goodness-of-fit quantity in that it effectively discriminates between available hypotheses; however, it does not provide any information as to how well the best hypothesis describes the data. In this paper, we present an {\em ad-hoc} procedure for obtaining chi**2/n.d.f. values from unbinned maximum likelihood fits. This method does n…
The method of extended maximum likelihood is a well known concept of parameter estimation. One can implement external knowledge on the unknown parameters by multiplying the likelihood by constraint terms. In this note, we emphasize that this is also true for yield parameters in an extended maximum likelihood fit, which is widely used in the particle physics community. We recommend a way to generate pseudo-experiments in presence of constraint terms on yield parameters, and point to pitfalls inside the RooFit framework.
The possibility of strong biases in a multicomponent Maximum Likelihood fits with component-dependent templates has been demonstrated in some toy problems. We discuss here in detail a problem of practical interest, particle identification based on time-of-flight or dE/dx information. We show that large biases can occur in estimating particle fractions in a sample if differences between the momentum spectra of particles are ignored, and we present a more robust fit technique, allowing bias-free estimation even when the particle spectra in the sample are unknown.
Straightforward methods for adapting the familiar chi^2 statistic to histograms of discrete events and other Poisson distributed data generally yield biased estimates of the parameters of a model. The bias can be important even when the total number of events is large. For the case of estimating a microcalorimeter's energy resolution at 6 keV from the observed shape of the Mn K-alpha fluorescence spectrum, a poor choice of chi^2 can lead to biases of at least 10% in the estimated resolution when up to thousands of photons are observed. The best remedy is a Poisson maximum-likelihood fit, throu…
We consider the problem of estimating the population probability distribution given a finite set of multivariate samples, using the maximum entropy approach. In strict keeping with Jaynes' original definition, our precise formulation of the problem considers contributions only from the smoothness of the estimated distribution (as measured by its entropy) and the loss functional associated with its goodness-of-fit to the sample data, and in particular does not make use of any additional constraints that cannot be justified from the sample data alone. By mapping the general multivariate problem …
Metrics of model goodness-of-fit, model comparison, and model parameter estimation are the main categories of statistical problems in science. Bayesian and frequentist methods that address these questions often rely on a likelihood function, which is the key ingredient in order to assess the plausibility of model parameters given observed data. In some complex systems or experimental setups, predicting the outcome of a model cannot be done analytically, and Monte Carlo techniques are used. In this paper, we present a new analytic likelihood that takes into account Monte Carlo uncertainties, ap…
Binned maximum likelihood fits are an attractive option when analysing large datasets, but require care when computing likelihoods of continuous PDFs in bins. For many years the widely used statistical modelling package RooFit evaluated probabilities at the bin centre, leading to significant biases for strongly curved probability density functions. We demonstrate the biases with real-world examples, and introduce a PDF class to RooFit that removes these biases. The physics and computation performance of this new class are discussed.
Investigating the power density spectrum of fluctuations in Rayleigh-Taylor (RT) interfacial mixing is a means of studying characteristic length- and time-scales, anisotropies and anomalous processes. Guided by group theory, analysing the invariance-based properties of the fluctuations, our work examines raw time series from hot-wire anemometry measurements in the experiment by Akula et al., JFM 816, 619-660 (2017). The results suggest that the power density spectrum can be modelled as a compound function presented as the product of a power law and an exponential. The data analysis is based on…
Conventional statistics begins with a model, and assigns a likelihood of obtaining any particular set of data. The opposite approach, beginning with the data and assigning a likelihood to any particular model, is explored here for the case of points drawn randomly from a continuous probability distribution. A scalar field theory is used to assign a likelihood over the space of probability distributions. The most likely distribution may be calculated, providing an estimate of the underlying distribution and a convenient graphical representation of the raw data. Fluctuations around this maximum …
Likelihood ratio tests are performed for the hypothesis that charged-particle multiplicities measured in proton-(anti)proton collisions at $\sqrt{s}$ = 0.9 and 2.36 TeV are distributed according to the negative binomial form. Results indicate that the hypothesis should be rejected in the all cases of ALICE-LHC measurements in the limited pseudo-rapidity windows, whereas should be accepted in the corresponding cases of UA5 data. Possible explanations of that and of the disagreement with the least-squares fitting method are given.
Power-law distributions contain precious information about a large variety of processes in geoscience and elsewhere. Although there are sound theoretical grounds for these distributions, the empirical evidence in favor of power laws has been traditionally weak. Recently, Clauset et al. have proposed a systematic method to find over which range (if any) a certain distribution behaves as a power law. However, their method has been found to fail, in the sense that true (simulated) power-law tails are not recognized as such in some instances, and then the power-law hypothesis is rejected. Moreover…
The Well-Tempered Likelihood: Honest Confidence Intervals for Misspecified Models
Stanford University（斯坦福大学）；SLAC National Accelerator Laboratory（史丹福直线加速器中心）；Massachusetts Institute of Technology（麻省理工学院）；Institut des Hautes Études Scientifiques（高等科学研究所）；CEA Paris-Saclay（巴黎萨克雷原子能委员会）
