Optimal statistical inference in the presence of systematic uncertainties using neural network optimization based on binned Poisson likelihoods with nuisance parameters
Stefan Wunsch, Simon Jörger, Roger Wolf, Günter Quast
Data analysis in science, e.g., high-energy particle physics, is often subject to an intractable likelihood if the observables and observations span a high-dimensional input space. Typically the problem is solved by reducing the dimensionality using feature engineering and histograms, whereby the latter technique allows to build the likelihood using Poisson statistics. However, in the presence of systematic uncertainties represented by nuisance parameters in the likelihood, the optimal dimensionality reduction with a minimal loss of information about the parameters of interest is not known. This work presents a novel strategy to construct the dimensionality reduction with neural networks for feature engineering and a differential formulation of histograms so that the full workflow can be optimized with the result of the statistical inference, e.g., the variance of a parameter of interest, as objective. We discuss how this approach results in an estimate of the parameters of interest that is close to optimal and the applicability of the technique is demonstrated with a simple example based on pseudo-experiments and a more complex example from high-energy particle physics.
GollumFit: An IceCube Open-Source Framework for Binned-Likelihood Neutrino Telescope Analyses
RWTH Aachen University（亚琛工业大学）；University of Adelaide（阿德莱德大学）；University of Alaska Anchorage（阿拉斯加安克雷奇大学）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Georgia Institute of Technology（佐治亚理工学院）；Southern University（南方大学）；University of California, Berkeley（加州大学伯克利分校）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；Universit{\'e} Libre de Bruxelles（布鲁塞尔自由大学）；Simon Fraser University（西蒙菲莎大学）；Harvard University（哈佛大学）；Massachusetts Institute of Technology（麻省理工学院）；Chiba University（千叶大学）；Loyola University Chicago（芝加哥洛约拉大学）；University of Canterbury（坎特伯雷大学）；University of Maryland（马里兰大学）；Ohio State University（俄亥俄州立大学）；University of Copenhagen（哥本哈根大学）；TU Dortmund University（多特蒙德工业大学）；Michigan State University（密歇根州立大学）；University of Alberta（阿尔伯塔大学）；Universit{\'e} de Gen{\
We address the common problem of calculating intervals in the presence of systematic uncertainties. We aim to investigate several approaches, but here describe just a Bayesian technique for setting upper limits. The particular example we study is that of inferring the rate of a Poisson process when there are uncertainties on the acceptance and the background. Limit calculating software associated with this work is available in the form of C functions.
Many astrophysical phenomena are time-varying, in the sense that their intensity, energy spectrum, and/or the spatial distribution of the emission suddenly change. This paper develops a method for modeling a time series of images. Under the assumption that the arrival times of the photons follow a Poisson process, the data are binned into 4D grids of voxels (time, energy band, and x-y coordinates), and viewed as a time series of non-homogeneous Poisson images. The method assumes that at each time point, the corresponding multi-band image stack is an unknown 3D piecewise constant function inclu…
In this note we consider coverage of confidence intervals calculated with and without systematic uncertainties. These calculations follow the prescription originally proposed by Cousins & Highland but here extended to account for different shapes, size and types of systematic uncertainties. Also two different ordering schemes are considered: the Feldman & Cousins ordering and its variant where conditioning on the background expectation is applied as proposed by Roe & Woodroofe. Without uncertainties Feldman & Cousins method over-covers as expected because of the discreteness of the Poisson dis…
Targeted maximum likelihood estimation is a general methodology combining flexible ensemble learning and semiparametric efficiency theory in a two-step procedure for estimation of causal parameters. Proposed targeted maximum likelihood procedures for survival and competing risks analysis have so far focused on events taken values in discrete time. We here present a targeted maximum likelihood estimation procedure for event times that take values in R+. We focuson the estimation of intervention-specific mean outcomes with stochastic interventions on a time-fixed treatment. For data-adaptive est…
We introduce the DNNLikelihood, a novel framework to easily encode, through Deep Neural Networks (DNN), the full experimental information contained in complicated likelihood functions (LFs). We show how to efficiently parametrise the LF, treated as a multivariate function of parameters and nuisance parameters with high dimensionality, as an interpolating function in the form of a DNN predictor. We do not use any Gaussian approximation or dimensionality reduction, such as marginalisation or profiling over nuisance parameters, so that the full experimental information is retained. The procedure …
Counting experiments often rely on Monte Carlo simulations for predictions of Poisson expectations. The accompanying uncertainty from the finite Monte Carlo sample size can be incorporated into parameter estimation by modifying the Poisson likelihood. We first review previous Frequentist methods of this type by Barlow et al, Bohm et al, and Chirkin, as well as recently proposed probabilistic methods by the author and Argüelles et al. We show that all these approaches can be understood in a unified way: they all approximate the underlying probability distribution of the sum of weights in a give…
A method is discussed that allows combining sets of differential or inclusive measurements. It is assumed that at least one measurement was obtained with simultaneously fitting a set of nuisance parameters, representing sources of systematic uncertainties. As a result of beneficial constraints from the data all such fitted parameters are correlated among each other. The best approach for a combination of these measurements would be the maximisation of a combined likelihood, for which the full fit model of each measurement and the original data are required. However, only in rare cases this inf…
In this note we present studies of coverage and power for confidence intervals for a Poisson process with known background calculated using the Likelihood ratio (aka Feldman & Cousins) ordering with Bayesian treatment of uncertainties in nuisance parameters. We consider both the variant where the Bayesian integration is done in both the numerator and the denominator and the modification where the integration is done only in the numerator whereas in the denominator the likelihood is taken at the maximum likelihood estimate of the parameters. Furthermore we discuss how measurements can be combin…
In high energy physics, a widely used method to treat systematic uncertainties in confidence interval calculations is based on combining a frequentist construction of confidence belts with a Bayesian treatment of systematic uncertainties. In this note we present a study of the coverage of this method for the standard Likelihood Ratio (aka Feldman & Cousins) construction for a Poisson process with known background and Gaussian or log-Normal distributed uncertainties in the background or signal efficiency. For uncertainties in the signal efficiency of upto 40 % we find over-coverage on the level…
