One formulation of forensic identification of source problems is to determine the source of trace evidence, for instance, glass fragments found on a suspect for a crime. The current state of the science is to compute a Bayes factor (BF) comparing the marginal distribution of measurements of trace evidence under two competing propositions for whether or not the unknown source evidence originated from a specific source. The obvious problem with such an approach is the ability to tailor the prior distributions (placed on the features/parameters of the statistical model for the measurements of trace evidence) in favor of the defense or prosecution, which is further complicated by the fact that the typical number of measurements of trace evidence is typically sufficiently small that prior choice/specification has a strong influence on the value of the BF. To remedy this problem of prior specification and choice, we develop an alternative to the BF, within the framework of generalized fiducial inference (GFI), that we term a {\em generalized fiducial factor} (GFF). Furthermore, we demonstrate empirically, on the synthetic and real Netherlands Forensic Institute (NFI) casework data, deficiencies in the BF and classical/frequentist likelihood ratio (LR) approaches.
In works discussing the Bayesian paradigm for presenting forensic evidence in court, the concept of a `relevant population' is often mentioned, without a clear definition of what is meant, and without recommendations of how to select such populations. This note is to try to better understand this concept. Our analysis is intended to be general enough to be applicable to different forensic technologies and we shall consider both DNA profiling and speaker recognition as examples.
We introduce a Bayesian solution for the problem in forensic speaker recognition, where there may be very little background material for estimating score calibration parameters. We work within the Bayesian paradigm of evidence reporting and develop a principled probabilistic treatment of the problem, which results in a Bayesian likelihood-ratio as the vehicle for reporting weight of evidence. We show in contrast, that reporting a likelihood-ratio distribution does not solve this problem. Our solution is experimentally exercised on a simulated forensic scenario, using NIST SRE'12 scores, which …
Bayesian, frequentist and fiducial (BFF) inferences are much more congruous than they have been perceived historically in the scientific community (cf., Reid and Cox 2015; Kass 2011; Efron 1998). Most practitioners are probably more familiar with the two dominant statistical inferential paradigms, Bayesian inference and frequentist inference. The third, lesser known fiducial inference paradigm was pioneered by R.A. Fisher in an attempt to define an inversion procedure for inference as an alternative to Bayes' theorem. Although each paradigm has its own strengths and limitations subject to thei…
For more than a century, fingerprints have been used with considerable success to identify criminals or verify the identity of individuals. The categorical conclusion scheme used by fingerprint examiners, and more generally the inference process followed by forensic scientists, have been heavily criticised in the scientific and legal literature. Instead, scholars have proposed to characterise the weight of forensic evidence using the Bayes factor as the key element of the inference process. In forensic science, quantifying the magnitude of support is equally as important as determining which m…
When a latent shoeprint is discovered at a crime scene, forensic analysts inspect it for distinctive patterns of wear such as scratches and holes (known as accidentals) on the source shoe's sole. If its accidentals correspond to those of a suspect's shoe, the print can be used as forensic evidence to place the suspect at the crime scene. The strength of this evidence depends on the random match probability---the chance that a shoe chosen at random would match the crime scene print's accidentals. Evaluating random match probabilities requires an accurate model for the spatial distribution of ac…
Estimating the difference between two binomial proportions will be investigated, where Bayesian, frequentist and fiducial (BFF) methods will be considered. Three vague priors will be used, the Jeffreys prior, a divergence prior and the probability matching prior. A probability matching prior is a prior distribution under which the posterior probabilities of certain regions coincide with their coverage probabilities. Fiducial inference can be viewed as a procedure that obtains a measure on a parameter space while assuming less than what Bayesian inference does, i.e. no prior. Fisher introduced …
In the Bayesian paradigm for presenting forensic evidence to court, it is recommended that the weight of the evidence be summarized as a likelihood ratio (LR) between two opposing hypotheses of how the evidence could have been produced. Such LRs are necessarily based on probabilistic models, the parameters of which may be uncertain. It has been suggested by some authors that the value of the LR, being a function of the model parameters should therefore also be considered uncertain and that this uncertainty should be communicated to the court. In this tutorial, we consider a simple example of a…
The majority of the statisticians concluded many decades ago that fiducial inference was nonsensical to them. Hannig et al. (2016) and others have, however, contributed to a renewed interest and focus. Fiducial inference is similar to Bayesian analysis, but without requiring a prior. The prior information is replaced by assuming a particular data generating equation. Berger (1985) explains that Bayesian analysis and statistical decision theory are in harmony. Taraldsen and Lindqvist (2013) show that fiducial theory and statistical decision theory also play well together. The purpose of this te…
Forensic scientists are often criticised for the lack of quantitative support for the conclusions of their examinations. While scholars advocate for the use of a Bayes factor to quantify the weight of forensic evidence, it is often impossible to assign the necessary probability measures to perform likelihood-based inference for high-dimensional and complex data. To address this issue, we revisit a two-stage inference framework and leverage the properties of kernel functions to offer a method that allows for statistically supporting the inference of the identity of source of sets of trace and c…
We propose a way to construct fiducial distributions for a multidimensional parameter using a step-by-step conditional procedure related to the inferential importance of the components of the parameter. For discrete models, in which the non-uniqueness of the fiducial distribution is well known, we propose to use the geometric mean of the "extreme cases" and show its good behavior with respect to the more traditional arithmetic mean. Connections with the generalized fiducial inference approach developed by Hannig and with confidence distributions are also analyzed. The suggested procedure stron…
Over the past decade, the field of forensic science has received recommendations from the National Research Council of the U.S. National Academy of Sciences, the U.S. National Institute of Standards and Technology, and the U.S. President's Council of Advisors on Science and Technology to study the validity and reliability of forensic analyses. More specifically, these committees recommend estimation of the rates of occurrence of erroneous conclusions drawn from forensic analyses. "Black box" studies for the various subjective feature-based comparison methods are intended for this purpose. In g…
A principle is modified that underlies the theory of organic fiducial inference as this theory was presented in an earlier paper. This modification, which is arguably a natural one to make, allows Bayesian inference to sometimes have a minor role within the theory in question and, as a consequence, allows more information from the data to be incorporated into the way a full conditional fiducial density is defined in certain cases. The new version of the principle concerned is applied to examples that were analysed previously using the older version of this principle.
IT Security Infrastructures Lab, Friedrich-Alexander University Erlangen-N \"u rnberg, Germany
In multimedia forensics, learning-based methods provide state-of-the-art performance in determining origin and authenticity of images and videos. However, most existing methods are challenged by out-of-distribution data, i.e., with characteristics that are not covered in the training set. This makes it difficult to know when to trust a model, particularly for practitioners with limited technical background. In this work, we make a first step toward redesigning forensic algorithms with a strong focus on reliability. To this end, we propose to use Bayesian neural networks (BNN), which combine th…
Forensic science often involves the comparison of crime-scene evidence to a known-source sample to determine if the evidence and the reference sample came from the same source. Even as forensic analysis tools become increasingly objective and automated, final source identifications are often left to individual examiners' interpretation of the evidence. Each source identification relies on judgements about the features and quality of the crime-scene evidence that may vary from one examiner to the next. The current approach to characterizing uncertainty in examiners' decision-making has largely …
It is shown that the fiducial distribution in a group model, or more generally a quasigroup model, determines the optimal equivariant frequentist inference procedures. The proof does not rely on existence of invariant measures, and generalizes results corresponding to the choice of the right Haar measure as a Bayesian prior. Classical and more recent examples show that fiducial arguments can be used to give good candidates for exact or approximate confidence distributions. It is here suggested that the fiducial algorithm can be considered as an alternative to the Bayesian algorithm for the con…
The fiducial argument of Fisher (1973) has been described as his biggest blunder, but the recent review of Hannig et al. (2016) demonstrates the current and increasing interest in this brilliant idea. This short note analyses an example introduced by Seidenfeld (1992) where the fiducial distribution is restricted to a string. Keywords and phrases: Bayesian and fiducial inference, Restrictions on parameters, Uncertainty quantification, Epistemic probability, Statistics on a manifold.
The forensic science community has increasingly sought quantitative methods for conveying the weight of evidence. Experts from many forensic laboratories summarize their findings in terms of a likelihood ratio. Several proponents of this approach have argued that Bayesian reasoning proves it to be normative. We find this likelihood ratio paradigm to be unsupported by arguments of Bayesian decision theory, which applies only to personal decision making and not to the transfer of information from an expert to a separate decision maker. We further argue that decision theory does not exempt the pr…
In this study we illustrate a statistical approach to questioned document examination. Specifically, we consider the construction of three classifiers that predict the writer of a sample document based on categorical data. To evaluate these classifiers, we use a data set with a large number of writers and a small number of writing samples per writer. Since the resulting classifiers were found to have near perfect accuracy using leave-one-out cross-validation, we propose a novel Bayesian-based cross-validation method for evaluating the classifiers.
A goal in the forensic interpretation of scientific evidence is to make an inference about the source of a trace of unknown origin. The evidence is composed of the following three elements: (a) the trace of unknown origin, (b) a sample from a specific source, and (c) a collection of samples from the alternative source population. The inference process usually considers two propositions. The first proposition is usually referred to as the prosecution hypothesis and states that a given specific source is the actual source of the trace of unknown origin. The second, usually referred to as the def…
The likelihood ratio (LR) is largely used to evaluate the relative weight of forensic data regarding two hypotheses and for its assessment Bayesian methods are widespread in the forensic field. However, the Bayesian `recipe' for the LR presented in most of literature consists in plugging-in Bayesian estimates of the involved nuisance parameters into a frequentist-defined LR: frequentist and Bayesian methods are thus mixed, giving rise to solutions obtained by hybrid reasoning. This paper provides the derivation of a proper Bayesian approach to assess LR for the `rare type match problem', the s…
Under standard regularity assumptions, we provide simple approximations for specific classes of fiducial and confidence distributions and discuss their connections with objective Bayesian posteriors. For a real parameter the approximations are accurate at least to order O(1/n). For the mean parameter of a multivariate exponential family, our fiducial distribution is asymptotically normal and invariant to the importance ordering of the components of the mean parameter.
We study a regression characterization for the quadratic estimator of weak lensing, developed by Hu and Okamoto (2001,2002), for cosmic microwave background observations. This characterization motivates a modification of the quadratic estimator by an adaptive Wiener filter which uses the robust Bayesian techniques described in Strawderman (1971) and Berger (1980). This technique requires the user to propose a fiducial model for the spectral density of the unknown lensing potential but the resulting estimator is developed to be robust to misspecification of this model. The role of the fiducial …
Without assuming any pdf for some measured parameter, we derive a predictive pdf for the outcome of a second measurement, given the outcome of the first measurement and two common assumptions about the noise. These are that (1) it is additive, and (2) it is of some known pdf. The argument is based on a Bayesian analysis of the noise when no pdf is provided for the value of the parameter. In this way we avoid assuming an ad-hoc prior. We clarify how this method of direct predictive inference is distinct from fiducial prediction. We specify the distinct flaw in the fiducial argument, and outline…
In this paper, we discuss the construction of a multivariate generalisation of the Dirichlet-multinomial distribution. An example from forensic genetics in the statistical analysis of DNA mixtures motivates the study of this multivariate extension. In forensic genetics, adjustment of the match probabilities due to remote ancestry in the population is often done using the so-called θ-correction. This correction increases the probability of observing multiple copies of rare alleles and thereby reduces the weight of the evidence for rare genotypes. By numerical examples, we show how the θ-correct…
The fiducial coincides with the posterior in a group model equipped with the right Haar prior. This result is here generalized. For this the underlying probability space of Kolmogorov is replaced by a $σ$-finite measure space and fiducial theory is presented within this frame. Examples are presented that demonstrate that this also gives good alternatives to existing Bayesian sampling methods. It is proved that the results provided here for fiducial models imply that the theory of invariant measures for groups cannot be generalized directly to loops: There exist a smooth one-dimensional loop wh…
Many forensic genetics problems can be handled using structured systems of discrete variables, for which Bayesian networks offer an appealing practical modeling framework, and allow inferences to be computed by probability propagation methods. However, when standard assumptions are violated--for example, when allele frequencies are unknown, there is identity by descent or the population is heterogeneous--dependence is generated among founding genes, that makes exact calculation of conditional probabilities by propagation methods less straightforward. Here we illustrate different methodologies …
DNA is now routinely used in criminal investigations and court cases, although DNA samples taken at crime scenes are of varying quality and therefore present challenging problems for their interpretation. We present a statistical model for the quantitative peak information obtained from an electropherogram (EPG) of a forensic DNA sample and illustrate its potential use for the analysis of criminal cases. In contrast to most previously used methods, we directly model the peak height information and incorporates important artefacts associated with the production of the EPG. Our model has a numbe…
In statistics, there are a variety of methods for performing model selection that all stem from slightly different paradigms of statistical inference. The reasons for choosing one particular method over another seem to be based entirely on philosophical preferences. In the case of non-nested model selection, two of the prevailing techniques are the Bayes Factor and the likelihood ratio. This article focuses on reconciling the likelihood ratio and the Bayes Factor for comparing a pair of non-nested models under two different problem frameworks typical in forensic science, the common-source and …
Dust particles recovered from the soles of shoes may be indicative of the sites recently visited by an individual, and, in particular, of the presence of an individual at a particular site of interest, e.g., the scene of a crime. By describing the dust profile of a given site by a multinomial distribution over a fixed number of dust particle types, we can define the probability distribution of the mixture of dust recovered from the sole of a shoe via Latent Dirichlet Allocation. We use Variational Bayesian Inference to study the parameters of the model, and use their resulting posterior distri…
In forensic DNA calculations of relatedness of individuals and in DNA mixture analyses, two sources of uncertainty are present concerning the allele frequencies used for evaluating genotype probabilities when evaluating likelihoods. They are: (i) imprecision in the estimates of the allele frequencies in the population by using an inevitably finite database of DNA profiles to estimate them; and (ii) the existence of population substructure. Green and Mortera (2009) showed that these effects may be taken into account individually using a common Dirichlet model within a Bayesian network formulati…
For several decades, legal and scientific scholars have argued that conclusions from forensic examinations should be supported by statistical data and reported within a probabilistic framework. Multiple models have been proposed to quantify the probative value of forensic evidence. Unfortunately, several of these models rely on ad-hoc strategies that are not scientifically sound. The opacity of the technical jargon used to present these models and their results, and the complexity of the techniques involved make it very difficult for the untrained user to separate the wheat from the chaff. Thi…
The fiducial is not unique in general, but we prove that in a restricted class of models it is uniquely determined by the sampling distribution of the data. It depends in particular not on the choice of a data generating model. The arguments lead to a generalization of the classical formula found by Fisher (1930). The restricted class includes cases with discrete distributions, the case of the shape parameter in the Gamma distribution, and also the case of the correlation coefficient in a bivariate Gaussian model. One of the examples can also be used in a pedagogical context to demonstrate pos…
We formalize the idea of probability distributions that lead to reliable predictions about some, but not all aspects of a domain. The resulting notion of `safety' provides a fresh perspective on foundational issues in statistics, providing a middle ground between imprecise probability and multiple-prior models on the one hand and strictly Bayesian approaches on the other. It also allows us to formalize fiducial distributions in terms of the set of random variables that they can safely predict, thus taking some of the sting out of the fiducial idea. By restricting probabilistic inference to saf…
The Pitman-Yor process is a random discrete probability distribution of which the atoms can be used to model the relative abundance of species. The process is indexed by a type parameter $σ$, which controls the number of different species in a finite sample from a realization of the distribution. A random sample of size $n$ from the Pitman-Yor process of type $σ>0$ will contain of the order $n^σ$ distinct values (``species''). In this paper we consider the estimation of the type parameter by both empirical Bayes and full Bayes methods. We derive the asymptotic normality of the empirical Bayes …
Under a multinormal distribution with arbitrary unknown covariance matrix, the main purpose of this paper is to propose a framework to achieve the goal of reconciliation of Bayesian, frequentist and Fisherian paradigms for the problems of testing mean against restricted alternatives (closed convex cones). Combining Fisher's fiducial inference and Wald's decision theory via d-admissibility into an unified approach, the goal can then be achieved. To proceed, the tests constructed via the union-intersection principle are studied.
While linear mixed modeling methods are foundational concepts introduced in any statistical education, adequate general methods for interval estimation involving models with more than a few variance components are lacking, especially in the unbalanced setting. Generalized fiducial inference provides a possible framework that accommodates this absence of methodology. Under the fabric of generalized fiducial inference along with sequential Monte Carlo methods, we present an approach for interval estimation for both balanced and unbalanced Gaussian linear mixed models. We compare the proposed met…
Study of the bivariate normal distribution raises the full range of issues involving objective Bayesian inference, including the different types of objective priors (e.g., Jeffreys, invariant, reference, matching), the different modes of inference (e.g., Bayesian, frequentist, fiducial) and the criteria involved in deciding on optimal objective priors (e.g., ease of computation, frequentist performance, marginalization paradoxes). Summary recommendations as to optimal objective priors are made for a variety of inferences involving the bivariate normal distribution. In the course of the investi…
Reacting against the limitation of statistics to decision procedures, R. A. Fisher proposed for inductive reasoning the use of the fiducial distribution, a parameter-space distribution of epistemological probability transferred directly from limiting relative frequencies rather than computed according to the Bayes update rule. The proposal is developed as follows using the confidence measure of a scalar parameter of interest. (With the restriction to one-dimensional parameter space, a confidence measure is essentially a fiducial probability distribution free of complications involving ancillar…
Posterior probabilistic statistical inference without priors is an important but so far elusive goal. Fisher's fiducial inference, Dempster-Shafer theory of belief functions, and Bayesian inference with default priors are attempts to achieve this goal but, to date, none has given a completely satisfactory picture. This paper presents a new framework for probabilistic inference, based on inferential models (IMs), which not only provides data-dependent probabilistic measures of uncertainty about the unknown parameter, but does so with an automatic long-run frequency calibration property. The key…
Due to their accuracies, methods based on ensembles of regression trees are a popular approach for making predictions. Some common examples include Bayesian additive regression trees, boosting and random forests. This paper focuses on honest random forests, which add honesty to the original form of random forests and are proved to have better statistical properties. The main contribution is a new method that quantifies the uncertainties of the estimates and predictions produced by honest random forests. The proposed method is based on the generalized fiducial methodology, and provides a fiduci…
