Walley's Imprecise Dirichlet Model (IDM) for categorical data overcomes several fundamental problems which other approaches to uncertainty suffer from. Yet, to be useful in practice, one needs efficient ways for computing the imprecise=robust sets or intervals. The main objective of this work is to derive exact, conservative, and approximate, robust and credible interval estimates under the IDM for a large class of statistical estimators, including the entropy and mutual information.
Walley's Imprecise Dirichlet Model (IDM) for categorical i.i.d. data extends the classical Dirichlet model to a set of priors. It overcomes several fundamental problems which other approaches to uncertainty suffer from. Yet, to be useful in practice, one needs efficient ways for computing the imprecise=robust sets or intervals. The main objective of this work is to derive exact, conservative, and approximate, robust and credible interval estimates under the IDM for a large class of statistical estimators, including the entropy and mutual information.
We present an alternative approach to the Bayesian nonparametric analysis of conditional species richness under two-parameter Poisson Dirichlet priors. We rely on a known characterization by deletion of classes property and on results for Beta-Binomial distributions. Besides leading to simplified and much more direct proofs, our proposal provides a new scale mixture representation of the conditional asymptotic law.
This paper explores semi-qualitative probabilistic networks (SQPNs) that combine numeric and qualitative information. We first show that exact inferences with SQPNs are NPPP-Complete. We then show that existing qualitative relations in SQPNs (plus probabilistic logic and imprecise assessments) can be dealt effectively through multilinear programming. We then discuss learning: we consider a maximum likelihood method that generates point estimates given a SQPN and empirical data, and we describe a Bayesian-minded method that employs the Imprecise Dirichlet Model to generate set-valued estimates.
A Bayesian belief network is a model of a joint distribution over a finite set of variables, with a DAG structure representing immediate dependencies among the variables. For each node, a table of parameters (CPtable) represents local conditional probabilities, with rows indexed by conditioning events (assignments to parents). CP-table rows are usually modeled as independent random vectors, each assigned a Dirichlet prior distribution. The assumption that rows are independent permits a relatively simple analysis but may not reflect actual prior opinion about the parameters. Rows representing s…
Favaro, Lijoi, and Pruenster (2012, Biometrics, 68, 1188--1196) derive a novel Bayesian nonparametric estimator of the probability of detecting at the $(n+m+1)$th observation a species already observed with any given frequency in an enlarged sample of size $n+m$, conditionally on a basic sample of size $n$. Unfortunately the general result under Gibbs priors (Theorem 2), and consequently the explicit result under $(α, θ)$ Poisson-Dirichlet priors (Proposition 3), appear to be wrong. Here we provide the correct formulas for both the results, obtained by means of a new technique devised in Cerqu…
Mixture models are well-known for their versatility, and the Bayesian paradigm is a suitable platform for mixture analysis, particularly when the number of components is unknown. Bhattacharya (2008) introduced a mixture model based on the Dirichlet process, where an upper bound on the unknown number of components is to be specified. Here we consider a Bayesian asymptotic framework for objectively specifying the upper bound, which we assume to depend on the sample size. In particular, we define a Bayesian analogue of the mean integrated squared error (Bayesian MISE), and select that form of the…
We consider the problem of estimating the transition rate matrix of a continuous-time Markov chain from a finite-duration realisation of this process. We approach this problem in an imprecise probabilistic framework, using a set of prior distributions on the unknown transition rate matrix. The resulting estimator is a set of transition rate matrices that, for reasons of conjugacy, is easy to find. To determine the hyperparameters for our set of priors, we reconsider the problem in discrete time, where we can use the well-known Imprecise Dirichlet Model. In particular, we show how the limit of …
The compound Poisson process and the Dirichlet process are the pillar structures of Renewal theory and Bayesian nonparametric theory, respectively. Both processes have many useful extensions to fulfill the practitioners needs to model the particularities of data structures. Accordingly, in this contribution, we joined their primal ideas to construct the compound Dirichlet process and the compound Dirichlet process mixture. As a consequence, these new processes had a fruitful structure to model the time occurrence among events, with also a flexible structure on the arrival variables. These mode…
Online estimation and modelling of i.i.d. data for short sequences over large or complex "alphabets" is a ubiquitous (sub)problem in machine learning, information theory, data compression, statistical language processing, and document analysis. The Dirichlet-Multinomial distribution (also called Polya urn scheme) and extensions thereof are widely applied for online i.i.d. estimation. Good a-priori choices for the parameters in this regime are difficult to obtain though. I derive an optimal adaptive choice for the main parameter via tight, data-dependent redundancy bounds for a related model. T…
We introduce a Bayesian estimator of the underlying class structure in the stochastic block model, when the number of classes is known. The estimator is the posterior mode corresponding to a Dirichlet prior on the class proportions, a generalized Bernoulli prior on the class labels, and a beta prior on the edge probabilities. We show that this estimator is strongly consistent when the expected degree is at least of order $\log^2{n}$, where $n$ is the number of nodes in the network.
This article focuses on the performance of Bayes estimators, in comparison with the MLE, in multinomial models with a relatively large number of cells. The prior for the Bayes estimator is taken to be the conjugate Dirichlet, i.e., the multivariate Beta, with exchangeable distributions over the coordinates, including the non-informative uniform distribution. The choice of the multinomial is motivated by its many applications in business and industry, but also by its use in providing a simple nonparametric estimator of an unknown distribution. It is striking that the Bayes procedure outperforms…
We consider the classic problem of estimating T, the total number of species in a population, from repeated counts in a simple random sample. We look first at the Chao-Lee estimator: we initially show that such estimator can be obtained by reconciling two estimators of the unobserved probability, and then develop a sequence of improvements culminating in a Dirichlet prior Bayesian reinterpretation of the estimation problem. By means of this, we obtain simultaneous estimates of T, of the normalized interspecies variance $γ^2$ and of the parameter $λ$ of the prior. Several simulations show that …
We develop a theoretical framework for studying numerical estimation of lower previsions, generally applicable to two-level Monte Carlo methods, importance sampling methods, and a wide range of other sampling methods one might devise. We link consistency of these estimators to Glivenko-Cantelli classes, and for the sub-Gaussian case we show how the correlation structure of this process can be used to bound the bias and prove consistency. We also propose a new upper estimator, which can be used along with the standard lower estimator, in order to provide a simple confidence interval. As a case …
In a standard Bayesian approach to the alpha-factor model for common-cause failure, a precise Dirichlet prior distribution models epistemic uncertainty in the alpha-factors. This Dirichlet prior is then updated with observed data to obtain a posterior distribution, which forms the basis for further inferences. In this paper, we adapt the imprecise Dirichlet model of Walley to represent epistemic uncertainty in the alpha-factors. In this approach, epistemic uncertainty is expressed more cautiously via lower and upper expectations for each alpha-factor, along with a learning parameter which dete…
We present a novel approach for estimating conditional probability tables, based on a joint, rather than independent, estimate of the conditional distributions belonging to the same table. We derive exact analytical expressions for the estimators and we analyse their properties both analytically and via simulation. We then apply this method to the estimation of parameters in a Bayesian network. Given the structure of the network, the proposed approach better estimates the joint distribution and significantly improves the classification performance with respect to traditional approaches.
In a smooth semiparametric model, the marginal posterior distribution of the finite dimensional parameter of interest is expected to be asymptotically equivalent to the sampling distribution of frequentist's efficient estimators. This is the assertion of the so-called Bernstein-von Mises theorem, and recently, it has been proved in many interesting semiparametric models. In this thesis, we consider the semiparametric Bernstein-von Mises theorem in some models which have symmetric errors. The simplest example of these models is the symmetric location model that has 1-dimensional location parame…
Discrete random probability measures and the exchangeable random partitions they induce are key tools for addressing a variety of estimation and prediction problems in Bayesian inference. Indeed, many popular nonparametric priors, such as the Dirichlet and the Pitman-Yor process priors, select discrete probability distributions almost surely and, therefore, automatically induce exchangeable random partitions. Here we focus on the family of Gibbs-type priors, a recent and elegant generalization of the Dirichlet and the Pitman-Yor process priors. These random probability measures share propertie…
We propose an effective method to solve the event sequence clustering problems based on a novel Dirichlet mixture model of a special but significant type of point processes --- Hawkes process. In this model, each event sequence belonging to a cluster is generated via the same Hawkes process with specific parameters, and different clusters correspond to different Hawkes processes. The prior distribution of the Hawkes processes is controlled via a Dirichlet distribution. We learn the model via a maximum likelihood estimator (MLE) and propose an effective variational Bayesian inference algorithm.…
In Bayesian nonparametric inference, random discrete probability measures are commonly used as priors within hierarchical mixture models for density estimation and for inference on the clustering of the data. Recently, it has been shown that they can also be exploited in species sampling problems: indeed they are natural tools for modeling the random proportions of species within a population thus allowing for inference on various quantities of statistical interest. For applications that involve large samples, the exact evaluation of the corresponding estimators becomes impracticable and, ther…
Suppose $X_1,\dots, X_n$ is a random sample from a bounded and decreasing density $f_0$ on $[0,\infty)$. We are interested in estimating such $f_0$, with special interest in $f_0(0)$. This problem is encountered in various statistical applications and has gained quite some attention in the statistical literature. It is well known that the maximum likelihood estimator is inconsistent at zero. This has led several authors to propose alternative estimators which are consistent. As any decreasing density can be represented as a scale mixture of uniform densities, a Bayesian estimator is obtained b…
The problem of estimating discovery probabilities originated in the context of statistical ecology, and in recent years it has become popular due to its frequent appearance in challenging applications arising in genetics, bioinformatics, linguistics, designs of experiments, machine learning, etc. A full range of statistical approaches, parametric and nonparametric as well as frequentist and Bayesian, has been proposed for estimating discovery probabilities. In this paper we investigate the relationships between the celebrated Good-Turing approach, which is a frequentist nonparametric approach …
Usually, methods evaluating system reliability require engineers to quantify the reliability of each of the system components. For series and parallel systems, there are some options to handle the estimation of each component's reliability. We will treat the reliability estimation of complex problems of two classes of coherent systems: series-parallel, and parallel-series. In both of the cases, the component reliabilities may be unknown. We will present estimators for reliability functions at all levels of the system (component and system reliabilities). Nonparametric Bayesian estimators of al…
We study theoretically, for the first time, the Dirichlet kernel estimator introduced by Aitchison and Lauder (1985) for the estimation of multivariate densities supported on the $d$-dimensional simplex. The simplex is an important case as it is the natural domain of compositional data and has been neglected in the literature on asymmetric kernels. The Dirichlet kernel estimator, which generalizes the (non-modified) unidimensional Beta kernel estimator from Chen (1999), is free of boundary bias and non-negative everywhere on the simplex. We show that it achieves the optimal convergence rate $O…
In a smooth semi-parametric model, the marginal posterior distribution for a finite dimensional parameter of interest is expected to be asymptotically equivalent to the sampling distribution of any efficient point-estimator. The assertion leads to asymptotic equivalence of credible and confidence sets for the parameter of interest and is known as the semi-parametric Bernstein-von Mises theorem. In recent years, it has received much attention and has been applied in many examples. We consider models in which errors with symmetric densities play a role; more specifically, it is shown that the ma…
This paper considers the asymptotic behavior in $β$-Hölder spaces, and under $L^p$ losses, of a Dirichlet kernel density estimator proposed by Aitchison and Lauder (1985) for the analysis of compositional data. In recent work, Ouimet and Tolosana-Delgado (2022) established the uniform strong consistency and asymptotic normality of this estimator. As a complement, it is shown here that the Aitchison-Lauder estimator can achieve the minimax rate asymptotically for a suitable choice of bandwidth whenever $(p,β) \in [1, 3) \times (0, 2]$ or $(p, β) \in \mathcal{A}_d$, where $\mathcal{A}_d$ is a sp…
Bayesian density deconvolution using nonparametric prior distributions is a useful alternative to the frequentist kernel based deconvolution estimators due to its potentially wide range of applicability, straightforward uncertainty quantification and generalizability to more sophisticated models. This article is the first substantive effort to theoretically quantify the behavior of the posterior in this recent line of research. In particular, assuming a known supersmooth error density, a Dirichlet process mixture of Normals on the true density leads to a posterior convergence rate same as the …
A. Alikhanyan National Laboratory (Yerevan Physics Institute)（A·阿利哈尼扬国家实验室（埃里温物理研究所））
Maximum entropy (MAXENT) method has a large number of applications in theoretical and applied machine learning, since it provides a convenient non-parametric tool for estimating unknown probabilities. The method is a major contribution of statistical physics to probabilistic inference. However, a systematic approach towards its validity limits is currently missing. Here we study MAXENT in a Bayesian decision theory set-up, i.e. assuming that there exists a well-defined prior Dirichlet density for unknown probabilities, and that the average Kullback-Leibler (KL) distance can be employed for dec…
Given a sample of size $n$ from a population of individuals belonging to different species with unknown proportions, a popular problem of practical interest consists in making inference on the probability $D_{n}(l)$ that the $(n+1)$-th draw coincides with a species with frequency $l$ in the sample, for any $l=0,1,\ldots,n$. This paper contributes to the methodology of Bayesian nonparametric inference for $D_{n}(l)$. Specifically, under the general framework of Gibbs-type priors we show how to derive credible intervals for a Bayesian nonparametric estimation of $D_{n}(l)$, and we investigate th…
Penalized B-splines are routinely used in additive models to describe smooth changes in a response with quantitative covariates. It is typically done through the conditional mean in the exponential family using generalized additive models with an indirect impact on other conditional moments. Another common strategy consists in focussing on several low-order conditional moments, leaving the complete conditional distribution unspecified. Alternatively, a multi-parameter distribution could be assumed for the response with several of its parameters jointly regressed on covariates using additive ex…
Shrinkage prior has gained great successes in many data analysis, however, its applications mostly focus on the Bayesian modeling of sparse parameters. In this work, we will apply Bayesian shrinkage to model high dimensional parameter that possesses an unknown blocking structure. We propose to impose heavy-tail shrinkage prior, e.g., $t$ prior, on the differences of successive parameter entries, and such a fusion prior will shrink successive differences towards zero and hence induce posterior blocking. Comparing to conventional Bayesian fused lasso which implements Laplace fusion prior, $t$ fu…
Functional Analysis of Variance (FANOVA) from Hilbert-valued correlated data with spatial rectangular or circular supports is analyzed, when Dirichlet conditions are assumed on the boundary. Specifically, a Hilbert-valued fixed effect model with error term defined from an Autoregressive Hilbertian process of order one (ARH(1) process) is considered, extending the formulation given in Ruiz-Medina (2016). A new statistical test is also derived to contrast the significance of the functional fixed effect parameters. The Dirichlet conditions established at the boundary affect the dependence range o…
