Species-sampling problems (SSPs) refer to a vast class of statistical problems calling for the estimation of (discrete) functionals of the unknown species composition of an unobservable population. A common feature of SSPs is their invariance with respect to species labeling, which is at the core of the Bayesian nonparametric (BNP) approach to SSPs under the popular Pitman-Yor process (PYP) prior. In this paper, we develop a BNP approach to SSPs that are not "invariant" to species labeling, in the sense that an ordering or ranking is assigned to species' labels. Inspired by the population genetics literature on age-ordered alleles' compositions, we study the following SSP with ordering: given an observable sample from an unknown population of individuals belonging to species (alleles), with species' labels being ordered according to weights (ages), estimate the frequencies of the first $r$ order species' labels in an enlarged sample obtained by including additional unobservable samples. By relying on an ordered PYP prior, we obtain an explicit posterior distribution of the first $r$ order frequencies, with estimates being of easy implementation and computationally efficient. We apply our approach to the analysis of genetic variation, showing its effectiveness in estimating the frequency of the oldest allele, and then we discuss other potential applications.
We derive explicit Bayesian nonparametric analysis for a species sampling model with finitely many types of Gibbs form of type $α= -1$ recently introduced in Gnedin (2009). Our results complement existing analysis under Gibbs priors of type $α\in [0, 1)$ proposed in Lijoi et al. (2008). Calculations rely on a groups sequential construction of Gibbs partitions introduced in Cerquetti (2008).
We present an alternative approach to the Bayesian nonparametric analysis of conditional species richness under two-parameter Poisson Dirichlet priors. We rely on a known characterization by deletion of classes property and on results for Beta-Binomial distributions. Besides leading to simplified and much more direct proofs, our proposal provides a new scale mixture representation of the conditional asymptotic law.
We aim at modelling the appearance of distinct tags in a sequence of labelled objects. Common examples of this type of data include words in a corpus or distinct species in a sample. These sequential discoveries are often summarised via accumulation curves, which count the number of distinct entities observed in an increasingly large set of objects. We propose a novel Bayesian nonparametric method for species sampling modelling by directly specifying the probability of a new discovery, therefore allowing for flexible specifications. The asymptotic behavior and finite sample properties of such …
A Bayesian nonparametric approach to the study of species diversity based on choosing a random discrete distribution as a prior model for the unknown relative abundances of species has been recently introduced in Lijoi et al. (2007, 2008). Explicit posterior predictive estimation of {\it species richness} has been obtained under priors belonging to the $α$-Gibbs class (Gnedin & Pitman, 2006). Here we focus on posterior estimation of {\it species evenness} which accounts for diversity in terms of the proximity to the situation of uniform distribution of the population into different species. We…
We introduce a dependent Bayesian nonparametric model for the probabilistic modeling of membership of subgroups in a community based on partially replicated data. The focus here is on species-by-site data, i.e. community data where observations at different sites are classified in distinct species. Our aim is to study the impact of additional covariates, for instance environmental variables, on the data structure, and in particular on the community diversity. To that purpose, we introduce dependence a priori across the covariates, and show that it improves posterior inference. We use a depende…
We present a novel Bayesian nonparametric regression model for covariates X and continuous, real response variable Y. The model is parametrized in terms of marginal distributions for Y and X and a regression function which tunes the stochastic ordering of the conditional distributions F(y|x). By adopting an approximate composite likelihood approach, we show that the resulting posterior inference can be decoupled for the separate components of the model. This procedure can scale to very large datasets and allows for the use of standard, existing, software from Bayesian nonparametric density est…
We consider nonparametric inference for event time distributions based on current status data. We show that in this scenario conventional mixture priors, including the popular Dirichlet process mixture prior, lead to biologically uninterpretable results as they unnaturally skew the probability mass for the event times toward the extremes of the observed data. Simple assumptions on dependent censoring can fix the problem. We then extend the discussion to bivariate current status data with partial ordering of the two outcomes. In addition to dependent censoring, we also exploit some minimal know…
In Bayesian nonparametric inference, random discrete probability measures are commonly used as priors within hierarchical mixture models for density estimation and for inference on the clustering of the data. Recently, it has been shown that they can also be exploited in species sampling problems: indeed they are natural tools for modeling the random proportions of species within a population thus allowing for inference on various quantities of statistical interest. For applications that involve large samples, the exact evaluation of the corresponding estimators becomes impracticable and, ther…
This paper introduces a general class of hierarchical nonparametric prior distributions. The random probability measures are constructed by a hierarchy of generalized species sampling processes with possibly non-diffuse base measures. The proposed framework provides a general probabilistic foundation for hierarchical random measures with either atomic or mixed base measures and allows for studying their properties, such as the distribution of the marginal and total number of clusters. We show that hierarchical species sampling models have a Chinese Restaurants Franchise representation and can …
Prior distributions play a crucial role in Bayesian approaches to clustering. Two commonly-used prior distributions are the Dirichlet and Pitman-Yor processes. In this paper, we investigate the predictive probabilities that underlie these processes, and the implicit "rich-get-richer" characteristic of the resulting partitions. We explore an alternative prior for nonparametric Bayesian clustering -- the uniform process -- for applications where the "rich-get-richer" property is undesirable. We also explore the cost of this process: partitions are no longer exchangeable with respect to the order…
The number of species can be estimated by sampling individuals from a species assemblage. The problem of estimating generalized species accumulation curve is addressed in a nonparametric Poisson mixture model. A likelihood-based estimator is proposed and illustrated by real examples.
Given a sample of size $n$ from a population of individuals belonging to different species with unknown proportions, a popular problem of practical interest consists in making inference on the probability $D_{n}(l)$ that the $(n+1)$-th draw coincides with a species with frequency $l$ in the sample, for any $l=0,1,\ldots,n$. This paper contributes to the methodology of Bayesian nonparametric inference for $D_{n}(l)$. Specifically, under the general framework of Gibbs-type priors we show how to derive credible intervals for a Bayesian nonparametric estimation of $D_{n}(l)$, and we investigate th…
A Bayesian nonparametric method for unimodal densities on the real line is provided by considering a class of species sampling mixture models containing random densities that are unimodal and not necessarily symmetric. This class of densities generalize the model considered by Brunner (1992, Statist. Probab. Lett.), in which the Dirichlet process is replaced by a more general class of species sampling models. A novel and explicit characterization of the posterior distribution via a finite mixture of two dependent $\mathbf{S}$-paths is derived. This results in a closed-form and tractable Bayes …
Gibbs partition models are the largest class of infinite exchangeable partitions of the positive integers generalizing the product form of the probability function of the two-parameter Poisson-Dirichlet family. Recently those models have been investigated in a Bayesian nonparametric approach to species sampling problems as alternatives to the Dirichlet and the Pitman-Yor process priors. Here we derive marginals of conditional and unconditional multivariate distributions arising from exchangeable Gibbs partitions to obtain explicit formulas for joint falling factorial moments of corresponding c…
This paper proposes the estimation of a smooth graphon model for network data analysis using principles of the EM algorithm. The approach considers both variability with respect to ordering the nodes of a network and smooth estimation of the graphon by nonparametric regression. To do so, (linear) B-splines are used, which allow for smooth estimation of the graphon, conditional on the node ordering. This provides the M-step. The true ordering of the nodes arising from the graphon model remains unobserved and Bayesian ideas are employed to obtain posterior samples given the network data. This yi…
We consider the estimation of the total number $N$ of species based on the abundances of species that have been observed. We adopt a non parametric approach where the true abundance distribution $p$ is only supposed to be convex. From this assumption, we propose a definition for convex abundance distributions. We use a least-squares estimate of the truncated version of $p$ under the convexity constraint. We deduce two estimators of the total number of species, the asymptotic distribution of which are derived. We propose three different procedures, including a bootstrap one, to obtain a confide…
Robust statistical data modelling under potential model mis-specification often requires leaving the parametric world for the nonparametric. In the latter, parameters are infinite dimensional objects such as functions, probability distributions or infinite vectors. In the Bayesian nonparametric approach, prior distributions are designed for these parameters, which provide a handle to manage the complexity of nonparametric models in practice. However, most modern Bayesian nonparametric models seem often out of reach to practitioners, as inference algorithms need careful design to deal with the …
Favaro, Lijoi, and Pruenster (2012, Biometrics, 68, 1188--1196) derive a novel Bayesian nonparametric estimator of the probability of detecting at the $(n+m+1)$th observation a species already observed with any given frequency in an enlarged sample of size $n+m$, conditionally on a basic sample of size $n$. Unfortunately the general result under Gibbs priors (Theorem 2), and consequently the explicit result under $(α, θ)$ Poisson-Dirichlet priors (Proposition 3), appear to be wrong. Here we provide the correct formulas for both the results, obtained by means of a new technique devised in Cerqu…
Human microbiome studies use sequencing technologies to measure the abundance of bacterial species or Operational Taxonomic Units (OTUs) in samples of biological material. Typically the data are organized in contingency tables with OTU counts across heterogeneous biological samples. In the microbial ecology community, ordination methods are frequently used to investigate latent factors or clusters that capture and describe variations of OTU counts across biological samples. It remains important to evaluate how uncertainty in estimates of each biological sample's microbial distribution propagat…
Directed graphs occur throughout statistical modeling of networks, and exchangeability is a natural assumption when the ordering of vertices does not matter. There is a deep structural theory for exchangeable undirected graphs, which extends to the directed case via measurable objects known as digraphons. Using digraphons, we first show how to construct models for exchangeable directed graphs, including special cases such as tournaments, linear orderings, directed acyclic graphs, and partial orderings. We then show how to construct priors on digraphons via the infinite relational digraphon mod…
We introduce a three-parameter random walk with reinforcement, called the $(θ,α,β)$ scheme, which generalizes the linearly edge reinforced random walk to uncountable spaces. The parameter $β$ smoothly tunes the $(θ,α,β)$ scheme between this edge reinforced random walk and the classical exchangeable two-parameter Hoppe urn scheme, while the parameters $α$ and $θ$ modulate how many states are typically visited. Resorting to de Finetti's theorem for Markov chains, we use the $(θ,α,β)$ scheme to define a nonparametric prior for Bayesian analysis of reversible Markov chains. The prior is applied in…
Motivated by the fundamental problem of measuring species diversity, this paper introduces the concept of a cluster structure to define an exchangeable cluster probability function that governs the joint distribution of a random count and its exchangeable random partitions. A cluster structure, naturally arising from a completely random measure mixed Poisson process, allows the probability distribution of the random partitions of a subset of a sample to be dependent on the sample size, a distinct and motivated feature that differs it from a partition structure. A generalized negative binomial …
The problem of maximizing cell type discovery under budget constraints is a fundamental challenge for the collection and analysis of single-cell RNA-sequencing (scRNA-seq) data. In this paper, we introduce a simple, computationally efficient, and scalable Bayesian nonparametric sequential approach to optimize the budget allocation when designing a large scale experiment for the collection of scRNA-seq data for the purpose of, but not limited to, creating cell atlases. Our approach relies on the following tools: i) a hierarchical Pitman-Yor prior that recapitulates biological assumptions regard…
This paper presents a methodology for creating streaming, distributed inference algorithms for Bayesian nonparametric (BNP) models. In the proposed framework, processing nodes receive a sequence of data minibatches, compute a variational posterior for each, and make asynchronous streaming updates to a central model. In contrast to previous algorithms, the proposed framework is truly streaming, distributed, asynchronous, learning-rate-free, and truncation-free. The key challenge in developing the framework, arising from the fact that BNP models do not impose an inherent ordering on their compon…
In Bayesian nonparametrics there exists a rich variety of discrete priors, including the Dirichlet process and its generalizations, which are nowadays well-established tools. Despite the remarkable advances, few proposals are tailored for modeling observations lying on product spaces, such as $\mathbb{R}^p$. Indeed, for multivariate random measures, most available priors lack flexibility and do not allow for separate partition structures among the spaces. We introduce a discrete nonparametric prior, termed enriched Pitman-Yor process (EPY), aimed at addressing these issues. Theoretical propert…
Harvard University（哈佛大学）；Universita di Pavia（帕维亚大学）；University of Kent（肯特大学）；MD Anderson Cancer Center（MD安德森癌症中心）
Many popular Bayesian nonparametric priors can be characterized in terms of exchangeable species sampling sequences. However, in some applications, exchangeability may not be appropriate. We introduce a {novel and probabilistically coherent family of non-exchangeable species sampling sequences characterized by a tractable predictive probability function with weights driven by a sequence of independent Beta random variables. We compare their theoretical clustering properties with those of the Dirichlet Process and the two parameters Poisson-Dirichlet process. The proposed construction provides …
We present a novel nonparametric Bayesian approach based on Lévy Adaptive Regression Kernels (LARK) to model spectral data arising from MALDI-TOF (Matrix Assisted Laser Desorption Ionization Time-of-Flight) mass spectrometry. This model-based approach provides identification and quantification of proteins through model parameters that are directly interpretable as the number of proteins, mass and abundance of proteins and peak resolution, while having the ability to adapt to unknown smoothness as in wavelet based methods. Informative prior distributions on resolution are key to distinguishing …
Gibbs-type random probability measures and the exchangeable random partitions they induce represent the subject of a rich and active literature. They provide a probabilistic framework for a wide range of theoretical and applied problems that are typically referred to as species sampling problems. In this paper, we consider the class of looking-backward species sampling problems introduced in Lijoi et al. (Ann. Appl. Probab. 18 (2008) 1519-1547) in Bayesian nonparametrics. Specifically, given some information on the random partition induced by an initial sample from a Gibbs-type random probabil…
While the analysis of airborne laser scanning (ALS) data often provides reliable estimates for certain forest stand attributes -- such as total volume or basal area -- there is still room for improvement, especially in estimating species-specific attributes. Moreover, while information on the estimate uncertainty would be useful in various economic and environmental analyses on forests, a computationally feasible framework for uncertainty quantifying in ALS is still missing. In this article, the species-specific stand attribute estimation and uncertainty quantification (UQ) is approached using…
We develop a method to infer community structure in directed networks where the groups are ordered in a latent one-dimensional hierarchy that determines the preferred edge direction. Our nonparametric Bayesian approach is based on a modification of the stochastic block model (SBM), which can take advantage of rank alignment and coherence to produce parsimonious descriptions of networks that combine ordered hierarchies with arbitrary mixing patterns between groups. Since our model also includes directed degree correction, we can use it to distinguish non-local hierarchical structure from local …
This work proposes Quor, a simple yet effective nonparametric method to compare independent samples with respect to corresponding quantiles of their populations. The method is solely based on the order statistics of the samples, and independence is its only requirement. All computations are performed using exact distributions with no need for any asymptotic considerations, and yet can be run using a fast quadratic-time dynamic programming idea. Computational performance is essential in high-dimensional domains, such as gene expression data. We describe the approach and discuss on the most impo…
The statistical problem of using an initial sample to estimate the number of species in a larger sample has found important applications in fields far removed from ecology. Here we address the general problem of estimating the number of species that will be represented by at least a number r of observations in a future sample. The number r indicates species with sufficient observations, which are commonly used as a necessary condition for any robust statistical inference. We derive a procedure to construct consistent estimators that apply universally for a given population: once constructed, t…
Tsallis entropy is a generalized diversity index first derived in Patil and Taillie (1982) and then rediscovered in community ecology by Keylock (2005). Bayesian nonparametric estimation of Shannon entropy and Simpson's diversity under uniform and symmetric Dirichlet priors has been already advocated as an alternative to maximum likelihood estimation based on frequency counts, which is negatively biased in the undersampled regime. Here we present a fully general Bayesian nonparametric estimation of the whole class of Tsallis diversity indices under Gnedin-Pitman priors, a large family of rando…
We propose a class of methods for graphon estimation based on exploiting connections with nonparametric regression. The idea is to construct an ordering of the nodes in the network, similar in spirit to Chan and Airoldi (2014). However, rather than only considering orderings based on the empirical degree as in Chan and Airoldi (2014), we use the nearest neighbor algorithm which is an approximating solution to the traveling salesman problem. This in turn can handle general distances $\hat{d}$ between the nodes, something that allows us to incorporate rich information of the network. Once an ord…
In spatial statistics, it is often assumed that the spatial field of interest is stationary and its covariance has a simple parametric form, but these assumptions are not appropriate in many applications. Given replicate observations of a Gaussian spatial field, we propose nonstationary and nonparametric Bayesian inference on the spatial dependence. Instead of estimating the quadratic (in the number of spatial locations) entries of the covariance matrix, the idea is to infer a near-linear number of nonzero entries in a sparse Cholesky factor of the precision matrix. Our prior assumptions are m…
Given a dataset of careers and incomes, how large a difference of income between any pair of careers would be? Given a dataset of travel time records, how long do we need to spend more when choosing a public transportation mode $A$ instead of $B$ to travel? In this paper, we propose a framework that is able to infer orders of categories as well as magnitudes of difference of real numbers between each pair of categories using Estimation statistics framework. Not only reporting whether an order of categories exists, but our framework also reports the magnitude of difference of each consecutive p…
Feature allocation models generalize species sampling models by allowing every observation to belong to more than one species, now called features. Under the popular Bernoulli product model for feature allocation, given $n$ samples, we study the problem of estimating the missing mass $M_{n}$, namely the expected number hitherto unseen features that would be observed if one additional individual was sampled. This is motivated by numerous applied problems where the sampling procedure is expensive, in terms of time and/or financial resources allocated, and further samples can be only motivated by…
Gibbs-type priors are widely used as key components in several Bayesian nonparametric models. By virtue of their flexibility and mathematical tractability, they turn out to be predominant priors in species sampling problems, clustering and mixture modelling. We introduce a new family of processes which extend the Gibbs-type one, by including a contaminant component in the model to account for the presence of anomalies (outliers) or an excess of observations with frequency one. We first investigate the induced random partition, the associated predictive distribution and we characterize the asym…
