Consider the random Dirichlet partition of the interval into $n$ fragments with parameter $θ>0$. We recall the unordered Ewens sampling formulae from finite Dirichlet partitions. As this is a key variable for estimation purposes, focus is on the number of distinct visited species in the sampling process. These are illustrated in specific cases. We use these preliminary statistical results on frequencies distribution to address the following sampling problem: what is the estimated number of species when sampling is from Dirichlet populations? The obtained results are in accordance with the ones found in sampling theory from random proportions with Poisson-Dirichlet distribution. To conclude with, we apply the different estimators suggested to two different sets of real data.
Species distribution models usually attempt to explain presence-absence or abundance of a species at a site in terms of the environmental features (socalled abiotic features) present at the site. Historically, such models have considered species individually. However, it is well-established that species interact to influence presence-absence and abundance (envisioned as biotic factors). As a result, there has been substantial recent interest in joint species distribution models with various types of response, e.g., presence-absence, continuous and ordinal data. Such models incorporate dependen…
Harvard University（哈佛大学）；Universita di Pavia（帕维亚大学）；University of Kent（肯特大学）；MD Anderson Cancer Center（MD安德森癌症中心）
Many popular Bayesian nonparametric priors can be characterized in terms of exchangeable species sampling sequences. However, in some applications, exchangeability may not be appropriate. We introduce a {novel and probabilistically coherent family of non-exchangeable species sampling sequences characterized by a tractable predictive probability function with weights driven by a sequence of independent Beta random variables. We compare their theoretical clustering properties with those of the Dirichlet Process and the two parameters Poisson-Dirichlet process. The proposed construction provides …
We propose a truncation model for abundance distribution in the species richness estimation. This model is inherently semiparametric and incorporates an unknown truncation threshold between rare and abundant counts observations. Using the conditional likelihood, we derive a class of estimators for the parameters in the model by a stepwise maximisation. The species richness estimator is given by the integer maximising the binomial likelihood when all other parameters in the model are know. Under regularity conditions, we show that the estimators of the model parameters are asymptotically effici…
The statistical problem of using an initial sample to estimate the number of species in a larger sample has found important applications in fields far removed from ecology. Here we address the general problem of estimating the number of species that will be represented by at least a number r of observations in a future sample. The number r indicates species with sufficient observations, which are commonly used as a necessary condition for any robust statistical inference. We derive a procedure to construct consistent estimators that apply universally for a given population: once constructed, t…
In the 1920's, the English philosopher W.E. Johnson introduced a characterization of the symmetric Dirichlet prior distribution in terms of its predictive distribution. This is typically referred to as Johnson's "sufficientness" postulate, and it has been the subject of many contributions in Bayesian statistics, leading to predictive characterization for infinite-dimensional generalizations of the Dirichlet distribution, i.e. species-sampling models. In this paper, we review "sufficientness" postulates for species-sampling models, and then investigate analogous predictive characterizations for…
We derive the Jeffreys prior for the parameter of the Multivariate Ewens Distribution and study some of its properties. In particular, we show that this prior is proper and has no finite moments. We also investigate the impact of this default prior on the a priori distribution of the number of species and the a priori probability of discovery of a new species, which are usually employed in subjective prior elicitation. The effect of the Jeffreys prior for posterior inference is illustrated using examples arising in the context of inference for species sampling models and Dirichlet process mixt…
Finite Mixtures of Generalized Estimating Equations for Clustering Multivariate Correlated Outcomes
Faculty of Economics, Keio University（庆应义塾大学经济学部）；Research School of Finance, Actuarial Studies and Statistics, The Australian National University（澳大利亚国立大学金融、精算与统计研究学院）
Heterogeneity of response patterns is important in estimating the size of a closed population from multiple recapture data when capture patterns are different over time and location. In this paper, we extend the non-parametric one layer latent class model for multiple recapture data proposed by Manrique-Vallier (2016) to a nested latent class model with the first layer modeling individual heterogeneity and the second layer modeling location-time differences. Location-time groups with similar recording patterns are in the same top layer latent class and individuals within each top layer class a…
Ecologists are interested in modeling the population growth of species in various ecosystems. Studying population dynamics can assist environmental managers in making better decisions for the environment. Traditionally, the sampling of species and tracking of populations have been recorded on a regular time frequency. However, sampling can be an expensive process due to available resources, money and time. Limiting sampling makes it challenging to properly track the growth of a population. Thus, we propose a new and novel approach to designing sampling regimes based on the dynamics associated …
Discrete random probability measures and the exchangeable random partitions they induce are key tools for addressing a variety of estimation and prediction problems in Bayesian inference. Indeed, many popular nonparametric priors, such as the Dirichlet and the Pitman-Yor process priors, select discrete probability distributions almost surely and, therefore, automatically induce exchangeable random partitions. Here we focus on the family of Gibbs-type priors, a recent and elegant generalization of the Dirichlet and the Pitman-Yor process priors. These random probability measures share propertie…
Models of the spatial distribution of animals provide useful tools to help ecologists quantify species-environment relationships, and they are increasingly being used to help determine the impacts of climate and habitat changes on species. While high-quality survey-style data with known effort are sometimes available, often researchers have multiple datasets of varying quality and type. In particular, collections of sightings made by citizen scientists are becoming increasingly common, with no information typically provided on their observer effort. Many standard modelling approaches ignore ob…
In this article, we consider a non-parametric Bayesian approach to multivariate quantile regression. The collection of related conditional distributions of a response vector Y given a univariate covariate X is modeled using a Dependent Dirichlet Process (DDP) prior. The DDP is used to introduce dependence across x. As the realizations from a Dirichlet process prior are almost surely discrete, we need to convolve it with a kernel. To model the error distribution as flexibly as possible, we use a countable mixture of multidimensional normal distributions as our kernel. For posterior computations…
A factor effect study was conducted on a set of observations at the contingency of a series of plant species and bacteria species regarding the antibacterial activity of essential oil extracts. The study reveals a very good agreement between the observations and the hypothesis of independent and multiplicative effect of plant and bacteria species factors on the antibacterial activity. Shaping of the observable to a Negative Binomial distribution allowed the separation of two convoluted Gamma distributions in the observable further assigned to the distribution of factors. Statistics of the Gamm…
This paper develops tools to characterize how species are affected by environmental variability, based on a functional single index model relating a response such as growth rate or survival to environmental conditions. In ecology, the curvature of such responses are used, via Jensen's inequality, to determine whether environmental variability is harmful or beneficial, and differing nonlinear responses to environmental variability can contribute to the coexistence of competing species. Here, we address estimation and inference for these models with observational data on individual responses to …
Challenges of assessing complexity and clonality in populations of mixed species arise in diverse areas of modern biology, including estimating diversity and clonality in microbiome populations, measuring patterns of T and B cell clonality, and determining the underlying tumor cell population structure in cancer. Here we address the problem of quantifying populations, with our analysis directed toward systems for which previously defined algorithms allow the sequence-based identification of clonal subpopulations. Data come from replicate sequencing libraries generated from a sample, potentiall…
Bocconi Institute for Data Science and Analytics, Bocconi University（博科尼大学数据科学与分析研究所）；Department of Economics, Università degli studi di Bergamo（贝加莫大学经济系）；Department of Economics, Management and Statistics, University of Milano-Bicocca（米兰比可卡大学经济、管理与统计系）；Department of Statistical Sciences, Università Cattolica del Sacro Cuore（天主教大学统计科学系）
We estimate the number of fatal victims of the Peruvian internal armed conflict between 1980-2000 using stratified seven-list Capture-Recapture methods based on Dirichlet process mixtures, which we extend to accommodate incomplete stratification information. We use matched data from six sources, originally analyzed by the Peruvian Truth and Reconciliation Commission in 2003, together with a new large dataset, originally published in 2006 by the Peruvian government. We deal with missing stratification labels by developing a general framework and estimation methods based on MCMC sampling for joi…
Estimation of the number of species or unobserved classes from a random sample of the underlying population is a ubiquitous problem in statistics. In classical settings, the size of the sample is usually small. New technologies such as high-throughput DNA sequencing have allowed for the sampling of extremely large and heterogeneous populations at scales not previously attainable or even considered. New algorithms are required that take advantage of the size of the data to account for heterogeneity, but are also sufficiently fast and scale well with large data. We present a non-parametric momen…
Estimating the number $n$ of unseen species from a $k-$sample displaying only $p\leq k$ distinct sampled species has received attention for long. It requires a model of species abundance together with a sampling model. We start with a discrete model of iid stochastic species abundances, each with Gibbs-Poisson distribution. A $k-$sample drawn from the $n-$species abundances vector is the one obtained while conditioning it on summing to $k$% . We discuss the sampling formulae (species occupancy distributions, frequency of frequencies) in this context. We then develop some aspects of the estimat…
