We offer ShiftConvolvePoibin, a fast exact method to compute the tail of a Poisson-Binomial distribution (PBD). Our method employs an exponential shift to retain its accuracy when computing a tail probability, and in practice we find that it is immune to the significant relative errors that other methods, exact or approximate, can suffer from when computing very small tail probabilities of the PBD. The accompanying R package is also competitive with the fastest implementations for computing the entire PBD.
Beta-binomial/Poisson models have been used by many authors to model multivariate count data. Lora and Singer (Statistics in Medicine, 2008) extended such models to accommodate repeated multivariate count data with overdipersion in the binomial component. To overcome some of the limitations of that model, we consider a beta-binomial/gamma-Poisson alternative that also allows for both overdispersion and different covariances between the Poisson counts. We obtain maximum likelihood estimates for the parameters using a Newton-Raphson algorithm and compare both models in a practical example.
In this paper, we have developed a new class of sampling schemes for estimating parameters of binomial and Poisson distributions. Without any information of the unknown parameters, our sampling schemes rigorously guarantee prescribed levels of precision and confidence.
A beta-negative binomial (BNB) process is proposed, leading to a beta-gamma-Poisson process, which may be viewed as a "multi-scoop" generalization of the beta-Bernoulli process. The BNB process is augmented into a beta-gamma-gamma-Poisson hierarchical structure, and applied as a nonparametric Bayesian prior for an infinite Poisson factor analysis model. A finite approximation for the beta process Levy random measure is constructed for convenient implementation. Efficient MCMC computations are performed with data augmentation and marginalization techniques. Encouraging results are shown on docu…
We consider the segmentation problem of Poisson and negative binomial (i.e. overdispersed Poisson) rate distributions. In segmentation, an important issue remains the choice of the number of segments. To this end, we propose a penalized log-likelihood estimator where the penalty function is constructed in a non-asymptotic context following the works of L. Birgé and P. Massart. The resulting estimator is proved to satisfy an oracle inequality. The performances of our criterion is assessed using simulated and real datasets in the RNA-seq data analysis context.
Rgbp is an R package that provides estimates and verifiable confidence intervals for random effects in two-level conjugate hierarchical models for overdispersed Gaussian, Poisson, and Binomial data. Rgbp models aggregate data from k independent groups summarized by observed sufficient statistics for each random effect, such as sample means, possibly with covariates. Rgbp uses approximate Bayesian machinery with unique improper priors for the hyper-parameters, which leads to good repeated sampling coverage properties for random effects. A special feature of Rgbp is an option that generates synt…
Count data, for example the number of observed cases of a disease in a city, often arise in the fields of healthcare analytics and epidemiology. In this paper, we consider performing regression on multivariate data in which our outcome is a count. Specifically, we derive log-likelihood functions for finite mixtures of regression models involving counts that come from a Poisson distribution, as well as a negative binomial distribution when the counts are significantly overdispersed. Within our proposed modeling framework, we carry out optimal component selection using the information criteria s…
We consider a problem of ecological inference, in which individual-level covariates are known, but labeled data is available only at the aggregate level. The intended application is modeling voter preferences in elections. In Rosenman and Viswanathan (2018), we proposed modeling individual voter probabilities via a logistic regression, and posing the problem as a maximum likelihood estimation for the parameter vector beta. The likelihood is a Poisson binomial, the distribution of the sum of independent but not identically distributed Bernoulli variables, though we approximate it with a heteros…
Heavy-tailed random variables have been used in insurance research to model both loss frequencies and loss severities, with substantially more emphasis on the latter. In the present work, we take a step toward addressing this imbalance by exploring the class of heavy-tailed frequency models formed by continuous mixtures of Negative Binomial and Poisson random variables. We begin by defining the concept of a calibrative family of mixing distributions (each member of which is identifiable from its associated Negative Binomial mixture), and show how to construct such families from only a single m…
Generalized Poisson Matrix Factorization for Overdispersed Count Data
Graduate School of Science and Engineering, Chuo University（中央大学理工学研究科）；School of Pharmaceutical Sciences, Wakayama Medical University（和歌山医科大学药学部）；Faculty of Science and Engineering, Chuo University（中央大学理工学部）
A Revisit to Point Estimation Through the Empirical Bayes Method: The Case of Binomial Distribution with Beta Prior and Extension to Poisson Distribution
Ho Chi Minh City University of Technology（胡志明市理工大学）；Vietnam National University Ho Chi Minh City（越南国立大学胡志明市分校）；Ton Duc Thang University（崇正大学）
We propose in this paper a random intercept Poisson model in which the random effect distribution is assumed to follow a generalized log-gamma (GLG) distribution. We derive the first two moments for the marginal distribution as well as the intraclass correlation. Even though numerical integration methods are in general required for deriving the marginal models, we obtain the multivariate negative binomial model for a particular parameter setting of the hierarchical model. An iterative process is derived for obtaining the maximum likelihood estimates for the parameters in the multivariate negat…
State space modelling is an efficient and flexible method for statistical inference of a broad class of time series and other data. This paper describes an R package KFAS for state space modelling with the observations from an exponential family, namely Gaussian, Poisson, binomial, negative binomial and gamma distributions. After introducing the basic theory behind Gaussian and non-Gaussian state space models, an illustrative example of Poisson time series forecasting is provided. Finally, a comparison to alternative R packages suitable for non-Gaussian time series modelling is presented.
The seemingly disjoint problems of count and mixture modeling are united under the negative binomial (NB) process. A gamma process is employed to model the rate measure of a Poisson process, whose normalization provides a random probability measure for mixture modeling and whose marginalization leads to an NB process for count modeling. A draw from the NB process consists of a Poisson distributed finite number of distinct atoms, each of which is associated with a logarithmic distributed number of data samples. We reveal relationships between various count- and mixture-modeling distributions an…
The generalized negative binomial distribution (GNB) is a new flexible family of discrete distributions that are mixed Poisson laws with the mixing generalized gamma (GG) distributions. This family of discrete distributions is very wide and embraces Poisson distributions, negative binomial distributions, Sichel distributions, Weibull--Poisson distributions and many other types of distributions supplying descriptive statistics with many flexible models. These distributions seem to be very promising for the statistical description of many real phenomena. GG distributions are widely applied in si…
Split sample methods have recently been put forward as a way to reduce the coverage oscillations that haunt confidence intervals for parameters of lattice distributions, such as the binomial and Poisson distributions. We study split sample intervals in the binomial setting, showing that these intervals can be viewed as being based on adding discrete random noise to the data. It is shown that they can be improved upon by using noise with a continuous distribution instead, regardless of whether the randomization is determined by the data or an external source of randomness. We compare split samp…
In this article, we discuss a bivariate distribution whose conditionals are univariate binomial distributions and the marginals are not binomial that exhibits negative correlation. Some useful structural properties of this distribution namely marginals, moments, generating functions, stochastic ordering are investigated. Simple proofs of negative correlation, marginal over-dispersion, distribution of sum and conditional given the sum are also derived. The distribution is shown to be a member of the multi-parameter exponential family and some natural but useful consequences are also outlined. T…
In this paper, we propose a closed form approximation to the mean and variance of a new generalization of negative binomial (NGNB) distribution arising from the Extended COM-Poisson (ECOMP) distribution developed by Chakraborty and Imoto (2016)(see [4]). The NGNB is a special case of the ECOMP distribution and was named so by these authors. This distribution is more flexible in terms of the dispersion index as compared to its ordinary counterparts. It approaches the COM-Poisson distribution (Shmueli et al. 2005) [11] under suitable limiting conditions. The NGNB can also be obtained from the CO…
The robust Poisson method is becoming increasingly popular when estimating the association of exposures with a binary outcome. Unlike the logistic regression model, the robust Poisson method yields results that can be interpreted as risk or prevalence ratios. In addition, it does not suffer from frequent non-convergence problems like the most common implementations of maximum likelihood estimators of the log-binomial model. However, using a Poisson distribution to model a binary outcome may seem counterintuitive. Methodological papers have often presented this as a good approximation to the mo…
Modeling spatial overdispersion requires point processes models with finite dimensional distributions that are overdisperse relative to the Poisson. Fitting such models usually heavily relies on the properties of stationarity, ergodicity, and orderliness. And, though processes based on negative binomial finite dimensional distributions have been widely considered, they typically fail to simultaneously satisfy the three required properties for fitting. Indeed, it has been conjectured by Diggle and Milne that no negative binomial model can satisfy all three properties. In light of this, we chang…
We consider the problem of comparing two Poisson parameters from the Bayesian perspective. Kawasaki and Miyaoka (2012b) proposed the Bayesian index $P(λ_1 < λ_2 | X_1,X_2)$ and expressed it using the hypergeometric series. In this paper, under some conditions, we give four other expressions of the Bayesian index in terms of the cumulative distribution functions of beta, $F$, binomial, and negative binomial distribution. Next, we investigate the relationship between the Bayesian index and the $p$-value of the conditional test with the null hypothesis $H_0: λ_1 \geq λ_2 $ versus an alternative h…
In regression analysis of counts, a lack of simple and efficient algorithms for posterior computation has made Bayesian approaches appear unattractive and thus underdeveloped. We propose a lognormal and gamma mixed negative binomial (NB) regression model for counts, and present efficient closed-form Bayesian inference; unlike conventional Poisson models, the proposed approach has two free parameters to include two different kinds of random effects, and allows the incorporation of prior information, such as sparsity in the regression coefficients. By placing a gamma distribution prior on the NB…
In this article, we propose a new three parameter distribution by compounding negative binomial with reciprocal inverse Gaussian model called negative binomial-reciprocal inverse Gaussian distribution. This model is tractable with some important properties not only in actuarial science but in other fields as well where overdispersion pattern is seen. Some basic properties including recurrence relation of probabilities for computation of successive probabilities have been discussed. In its compound version, when the claims are absolutely continuous, an integral equation for the probability dens…
This article introduces a k-Inflated Negative Binomial mixture distribution/regression model as a more flexible alternative to zero-inflated Poisson distribution/regression model. An EM algorithm has been employed to estimate the model's parameters. Then, such new model along with a Pareto mixture model have been employed to design an optimal rate--making system. Namely, this article employs number/size of reported claims of Iranian third party insurance dataset. Then, it employs the k-Inflated Negative Binomial mixture distribution/regression model as well as other well developed counting mod…
In this paper, we consider Bayesian point estimation and predictive density estimation in the binomial case. After presenting preliminary results on these problems, we compare the risk functions of the Bayes estimators based on the truncated and untruncated beta priors and obtain dominance conditions when the probability parameter is less than or equal to a known constant. The case where there are both a lower bound restriction and an upper bound restriction is also treated. Then our problems are shown to be related to similar problems in the Poisson case. Finally, numerical studies are presen…
We define a family of probability distributions for random count matrices with a potentially unbounded number of rows and columns. The three distributions we consider are derived from the gamma-Poisson, gamma-negative binomial, and beta-negative binomial processes. Because the models lead to closed-form Gibbs sampling update equations, they are natural candidates for nonparametric Bayesian priors over count matrices. A key aspect of our analysis is the recognition that, although the random count matrices within the family are defined by a row-wise construction, their columns can be shown to be…
Motivated by the fundamental problem of measuring species diversity, this paper introduces the concept of a cluster structure to define an exchangeable cluster probability function that governs the joint distribution of a random count and its exchangeable random partitions. A cluster structure, naturally arising from a completely random measure mixed Poisson process, allows the probability distribution of the random partitions of a subset of a sample to be dependent on the sample size, a distinct and motivated feature that differs it from a partition structure. A generalized negative binomial …
The paper introduces the concept of a cluster structure to define a joint distribution of the sample size and its exchangeable random partitions. The cluster structure allows the probability distribution of the random partitions of a subset of the sample to be dependent on the sample size, a feature not presented in a partition structure. A generalized negative binomial process count-mixture model is proposed to generate a cluster structure, where in the prior the number of clusters is finite and Poisson distributed and the cluster sizes follow a truncated negative binomial distribution. The n…
