Using the technique developed in approximation theory, we construct examples of exponential families of infinitely divisible laws which can be viewed as deformations of the normal, gamma, and Poisson exponential families. Replacing the differential equation of approximation theory by a q-differential equation, we define the q-exponential families, and we identify all q-exponential families with quadratic variance functions when |q|<1. We elaborate on the case of q=0 which is related to free convolution of measures. We conclude by considering briefly the case q>1, and other related generalizations.
This article studies exponential families $\mathcal{E}$ on finite sets such that the information divergence $D(P\|\mathcal{E})$ of an arbitrary probability distribution from $\mathcal{E}$ is bounded by some constant $D>0$. A particular class of low-dimensional exponential families that have low values of $D$ can be obtained from partitions of the state space. The main results concern optimality properties of these partition exponential families. Exponential families where $D=\log(2)$ are studied in detail. This case is special, because if $D<\log(2)$, then $\mathcal{E}$ contains all probabilit…
Exponential families comprise a broad class of statistical models and parametric families like normal distributions, binomial distributions, gamma distributions or exponential distributions. Thereby the formal representation of its probability distributions induces a confined intrinsic structure, which appears to be that of a dually flat statistical manifold. Conversely it can be shown, that any dually flat statistical manifold, which is given by a regular Bregman divergence uniquely induced a regular exponential family, such that exponential families may - with some restrictions - be regarded…
Equivalence on exponential families as Hessian manifolds and classification of exponential families of order 1 with constant Hessian sectional curvature
Tokai University（东海大学）；The University of Tokyo（东京大学）
We characterize bivariate natural exponential families having the diagonal of the variance function of the form \[ \textrm{diag} V(m_1,m_2)=\left(Am_1^2+am_1+bm_2+e,Am_2^2+cm_1+dm_2+f\right), \] with $A<0$ and $a,\ldots,f\in\mathbb{R}$. The solution of the problem relies on finding the conditions under which a specific parametric family of functions consists of Laplace transforms of some probability measures.
Exponential families are the workhorses of parametric modelling theory. One reason for their popularity is their associated inference theory, which is very clean, both from a theoretical and a computational point of view. One way in which this set of tools can be enriched in a natural and interpretable way is through mixing. This paper develops and applies the idea of local mixture modelling to exponential families. It shows that the highly interpretable and flexible models which result have enough structure to retain the attractive inferential properties of exponential families. In particular…
Free exponential families have been previously introduced as a special case of the q-exponential family. We show that free exponential families arise also from a procedure analogous to the definition of exponential families by using the Cauchy-Stieltjes kernel instead of the exponential kernel. We use this approach to re-derive several known results and to study further similarities with exponential families and reproductive exponential models.
The closure of a discrete exponential family is described by a finite set of equations corresponding to the circuits of an underlying oriented matroid. These equations are similar to the equations used in algebraic statistics, although they need not be polynomial in the general case. This description allows for a combinatorial study of the possible support sets in the closure of an exponential family. If two exponential families induce the same oriented matroid, then their closures have the same support sets. Furthermore, the positive cocircuits give a parameterization of the closure of the ex…
The Bayesian predictive density has complex representation and does not belong to any finite-dimensional statistical model except for in limited situations. In this paper, we introduce its simple approximate representation employing its projection onto a finite-dimensional exponential family. Its theoretical properties are established parallelly to those of the Bayesian predictive density when the model belongs to curved exponential families. It is also demonstrated that the projection asymptotically coincides with the plugin density with the posterior mean of the expectation parameter of the …
Suppose that the distribution of $X_a$ belongs to a natural exponential family concentrated on the nonegative integers and is such that $\E(z^{X_a})=f(az)/f(a)$. Assume that $\Pr(X_a\leq k)$ has the form $c_k\int_a ^{\infty}u^kμ(du)$ for some number $c_k$ and some positive measure $μ,$ both independent of $a.$ We show that this asumption implies that the exponential family is either a binomial, or the Poisson, or a negative binomial family. Next, we study an analogous property for continuous distributions and we find that it is satisfied if and only the families are either Gaussian or Gamma. U…
In many families of distributions, maximum likelihood estimation is intractable because the normalization constant for the density which enters into the likelihood function is not easily available. The score matching estimator of Hyvärinen (2005) provides an alternative where this normalization constant is not required. The corresponding estimating equations become linear for an exponential family. The score matching estimator is shown to be consistent and asymptotically normally distributed for such models, although not necessarily efficient. Gaussian linear concentration models are examples …
The lower bound of Cramer and Rao is generalized to pairs of families of probability distributions, one of which is escort to the other. This bound is optimal for certain families, called phi-exponential in the paper. Their dual structure is explored.
Most results in nonparametric regression theory are developed only for the case of additive noise. In such a setting many smoothing techniques including wavelet thresholding methods have been developed and shown to be highly adaptive. In this paper we consider nonparametric regression in exponential families with the main focus on the natural exponential families with a quadratic variance function, which include, for example, Poisson regression, binomial regression and gamma regression. We propose a unified approach of using a mean-matching variance stabilizing transformation to turn the relat…
We employ a parameter-free distribution estimation framework where estimators are random distributions and utilize the Kullback-Leibler (KL) divergence as a loss function. Wu and Vos [J. Statist. Plann. Inference 142 (2012) 1525-1536] show that when an estimator obtained from an i.i.d. sample is viewed as a random distribution, the KL risk of the estimator decomposes in a fashion parallel to the mean squared error decomposition when the estimator is a real-valued random variable. In this paper, we explore how conditional versions of distribution expectation ($E^{\dagger}$) can be defined so th…
RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；The University of Tokyo（东京大学）
In this paper, we give a method to construct "good" exponential families systematically by representation theory. More precisely, we consider a homogeneous space $G/H$ as a sample space and construct an exponential family invariant under the transformation group $G$ by using a representation of $G$. The method generates widely used exponential families such as normal, gamma, Bernoulli, categorical, Wishart, von Mises, Fisher-Bingham and hyperboloid distributions.
The logistic regression model is known to converge to a Poisson point process model if the binary response tends to infinitely imbalanced. In this paper, it is shown that this phenomenon is universal in a wide class of link functions on binomial regression. The proof relies on the extreme value theory. For the logit, probit and complementary log-log link functions, the intensity measure of the point process becomes an exponential family. For some other link functions, deformed exponential families appear. A penalized maximum likelihood estimator for the Poisson point process model is suggested…
We study exponential families of distributions that are multivariate totally positive of order 2 (MTP2), show that these are convex exponential families, and derive conditions for existence of the MLE. Quadratic exponential familes of MTP2 distributions contain attractive Gaussian graphical models and ferromagnetic Ising models as special examples. We show that these are defined by intersecting the space of canonical parameters with a polyhedral cone whose faces correspond to conditional independence relations. Hence MTP2 serves as an implicit regularizer for quadratic exponential families and…
We describe a procedure to introduce general dependence structures on a set of random variables. These include order-$q$ moving average-type structures, as well as seasonal, periodic, spatial and spatio-temporal dependences. The invariant marginal distribution can be in any family that is conjugate to an exponential family with quadratic variance function. Dependence is induced via a set of suitable latent variables whose conditional distribution mirrors the sampling distribution in a Bayesian conjugate analysis of such exponential families. We obtain strict stationarity as a special case.
Let G = An be the graph corresponding to the graphical model of nearest neighbour interaction in a Gaussian character. We study Natural Exponential Families( NEF) ofWishart distributions on convex cones QG and PG, where PG is the cone of positive definite real symmetric matrices with obligatory zeros prescribed by G, and QG is the dual cone of PG. The Wishart NEF that we construct include Wishart distributions considered earlier by Lauritzen (1996) and Letac and Massam (2007) for models based on decomposable graphs. Our approach is however different and allows us to study the basic objects ofW…
We prove the large deviation principle (LDP) for posterior distributions arising from subfamilies of full exponential families, allowing misspecification of the model. Moreover, motivated by the so-called inverse Sanov Theorem (see e.g. Ganesh and O'Connell 1999 and 2000), we prove the LDP for the corresponding maximum likelihood estimator, and we study the relationship between rate functions. In our setting, even in the non misspecified case, it is not true in general that the rate functions for posterior distributions and for maximum likelihood estimators are Kullback-Leibler divergences wit…
We characterize the existence of the maximum likelihood estimator for discrete exponential families. Our criterion is simple to apply as we show in various settings, most notably for exponential models of random graphs. As an application, we point out the size of independent identically distributed samples for which the maximum likelihood estimator exists with high probability.
We study shrinkage estimation of the mean parameters of a class of multivariate distributions for which the diagonal entries of the corresponding covariance matrix are certain quadratic functions of the mean parameter. This class of distributions includes the diagonal multivariate natural exponential families. We propose two classes of semi-parametric shrinkage estimators for the mean and construct unbiased estimators of the corresponding risk. We establish the asymptotic consistency and convergence rates for these shrinkage estimators under squared error loss as both $n$, the sample size, and…
Hazard rates play an important role in various areas, e.g., reliability theory, survival analysis, biostatistics, queueing theory and actuarial studies. Mixtures of distributions are also of a great preeminence in such areas as most populations of components are indeed heterogeneous. In this study we present a sufficient condition for mixtures of two elements\ of the same natural exponential family (NEF) to have an increasing hazard rate. We then apply this condition to some classical NEF's having either quadratic, or cubic variance functions (VF) and others as well. A particular attention is …
We demonstrate how to calculate posteriors for general CRM-based priors and likelihoods for Bayesian nonparametric models. We further show how to represent Bayesian nonparametric priors as a sequence of finite draws using a size-biasing approach---and how to represent full Bayesian nonparametric models via finite marginals. Motivated by conjugate priors based on exponential family representations of likelihoods, we introduce a notion of exponential families for CRMs, which we call exponential CRMs. This construction allows us to specify automatic Bayesian nonparametric conjugate priors for exp…
We study the problem of finding the smallest $m$ such that every element of an exponential family can be written as a mixture of $m$ elements of another exponential family. We propose an approach based on coverings and packings of the face lattice of the corresponding convex support polytopes and results from coding theory. We show that $m=q^{N-1}$ is the smallest number for which any distribution of $N$ $q$-ary variables can be written as mixture of $m$ independent $q$-ary variables. Furthermore, we show that any distribution of $N$ binary variables is a mixture of $m = 2^{N-(k+1)}(1+ 1/(2^k-…
The mathematical theory of reproducing kernel Hilbert spaces (RKHS) provides powerful tools for minimum variance estimation (MVE) problems. Here, we extend the classical RKHS based analysis of MVE in several directions. We develop a geometric formulation of five known lower bounds on the estimator variance (Barankin bound, Cramer-Rao bound, constrained Cramer-Rao bound, Bhattacharyya bound, and Hammersley-Chapman-Robbins bound) in terms of orthogonal projections onto a subspace of the RKHS associated with a given MVE problem. We show that, under mild conditions, the Barankin bound (the tightes…
We study the continuity of space translations on non-parametric exponential families based on the exponential Orlicz space with Gaussian reference density.
In a regular full exponential family, the maximum likelihood estimator (MLE) need not exist in the traditional sense. However, the MLE may exist in the completion of the exponential family. Existing algorithms for finding the MLE in the completion solve many linear programs; they are slow in small problems and too slow for large problems. We provide new, fast, and scalable methodology for finding the MLE in the completion of the exponential family. This methodology is based on conventional maximum likelihood computations which come close, in a sense, to finding the MLE in the completion of the…
When in a full exponential family the maximum likelihood estimate (MLE) does not exist, the MLE may exist in the Barndorff-Nielsen completion of the family. We propose a practical algorithm for finding the MLE in the completion based on repeated linear programming using the R contributed package rcdd and illustrate it with two generalized linear model examples. When the MLE for the null hypothesis lies in the completion, likelihood ratio tests of model comparison are almost unchanged from the usual case. Only the degrees of freedom need to be adjusted. When the MLE lies in the completion, conf…
The notion of multivariate total positivity has proved to be useful in finance and psychology but may be too restrictive in other applications. In this paper we propose a concept of local association, where highly connected components in a graphical model are positively associated and study its properties. Our main motivation comes from gene expression data, where graphical models have become a popular exploratory tool. The models are instances of what we term mixed convex exponential families and we show that a mixed dual likelihood estimator has simple exact properties for such families as w…
We propose extreme value analogues of natural exponential families and exponential dispersion models, and introduce the slope function as an analogue of the variance function. The set of quadratic and power slope functions characterize well-known families such as the Rayleigh, Gumbel, power, Pareto, logistic, negative exponential, Weibull and Fréchet. We show a convergence theorem for slope functions, by which we may express the classical extreme value convergence results in terms of asymptotics for extreme dispersion models. The main idea is to explore the parallels between location families …
Chentsov's theorem characterizes the Fisher information metric on statistical models as essentially the only Riemannian metric that is invariant under sufficient statistics. This implies that each statistical model is naturally equipped with a geometry, so Chentsov's theorem explains why many statistical properties can be described in geometric terms. However, despite being one of the foundational theorems of statistics, Chentsov's theorem has only been proved previously in very restricted settings or under relatively strong regularity and invariance assumptions. We therefore prove a version o…
We study the rate of convergence of posterior distributions in density estimation problems for log-densities in periodic Sobolev classes characterized by a smoothness parameter p. The posterior expected density provides a nonparametric estimation procedure attaining the optimal minimax rate of convergence under Hellinger loss if the posterior distribution achieves the optimal rate over certain uniformity classes. A prior on the density class of interest is induced by a prior on the coefficients of the trigonometric series expansion of the log-density. We show that when p is known, the posterio…
This paper studies a class of exponential family models whose canonical parameters are specified as linear functionals of an unknown infinite-dimensional slope function. The optimal minimax rates of convergence for slope function estimation are established. The estimators that achieve the optimal rates are constructed by constrained maximum likelihood estimation with parameters whose dimension grows with sample size. A change-of-measure argument, inspired by Le Cam's theory of asymptotic equivalence, is used to eliminate the bias caused by the non-linearity of exponential family models.
This paper studies a class of exponential family models whose canonical parameters are specified as linear functionals of an unknown infinite-dimensional slope function. The optimal minimax rates of convergence for slope function estimation are established. The estimators that achieve the optimal rates are constructed by constrained maximum likelihood estimation with parameters whose dimension grows with sample size. A change-of-measure argument, inspired by Le Cam's theory of asymptotic equivalence, is used to eliminate the bias caused by the nonlinearity of exponential family models.
Statistical inference may follow a frequentist approach or it may follow a Bayesian approach or it may use the minimum description length principle (MDL). Our goal is to identify situations in which these different approaches to statistical inference coincide. It is proved that for exponential families MDL and Bayesian inference coincide if and only if the renormalized saddle point approximation for the conjugated exponential family is exact. For 1-dimensional exponential families the only families with exact renormalized saddle point approximations are the Gaussian location family, the Gamma …
This paper continues the study of a kernel family which uses the Cauchy-Stieltjes kernel in place of the celebrated exponential kernel of the exponential families theory. We extend the theory to cover generating measures with support that is unbounded on one side. We illustrate the need for such an extension by showing that cubic pseudo-variance functions correspond to free-infinitely divisible laws without the first moment. We also determine the domain of means, advancing the understanding of Cauchy-Stieltjes kernel families also for compactly supported generating measures.
The minimum message length principle is an information theoretic criterion that links data compression with statistical inference. This paper studies the strict minimum message length (SMML) estimator for $d$-dimensional exponential families with continuous sufficient statistics, for all $d \ge 1$. The partition of an SMML estimator is shown to consist of convex polytopes (i.e. convex polygons when $d=2$) which can be described explicitly in terms of the assertions and coding probabilities. While this result is known, we give a new proof based on the calculus of variations, and this approach g…
Let $X_{1}=(W_{1},Y_{1}),\ldots,X_{n}=(W_{n},Y_{n})$ be $n$ pairs of independent random variables. We assume that, for each $i\in\{1,\ldots,n\}$, the conditional distribution of $Y_{i}$ given $W_{i}$ belongs to a one-parameter exponential family with parameter ${\boldsymbolγ}^{\star}(W_{i})\in{\mathbb{R}}$, or at least, is close enough to a distribution of this form. The objective of the present paper is to estimate these conditional distributions on the basis of the observation ${\boldsymbol{X}}=(X_{1},\ldots,X_{n})$ and to do so, we propose a model selection procedure together with a non-asy…
We introduce twenty four two-parameter families of advanced time series forecasting functions using a new and nonparametric approach. We also introduce the concept of powering and derive nonseasonal and seasonal models with examples in education, sales, finance and economy. We compare the performance of our twenty four models to both Holt--Winters and ARIMA models for both nonseasonal and seasonal times series. We show in particular that our models not only do not require a decomposition of a seasonal time series into trend, seasonal and random components, but leads also to substantially lower…
Sparse models are desirable for many applications across diverse domains as they can perform automatic variable selection, aid interpretability, and provide regularization. When fitting sparse models in a Bayesian framework, however, analytically obtaining a posterior distribution over the parameters of interest is intractable for all but the simplest cases. As a result practitioners must rely on either sampling algorithms such as Markov chain Monte Carlo or variational methods to obtain an approximate posterior. Mean field variational inference is a particularly simple and popular framework t…
This work studies the large sample properties of the posterior-based inference in the curved exponential family under increasing dimension. The curved structure arises from the imposition of various restrictions on the model, such as moment restrictions, and plays a fundamental role in econometrics and others branches of data analysis. We establish conditions under which the posterior distribution is approximately normal, which in turn implies various good properties of estimation and inference procedures based on the posterior. In the process we also revisit and improve upon previous results …
Many statistical experiments involve comparing multiple population groups. For example, a public opinion poll may ask which of several political candidates commands the most support; a social scientific survey may report the most common of several responses to a question; or, a clinical trial may compare binary patient outcomes under several treatment conditions to determine the most effective treatment. Having observed the "winner" (largest observed response) in a noisy experiment, it is natural to ask whether that candidate, survey response, or treatment is actually the "best" (stochasticall…
In this paper we find nonasymptotic exponential upper bounds for the deviation in the ergodic theorem for families of homogeneous Markov processes. We find some sufficient conditions for geometric ergodicity uniformly over a parametric family. We apply this property to the nonasymptotic nonparametric estimation problem for ergodic diffusion processes.
Stein's unbiased risk estimate (SURE) was proposed by Stein for the independent, identically distributed (iid) Gaussian model in order to derive estimates that dominate least-squares (LS). In recent years, the SURE criterion has been employed in a variety of denoising problems for choosing regularization parameters that minimize an estimate of the mean-squared error (MSE). However, its use has been limited to the iid case which precludes many important applications. In this paper we begin by deriving a SURE counterpart for general, not necessarily iid distributions from the exponential family.…
We consider the problem of finding anomalies in a $d$-dimensional field of independent random variables $\{Y_i\}_{i \in \left\{1,...,n\right\}^d}$, each distributed according to a one-dimensional natural exponential family $\mathcal F = \left\{F_θ\right\}_{θ\inΘ}$. Given some baseline parameter $θ_0 \inΘ$, the field is scanned using local likelihood ratio tests to detect from a (large) given system of regions $\mathcal{R}$ those regions $R \subset \left\{1,...,n\right\}^d$ with $θ_i \neq θ_0$ for some $i \in R$. We provide a unified methodology which controls the overall family wise error (FWE…
Analysis of low-degree polynomial algorithms is a powerful, newly-popular method for predicting computational thresholds in hypothesis testing problems. One limitation of current techniques for this analysis is their restriction to Bernoulli and Gaussian distributions. We expand this range of possibilities by performing the low-degree analysis of hypothesis testing for the Morris class of exponential families, giving a unified treatment of Gaussian, Poisson, gamma, binomial, negative binomial, and generalized hyperbolic secant distributions. We then give several algorithmic applications. 1. In…
In their fundamental paper on cubic variance functions, Letac and Mora (The Annals of Statistics,1990) presented a systematic, rigorous and comprehensive study of natural exponential families on the real line, their characterization through their variance functions and mean value parameterization. They presented a section that for some reason has been left unnoticed. This section deals with the construction of variance functions associated with natural exponential families of counting distributions on the set of nonnegative integers and allows to find the corresponding generating measures. As …
