In this paper we study a class of exponential family on permutations, which includes some of the commonly studied Mallows models. We show that the pseudo-likelihood estimator for the natural parameter in the exponential family is asymptotically normal, with an explicit variance. Using this, we are able to construct asymptotically valid confidence intervals. We also show that the MLE for the same problem is consistent everywhere, and asymptotically normal at the origin. In this special case, the asymptotic variance of the cost effective pseudo-likelihood estimator turns out to be the same as the cost prohibitive MLE. To the best of our knowledge, this is the first inference result on permutation models including Mallows models, excluding the very special case of Mallows model with Kendall's Tau.
We prove the large deviation principle (LDP) for posterior distributions arising from subfamilies of full exponential families, allowing misspecification of the model. Moreover, motivated by the so-called inverse Sanov Theorem (see e.g. Ganesh and O'Connell 1999 and 2000), we prove the LDP for the corresponding maximum likelihood estimator, and we study the relationship between rate functions. In our setting, even in the non misspecified case, it is not true in general that the rate functions for posterior distributions and for maximum likelihood estimators are Kullback-Leibler divergences wit…
We characterize the existence of the maximum likelihood estimator for discrete exponential families. Our criterion is simple to apply as we show in various settings, most notably for exponential models of random graphs. As an application, we point out the size of independent identically distributed samples for which the maximum likelihood estimator exists with high probability.
The logistic regression model is known to converge to a Poisson point process model if the binary response tends to infinitely imbalanced. In this paper, it is shown that this phenomenon is universal in a wide class of link functions on binomial regression. The proof relies on the extreme value theory. For the logit, probit and complementary log-log link functions, the intensity measure of the point process becomes an exponential family. For some other link functions, deformed exponential families appear. A penalized maximum likelihood estimator for the Poisson point process model is suggested…
We study exponential families of distributions that are multivariate totally positive of order 2 (MTP2), show that these are convex exponential families, and derive conditions for existence of the MLE. Quadratic exponential familes of MTP2 distributions contain attractive Gaussian graphical models and ferromagnetic Ising models as special examples. We show that these are defined by intersecting the space of canonical parameters with a polyhedral cone whose faces correspond to conditional independence relations. Hence MTP2 serves as an implicit regularizer for quadratic exponential families and…
This paper studies a class of exponential family models whose canonical parameters are specified as linear functionals of an unknown infinite-dimensional slope function. The optimal minimax rates of convergence for slope function estimation are established. The estimators that achieve the optimal rates are constructed by constrained maximum likelihood estimation with parameters whose dimension grows with sample size. A change-of-measure argument, inspired by Le Cam's theory of asymptotic equivalence, is used to eliminate the bias caused by the non-linearity of exponential family models.
Chentsov's theorem characterizes the Fisher information metric on statistical models as essentially the only Riemannian metric that is invariant under sufficient statistics. This implies that each statistical model is naturally equipped with a geometry, so Chentsov's theorem explains why many statistical properties can be described in geometric terms. However, despite being one of the foundational theorems of statistics, Chentsov's theorem has only been proved previously in very restricted settings or under relatively strong regularity and invariance assumptions. We therefore prove a version o…
There has been an explosion of interest in statistical models for analyzing network data, and considerable interest in the class of exponential random graph (ERG) models, especially in connection with difficulties in computing maximum likelihood estimates. The issues associated with these difficulties relate to the broader structure of discrete exponential families. This paper re-examines the issues in two parts. First we consider the closure of $k$-dimensional exponential families of distribution with discrete base measure and polyhedral convex support $\mathrm{P}$. We show that the normal fa…
In this note we prove the dual representation formula of the divergence between two distributions in a parametric model. Resulting estimators for the divergence as for the parameter are derived. These estimators do not make use of any grouping nor smoothing. It is proved that all differentiable divergences induce the same estimator of the parameter on any regular exponential family, which is nothing else but the MLE.
We propose a novel approach for density estimation with exponential families for the case when the true density may not fall within the chosen family. Our approach augments the sufficient statistics with features designed to accumulate probability mass in the neighborhood of the observed points, resulting in a non-parametric model similar to kernel density estimators. We show that under mild conditions, the resulting model uses only the sufficient statistics if the density is within the chosen exponential family, and asymptotically, it approximates densities outside of the chosen exponential f…
Free exponential families have been previously introduced as a special case of the q-exponential family. We show that free exponential families arise also from a procedure analogous to the definition of exponential families by using the Cauchy-Stieltjes kernel instead of the exponential kernel. We use this approach to re-derive several known results and to study further similarities with exponential families and reproductive exponential models.
Using the technique developed in approximation theory, we construct examples of exponential families of infinitely divisible laws which can be viewed as deformations of the normal, gamma, and Poisson exponential families. Replacing the differential equation of approximation theory by a q-differential equation, we define the q-exponential families, and we identify all q-exponential families with quadratic variance functions when |q|<1. We elaborate on the case of q=0 which is related to free convolution of measures. We conclude by considering briefly the case q>1, and other related generalizati…
Let G = An be the graph corresponding to the graphical model of nearest neighbour interaction in a Gaussian character. We study Natural Exponential Families( NEF) ofWishart distributions on convex cones QG and PG, where PG is the cone of positive definite real symmetric matrices with obligatory zeros prescribed by G, and QG is the dual cone of PG. The Wishart NEF that we construct include Wishart distributions considered earlier by Lauritzen (1996) and Letac and Massam (2007) for models based on decomposable graphs. Our approach is however different and allows us to study the basic objects ofW…
RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；The University of Tokyo（东京大学）
In this paper, we give a method to construct "good" exponential families systematically by representation theory. More precisely, we consider a homogeneous space $G/H$ as a sample space and construct an exponential family invariant under the transformation group $G$ by using a representation of $G$. The method generates widely used exponential families such as normal, gamma, Bernoulli, categorical, Wishart, von Mises, Fisher-Bingham and hyperboloid distributions.
This article studies exponential families $\mathcal{E}$ on finite sets such that the information divergence $D(P\|\mathcal{E})$ of an arbitrary probability distribution from $\mathcal{E}$ is bounded by some constant $D>0$. A particular class of low-dimensional exponential families that have low values of $D$ can be obtained from partitions of the state space. The main results concern optimality properties of these partition exponential families. Exponential families where $D=\log(2)$ are studied in detail. This case is special, because if $D<\log(2)$, then $\mathcal{E}$ contains all probabilit…
Exponential families comprise a broad class of statistical models and parametric families like normal distributions, binomial distributions, gamma distributions or exponential distributions. Thereby the formal representation of its probability distributions induces a confined intrinsic structure, which appears to be that of a dually flat statistical manifold. Conversely it can be shown, that any dually flat statistical manifold, which is given by a regular Bregman divergence uniquely induced a regular exponential family, such that exponential families may - with some restrictions - be regarded…
Equivalence on exponential families as Hessian manifolds and classification of exponential families of order 1 with constant Hessian sectional curvature
Tokai University（东海大学）；The University of Tokyo（东京大学）
We characterize bivariate natural exponential families having the diagonal of the variance function of the form \[ \textrm{diag} V(m_1,m_2)=\left(Am_1^2+am_1+bm_2+e,Am_2^2+cm_1+dm_2+f\right), \] with $A<0$ and $a,\ldots,f\in\mathbb{R}$. The solution of the problem relies on finding the conditions under which a specific parametric family of functions consists of Laplace transforms of some probability measures.
Exponential families are the workhorses of parametric modelling theory. One reason for their popularity is their associated inference theory, which is very clean, both from a theoretical and a computational point of view. One way in which this set of tools can be enriched in a natural and interpretable way is through mixing. This paper develops and applies the idea of local mixture modelling to exponential families. It shows that the highly interpretable and flexible models which result have enough structure to retain the attractive inferential properties of exponential families. In particular…
The closure of a discrete exponential family is described by a finite set of equations corresponding to the circuits of an underlying oriented matroid. These equations are similar to the equations used in algebraic statistics, although they need not be polynomial in the general case. This description allows for a combinatorial study of the possible support sets in the closure of an exponential family. If two exponential families induce the same oriented matroid, then their closures have the same support sets. Furthermore, the positive cocircuits give a parameterization of the closure of the ex…
The lower bound of Cramer and Rao is generalized to pairs of families of probability distributions, one of which is escort to the other. This bound is optimal for certain families, called phi-exponential in the paper. Their dual structure is explored.
We describe a procedure to introduce general dependence structures on a set of random variables. These include order-$q$ moving average-type structures, as well as seasonal, periodic, spatial and spatio-temporal dependences. The invariant marginal distribution can be in any family that is conjugate to an exponential family with quadratic variance function. Dependence is induced via a set of suitable latent variables whose conditional distribution mirrors the sampling distribution in a Bayesian conjugate analysis of such exponential families. We obtain strict stationarity as a special case.
