发散准则在连续族中的几个应用

Divergence functions in dually flat spaces and their properties

In the field of statistics, many kind of divergence functions have been studied as an amount which measures the discrepancy between two probability distributions. In the differential geometrical approach in statistics (information geometry), dually flat spaces play a key role. In a dually flat space, there exist dual affine coordinate systems and strictly convex functions called potential and a canonical divergence is naturally introduced as a function of the affine coordinates and potentials. The canonical divergence satisfies a relational expression called triangular relation. This can be re…

Hölder散度与函数密度幂散度之间的关系：交集与推广

Relationship between Hölder Divergence and Functional Density Power Divergence: Intersection and Generalization

本研究探讨Hölder散度与函数密度幂散度的交集与推广，定义广义的$\xi$-Hölder散度，并证明其非负性及与复合评分规则的联系。

Optimal Bounds between $f$-Divergences and Integral Probability Metrics

The families of $f$-divergences (e.g. the Kullback-Leibler divergence) and Integral Probability Metrics (e.g. total variation distance or maximum mean discrepancies) are widely used to quantify the similarity between probability distributions. In this work, we systematically study the relationship between these two families from the perspective of convex duality. Starting from a tight variational representation of the $f$-divergence, we derive a generalization of the moment-generating function, which we show exactly characterizes the best lower bound of the $f$-divergence as a function of a gi…

An inverse Sanov theorem for exponential families

We prove the large deviation principle (LDP) for posterior distributions arising from subfamilies of full exponential families, allowing misspecification of the model. Moreover, motivated by the so-called inverse Sanov Theorem (see e.g. Ganesh and O'Connell 1999 and 2000), we prove the LDP for the corresponding maximum likelihood estimator, and we study the relationship between rate functions. In our setting, even in the non misspecified case, it is not true in general that the rate functions for posterior distributions and for maximum likelihood estimators are Kullback-Leibler divergences wit…

Comparison of Channels: Criteria for Domination by a Symmetric Channel

This paper studies the basic question of whether a given channel $V$ can be dominated (in the precise sense of being more noisy) by a $q$-ary symmetric channel. The concept of "less noisy" relation between channels originated in network information theory (broadcast channels) and is defined in terms of mutual information or Kullback-Leibler divergence. We provide an equivalent characterization in terms of $χ^2$-divergence. Furthermore, we develop a simple criterion for domination by a $q$-ary symmetric channel in terms of the minimum entry of the stochastic matrix defining the channel $V$. The…

Hierarchical Low-rank Structure of Parameterized Distributions

This note shows that the matrix forms of several one-parameter distribution families satisfy a hierarchical low-rank structure. Such families of distributions include binomial, Poisson, and $χ^2$ distributions. The proof is based on a uniform relative bound of a related divergence function. Numerical results are provided to confirm the theoretical findings.

Projection Theorems of Divergences and Likelihood Maximization Methods

Projection theorems of divergences enable us to find reverse projection of a divergence on a specific statistical model as a forward projection of the divergence on a different but rather "simpler" statistical model, which, in turn, results in solving a system of linear equations. Reverse projection of divergences are closely related to various estimation methods such as the maximum likelihood estimation or its variants in robust statistics. We consider projection theorems of three parametric families of divergences that are widely used in robust statistics, namely the Rényi divergences (or th…

Density Estimation in Infinite Dimensional Exponential Families

In this paper, we consider an infinite dimensional exponential family, $\mathcal{P}$ of probability densities, which are parametrized by functions in a reproducing kernel Hilbert space, $H$ and show it to be quite rich in the sense that a broad class of densities on $\mathbb{R}^d$ can be approximated arbitrarily well in Kullback-Leibler (KL) divergence by elements in $\mathcal{P}$. The main goal of the paper is to estimate an unknown density, $p_0$ through an element in $\mathcal{P}$. Standard techniques like maximum likelihood estimation (MLE) or pseudo MLE (based on the method of sieves), wh…