In this paper we have considered a single inequality having 11 known divergence measures. This inequality include measures like: Jeffryes-Kullback-Leiber J-divergence, Jensen-Shannon divergence (Burbea-Rao, 1982), arithmetic-geometric mean divergence (Taneja, 1995), Hellinger discrimination, symmetric chi-square divergence, triangular discrimination, etc. All these measures are well-known in the literature on Information theory and Statistics. This sequence of 11 measures also include measures due to Kumar and Johnson (2005) and Jain and Srivastava (2007). Three measures arising due to some mean divergences also appears in this inequality. Based on non-negative differences arising due to this single inequality of 11 measures, we have put more than 40 divergence measures in nested or sequential form. Idea of reverse inequalities is also introduced.
There are three classical divergence measures exist in the literature on information theory and statistics. These are namely, Jeffryes-Kullback-Leiber J-divergence. Sibson-Burbea-Rao Jensen-Shannon divegernce and Taneja arithemtic-geometric mean divergence. These three measures bear an interesting relationship among each other and are based on logarithmic expressions. The divergence measures like Hellinger discrimination, symmetric chi-square divergence, and triangular discrimination are also known in the literature and are not based on logarithmic expressions. Past years Dragomir et al., Kuma…
Expressions for (EPI Shannon type) Divergence-Power Inequalities (DPI) in two cases (time-discrete and band-limited time-continuous) of stationary random processes are given. The new expressions connect the divergence rate of the sum of independent processes, the individual divergence rate of each process, and their power spectral densities. All divergences are between a process and a Gaussian process with same second order statistics, and are assumed to be finite. A new proof of the Shannon entropy-power inequality EPI, based on the relationship between divergence and causal minimum mean-squa…
In this paper we have considered two one parametric generalizations. These two generalizations have in articular the well known measures such as: J-divergence, Jensen-Shannon divergence and Arithmetic-Geometric mean divergence. These three measures are with logarithmic expressions. Also, we have particular cases the measures such as: Hellinger discrimination, symmetric chi-square divergence, and triangular discrimination. These three measures are also well-known in the literature of statistics, and are without logarithmic expressions. Still, we have one more non logarithmic measure as particul…
In 1938, Gini studied a mean having two parameters. Later, many authors studied properties of this mean. In particular, it contains the famous means as harmonic, geometric, arithmetic, etc. Here we considered a sequence of inequalities arising due to particular values of each parameter of Gini's mean. This sequence generates many nonnegative differences. Not all of them are convex. We have studied here convexity of these differences and again established new sequences of inequalities of these differences. Considering in terms of probability distributions these differences, we have made connect…
In 1938, Gini studied a mean having two parameters. Later, many authors studied properties of this mean. It contains as particular cases the famous means such as harmonic, geometric, arithmetic, etc. Also it contains, the power mean of order r and Lehmer mean as particular cases. In this paper we have considered inequalities arising due to Gini-Mean and Heron's mean, and improved them based on the results recently studied by the author (Taneja, 2011).
We generalise the classical Pinsker inequality which relates variational divergence to Kullback-Liebler divergence in two ways: we consider arbitrary f-divergences in place of KL divergence, and we assume knowledge of a sequence of values of generalised variational divergences. We then develop a best possible inequality for this doubly generalised situation. Specialising our result to the classical case provides a new and tight explicit bound relating KL to variational divergence (solving a problem posed by Vajda some 40 years ago). The solution relies on exploiting a connection between diverg…
Entropies, cross-entropies and Rényi divergence: sharp three-term inequalities for probability density functions
Universidad Rey Juan Carlos（雷乌斯·胡安·卡洛斯大学）；Data, Complex Networks and Cybersecurity Research Institute, Universidad Rey Juan Carlos（数据、复杂网络与网络安全研究所，雷乌斯·胡安·卡洛斯大学）
This paper is focused on $f$-divergences, consisting of three main contributions. The first one introduces integral representations of a general $f$-divergence by means of the relative information spectrum. The second part provides a new approach for the derivation of $f$-divergence inequalities, and it exemplifies their utility in the setup of Bayesian binary hypothesis testing. The last part of this paper further studies the local behavior of $f$-divergences.
The aim of this paper is to discuss new results concerning some kinds of parametric extended entropies and divergences. As a result of our studies for mathematical properties on entropy and divergence, we give new bounds for the Tsallis quasilinear entropy and divergence by applying the Hermite-Hadamard inequality. We also give bounds for biparametrical extended entropies and divergences which have been given in \cite{7}. In addition, we study $(r,q)$-quasilinear entropies and divergences as alternative biparametrical extended entropy and divergence, and then we give bounds for them. Finally w…
Divergences often play important roles for study in information science so that it is indispensable to investigate their fundamental properties. There is also a mathematical significance of such results. In this paper, we introduce some parametric extended divergences combining Jeffreys divergence and Tsallis entropy defined by generalized logarithmic functions, which lead to new inequalities. In addition, we give lower bounds for one-parameter extended Fermi-Dirac and Bose-Einstein divergences. Finally, we establish some inequalities for the Tsallis entropy, the Tsallis relative entropy and s…
We study conditions on $f$ under which an $f$-divergence $D_f$ will satisfy $D_f \geq c_f V^2$ or $D_f \geq c_{2,f} V^2 + c_{4,f} V^4$, where $V$ denotes variational distance and the coefficients $c_f$, $c_{2,f}$ and $c_{4,f}$ are {\em best possible}. As a consequence, we obtain lower bounds in terms of $V$ for many well known distance and divergence measures. For instance, let $D_{(α)} (P,Q) = [α(α-1)]^{-1} [\int q^α p^{1-α} d μ-1]$ and ${\cal I}_α(P,Q) = (α-1)^{-1} \log [\int p^αq^{1-α} d μ]$ be respectively the {\em relative information of type} ($1-α$) and {\em Rényi's information gain of …
This paper develops systematic approaches to obtain $f$-divergence inequalities, dealing with pairs of probability measures defined on arbitrary alphabets. Functional domination is one such approach, where special emphasis is placed on finding the best possible constant upper bounding a ratio of $f$-divergences. Another approach used for the derivation of bounds among $f$-divergences relies on moment inequalities and the logarithmic-convexity property, which results in tight bounds on the relative entropy and Bhattacharyya distance in terms of $χ^2$ divergences. A rich variety of bounds are sh…
We propose a modified $χ^β$-divergence, give some of its properties, and show that this leads to the definition of a generalized Fisher information. We give generalized Cramér-Rao inequalities, involving this Fisher information, an extension of the Fisher information matrix, and arbitrary norms and power of the estimation error. In the case of a location parameter, we obtain new characterizations of the generalized $q$-Gaussians, for instance as the distribution with a given moment that minimizes the generalized Fisher information. Finally we indicate how the generalized Fisher information can…
$f$-divergences are a general class of divergences between probability measures which include as special cases many commonly used divergences in probability, mathematical statistics and information theory such as Kullback-Leibler divergence, chi-squared divergence, squared Hellinger distance, total variation distance etc. In this paper, we study the problem of maximizing or minimizing an $f$-divergence between two probability measures subject to a finite number of constraints on other $f$-divergences. We show that these infinite-dimensional optimization problems can all be reduced to optimizat…
In this paper we shall consider some famous means such as arithmetic, harmonic, geometric, root square mean, etc. Considering the difference of these means, we can establish. some inequalities among them. Interestingly, the difference of mean considered is convex functions. Applying some properties, upper bounds on the probability of error are established in this paper. It is also shown that the results obtained are sharper than obtained directly applying known inequalities.
We investigate quantum Rényi entropic quantities, specifically those derived from 'sandwiched' divergence. This divergence is one of several proposed Rényi generalisations of the quantum relative entropy. We may define Rényi generalisations of the quantum conditional entropy and mutual information in terms of this divergence, from which they inherit many desirable properties. However, these quantities lack some of the convenient structure of their Shannon and von Neumann counterparts. We attempt to bridge this gap by establishing divergence inequalities for valid combinations of Rényi order wh…
Tighter Information-Theoretic Generalization Bounds via a Novel Class of Change of Measure Inequalities
Department of Electrical and Electronic Engineering, Imperial College London（帝国理工学院伦敦分校电子与电气工程系）；Department of Information Engineering, The Chinese University of Hong Kong（香港中文大学信息工程系）
Information-theoretic measures such as the entropy, cross-entropy and the Kullback-Leibler divergence between two mixture models is a core primitive in many signal processing tasks. Since the Kullback-Leibler divergence of mixtures provably does not admit a closed-form formula, it is in practice either estimated using costly Monte-Carlo stochastic integration, approximated, or bounded using various techniques. We present a fast and generic method that builds algorithmically closed-form lower and upper bounds on the entropy, the cross-entropy and the Kullback-Leibler divergence of mixtures. We …
Technion-Israel Institute of Technology（以色列理工学院）；Princeton University（普林斯顿大学）
This paper considers derivation of $f$-divergence inequalities via the approach of functional domination. Bounds on an $f$-divergence based on one or several other $f$-divergences are introduced, dealing with pairs of probability measures defined on arbitrary alphabets. In addition, a variety of bounds are shown to hold under boundedness assumptions on the relative information. The journal paper, which includes more approaches for the derivation of f-divergence inequalities and proofs, is available on the arXiv at https://arxiv.org/abs/1508.00335, and it has been published in the IEEE Trans. o…
A simple method is shown to provide optimal variational bounds on $f$-divergences with possible constraints on relative information extremums. Known results are refined or proved to be optimal as particular cases.
United States CCDC Army Research Laboratory（美国陆军作战能力发展司令部研究实验室）；Indian Institute of Technology Palakkad（印度理工学院帕拉卡德校区）
We examine the role of information geometry in the context of classical Cramér-Rao (CR) type inequalities. In particular, we focus on Eguchi's theory of obtaining dualistic geometric structures from a divergence function and then applying Amari-Nagoaka's theory to obtain a CR type inequality. The classical deterministic CR inequality is derived from Kullback-Leibler (KL)-divergence. We show that this framework could be generalized to other CR type inequalities through four examples: $α$-version of CR inequality, generalized CR inequality, Bayesian CR inequality, and Bayesian $α$-CR inequality.…
We give bounds on the difference between the weighted arithmetic mean and the weighted geometric mean. These imply refined Young inequalities and the reverses of the Young inequality. We also study some properties on the difference between the weighted arithmetic mean and the weighted geometric mean. Applying the newly obtained inequalities, we show some results on the Tsallis divergence, the Rényi divergence, the Jeffreys-Tsallis divergence and the Jensen-Shannon-Tsallis divergence.
This paper is focused on derivations of data-processing and majorization inequalities for $f$-divergences, and their applications in information theory and statistics. For the accessibility of the material, the main results are first introduced without proofs, followed by exemplifications of the theorems with further related analytical results, interpretations, and information-theoretic applications. One application refers to the performance analysis of list decoding with either fixed or variable list sizes; some earlier bounds on the list decoding error probability are reproduced in a unified…
We study data processing inequalities that are derived from a certain class of generalized information measures, where a series of convex functions and multiplicative likelihood ratios are nested alternately. While these information measures can be viewed as a special case of the most general Zakai-Ziv generalized information measure, this special nested structure calls for attention and motivates our study. Specifically, a certain choice of the convex functions leads to an information measure that extends the notion of the Bhattacharyya distance (or the Chernoff divergence): While the ordinar…
Tight Contraction Rates for Primitive Channels under Quantum $f$-Divergences
School of Computing National University of Singapore Singapore 117417, Singapore；Computer Engineering \& Centre for Quantum Technologies National University of Singapore Singapore 117543, Singapore；Centre for Quantum Technologies National University of Singapore Singapore 117543, Singapore
From geometrical point of view, Eve (2003) studied seven means. These means are Harmonic, Geometric, Arithmetic, Heronian, Contra-harmonic, Root-mean square and Centroidal mean. We have considered for the first time a new measure calling generalized triangular discrimination. Inequalities among non-negative differences arising due to seven means and particular cases of generalized triangular discrimination are considered. Some new generating measures and their exponential representations are also presented.
In this work, we connect the problem of bounding the expected generalisation error with transportation-cost inequalities. Exposing the underlying pattern behind both approaches we are able to generalise them and go beyond Kullback-Leibler Divergences/Mutual Information and sub-Gaussian measures. In particular, we are able to provide a result showing the equivalence between two families of inequalities: one involving functionals and one involving measures. This result generalises the one proposed by Bobkov and Götze that connects transportation-cost inequalities with concentration of measure. M…
École Polytechnique（巴黎综合理工学院）；Sony Computer Science Laboratories（索尼计算机科学研究所）；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；University of Geneva（日内瓦大学）
We describe a framework to build distances by measuring the tightness of inequalities, and introduce the notion of proper statistical divergences and improper pseudo-divergences. We then consider the Hölder ordinary and reverse inequalities, and present two novel classes of Hölder divergences and pseudo-divergences that both encapsulate the special case of the Cauchy-Schwarz divergence. We report closed-form formulas for those statistical dissimilarities when considering distributions belonging to the same exponential family provided that the natural parameter space is a cone (e.g., multivaria…
We study concentration inequalities for the Kullback--Leibler (KL) divergence between the empirical distribution and the true distribution. Applying a recursion technique, we improve over the method of types bound uniformly in all regimes of sample size $n$ and alphabet size $k$, and the improvement becomes more significant when $k$ is large. We discuss the applications of our results in obtaining tighter concentration inequalities for $L_1$ deviations of the empirical distribution from the true distribution, and the difference between concentration around the expectation or zero. We also obta…
Over the past few years, a family of interesting new inequalities for the entropies of sums and differences of random variables has been developed by Ruzsa, Tao and others, motivated by analogous results in additive combinatorics. The present work extends these earlier results to the case of random variables taking values in $\mathbb{R}^n$ or, more generally, in arbitrary locally compact and Polish abelian groups. We isolate and study a key quantity, the Ruzsa divergence between two probability distributions, and we show that its properties can be used to extend the earlier inequalities to the…
Institute of Science and Technology Austria（奥地利科学技术研究所）；Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
This paper focuses on parameter estimation and introduces a new method for lower bounding the Bayesian risk. The method allows for the use of virtually \emph{any} information measure, including Rényi's $α$, $φ$-Divergences, and Sibson's $α$-Mutual Information. The approach considers divergences as functionals of measures and exploits the duality between spaces of measures and spaces of functions. In particular, we show that one can lower bound the risk with any information measure by upper bounding its dual via Markov's inequality. We are thus able to provide estimator-independent impossibilit…
