This paper is focused on derivations of data-processing and majorization inequalities for $f$-divergences, and their applications in information theory and statistics. For the accessibility of the material, the main results are first introduced without proofs, followed by exemplifications of the theorems with further related analytical results, interpretations, and information-theoretic applications. One application refers to the performance analysis of list decoding with either fixed or variable list sizes; some earlier bounds on the list decoding error probability are reproduced in a unified way, and new bounds are obtained and exemplified numerically. Another application is related to a study of the quality of approximating a probability mass function, induced by the leaves of a Tunstall tree, by an equiprobable distribution. The compression rates of finite-length Tunstall codes are further analyzed for asserting their closeness to the Shannon entropy of a memoryless and stationary discrete source. Almost all the analysis is relegated to the appendices, which form a major part of this manuscript.
A natural link between the notions of majorization and strongly Sperner posets is elucidated. It is then used to obtain a variety of consequences, including new Rényi entropy inequalities for sums of independent, integer-valued random variables.
This work provides data-processing and majorization inequalities for $f$-divergences, and it considers some of their applications to coding problems. This work also provides tight bounds on the Rényi entropy of a function of a discrete random variable with a finite number of possible values, where the considered function is not one-to-one, and their derivation is based on majorization and the Schur-concavity of the Rényi entropy. One application of the $f$-divergence inequalities refers to the performance analysis of list decoding with either fixed or variable list sizes; some earlier bounds o…
Matrix Majorization in Large Samples with Varying Support Restrictions
National University of Singapore（新加坡国立大学）；Centre for Quantum Technologies, National University of Singapore（新加坡国立大学量子技术中心）；Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电气与计算机工程系）
Fano's inequality is one of the most elementary, ubiquitous, and important tools in information theory. Using majorization theory, Fano's inequality is generalized to a broad class of information measures, which contains those of Shannon and Rényi. When specialized to these measures, it recovers and generalizes the classical inequalities. Key to the derivation is the construction of an appropriate conditional distribution inducing a desired marginal distribution on a countably infinite alphabet. The construction is based on the infinite-dimensional version of Birkhoff's theorem proven by Révés…
This paper is focused on $f$-divergences, consisting of three main contributions. The first one introduces integral representations of a general $f$-divergence by means of the relative information spectrum. The second part provides a new approach for the derivation of $f$-divergence inequalities, and it exemplifies their utility in the setup of Bayesian binary hypothesis testing. The last part of this paper further studies the local behavior of $f$-divergences.
This paper develops systematic approaches to obtain $f$-divergence inequalities, dealing with pairs of probability measures defined on arbitrary alphabets. Functional domination is one such approach, where special emphasis is placed on finding the best possible constant upper bounding a ratio of $f$-divergences. Another approach used for the derivation of bounds among $f$-divergences relies on moment inequalities and the logarithmic-convexity property, which results in tight bounds on the relative entropy and Bhattacharyya distance in terms of $χ^2$ divergences. A rich variety of bounds are sh…
The aim of this paper is to discuss new results concerning some kinds of parametric extended entropies and divergences. As a result of our studies for mathematical properties on entropy and divergence, we give new bounds for the Tsallis quasilinear entropy and divergence by applying the Hermite-Hadamard inequality. We also give bounds for biparametrical extended entropies and divergences which have been given in \cite{7}. In addition, we study $(r,q)$-quasilinear entropies and divergences as alternative biparametrical extended entropy and divergence, and then we give bounds for them. Finally w…
Entropies, cross-entropies and Rényi divergence: sharp three-term inequalities for probability density functions
Universidad Rey Juan Carlos（雷乌斯·胡安·卡洛斯大学）；Data, Complex Networks and Cybersecurity Research Institute, Universidad Rey Juan Carlos（数据、复杂网络与网络安全研究所，雷乌斯·胡安·卡洛斯大学）
Technion-Israel Institute of Technology（以色列理工学院）；Princeton University（普林斯顿大学）
This paper considers derivation of $f$-divergence inequalities via the approach of functional domination. Bounds on an $f$-divergence based on one or several other $f$-divergences are introduced, dealing with pairs of probability measures defined on arbitrary alphabets. In addition, a variety of bounds are shown to hold under boundedness assumptions on the relative information. The journal paper, which includes more approaches for the derivation of f-divergence inequalities and proofs, is available on the arXiv at https://arxiv.org/abs/1508.00335, and it has been published in the IEEE Trans. o…
We investigate quantum Rényi entropic quantities, specifically those derived from 'sandwiched' divergence. This divergence is one of several proposed Rényi generalisations of the quantum relative entropy. We may define Rényi generalisations of the quantum conditional entropy and mutual information in terms of this divergence, from which they inherit many desirable properties. However, these quantities lack some of the convenient structure of their Shannon and von Neumann counterparts. We attempt to bridge this gap by establishing divergence inequalities for valid combinations of Rényi order wh…
Tighter Information-Theoretic Generalization Bounds via a Novel Class of Change of Measure Inequalities
Department of Electrical and Electronic Engineering, Imperial College London（帝国理工学院伦敦分校电子与电气工程系）；Department of Information Engineering, The Chinese University of Hong Kong（香港中文大学信息工程系）
Information-theoretic measures such as the entropy, cross-entropy and the Kullback-Leibler divergence between two mixture models is a core primitive in many signal processing tasks. Since the Kullback-Leibler divergence of mixtures provably does not admit a closed-form formula, it is in practice either estimated using costly Monte-Carlo stochastic integration, approximated, or bounded using various techniques. We present a fast and generic method that builds algorithmically closed-form lower and upper bounds on the entropy, the cross-entropy and the Kullback-Leibler divergence of mixtures. We …
United States CCDC Army Research Laboratory（美国陆军作战能力发展司令部研究实验室）；Indian Institute of Technology Palakkad（印度理工学院帕拉卡德校区）
We examine the role of information geometry in the context of classical Cramér-Rao (CR) type inequalities. In particular, we focus on Eguchi's theory of obtaining dualistic geometric structures from a divergence function and then applying Amari-Nagoaka's theory to obtain a CR type inequality. The classical deterministic CR inequality is derived from Kullback-Leibler (KL)-divergence. We show that this framework could be generalized to other CR type inequalities through four examples: $α$-version of CR inequality, generalized CR inequality, Bayesian CR inequality, and Bayesian $α$-CR inequality.…
A simple method is shown to provide optimal variational bounds on $f$-divergences with possible constraints on relative information extremums. Known results are refined or proved to be optimal as particular cases.
Tight Contraction Rates for Primitive Channels under Quantum $f$-Divergences
School of Computing National University of Singapore Singapore 117417, Singapore；Computer Engineering \& Centre for Quantum Technologies National University of Singapore Singapore 117543, Singapore；Centre for Quantum Technologies National University of Singapore Singapore 117543, Singapore
Data processing inequalities for $f$-divergences can be sharpened using constants called "contraction coefficients" to produce strong data processing inequalities. For any discrete source-channel pair, the contraction coefficients for $f$-divergences are lower bounded by the contraction coefficient for $χ^2$-divergence. In this paper, we elucidate that this lower bound can be achieved by driving the input $f$-divergences of the contraction coefficients to zero. Then, we establish a linear upper bound on the contraction coefficients for a certain class of $f$-divergences using the contraction c…
École Polytechnique（巴黎综合理工学院）；Sony Computer Science Laboratories（索尼计算机科学研究所）；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；University of Geneva（日内瓦大学）
We describe a framework to build distances by measuring the tightness of inequalities, and introduce the notion of proper statistical divergences and improper pseudo-divergences. We then consider the Hölder ordinary and reverse inequalities, and present two novel classes of Hölder divergences and pseudo-divergences that both encapsulate the special case of the Cauchy-Schwarz divergence. We report closed-form formulas for those statistical dissimilarities when considering distributions belonging to the same exponential family provided that the natural parameter space is a cone (e.g., multivaria…
We study concentration inequalities for the Kullback--Leibler (KL) divergence between the empirical distribution and the true distribution. Applying a recursion technique, we improve over the method of types bound uniformly in all regimes of sample size $n$ and alphabet size $k$, and the improvement becomes more significant when $k$ is large. We discuss the applications of our results in obtaining tighter concentration inequalities for $L_1$ deviations of the empirical distribution from the true distribution, and the difference between concentration around the expectation or zero. We also obta…
