The $α$-divergences include the well-known Kullback-Leibler divergence, Hellinger distance and $χ^2$-divergence. In this paper, we derive differential and integral relations between the $α$-divergences that are generalizations of the relation between the Kullback-Leibler divergence and the $χ^2$-divergence. We also show tight lower bounds for the $α$-divergences under given means and variances. In particular, we show a necessary and sufficient condition such that the binary divergences, which are divergences between probability measures on the same $2$-point set, always attain lower bounds. Kullback-Leibler divergence, Hellinger distance, and $χ^2$-divergence satisfy this condition.
Minimizing divergence measures under a constraint is an important problem. We derive a sufficient condition that binary divergence measures provide lower bounds for symmetric divergence measures under a given triangular discrimination or given means and variances. Assuming this sufficient condition, the former bounds are always tight, and the latter bounds are tight when two probability measures have the same variance. An application of these results for nonequilibrium physics is provided.
Tight bounds for several symmetric divergence measures are introduced, given in terms of the total variation distance. Each of these bounds is attained by a pair of 2 or 3-element probability distributions. An application of these bounds for lossless source coding is provided, refining and improving a certain bound by Csiszár. A new inequality relating $f$-divergences is derived, and its use is exemplified. The last section of this conference paper is not included in the recent journal paper that was published in the February 2015 issue of the IEEE Trans. on Information Theory (see arXiv:1403.…
The binary divergences that are divergences between probability measures defined on the same 2-point set have an interesting property. For the chi-squared divergence and the relative entropy, it is known that their binary divergence attain lower bounds with given means and variances, respectively. In this note, we show that the binary divergence of the squared Hellinger distance has the same property and propose an open problem that what conditions are needed for f-divergence to satisfy this property.
Tight bounds for several symmetric divergence measures are derived in terms of the total variation distance. It is shown that each of these bounds is attained by a pair of 2 or 3-element probability distributions. An application of these bounds for lossless source coding is provided, refining and improving a certain bound by Csiszár. Another application of these bounds has been recently introduced by Yardi. et al. for channel-code detection.
Tighter Information-Theoretic Generalization Bounds via a Novel Class of Change of Measure Inequalities
Department of Electrical and Electronic Engineering, Imperial College London（帝国理工学院伦敦分校电子与电气工程系）；Department of Information Engineering, The Chinese University of Hong Kong（香港中文大学信息工程系）
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Foundation for Research and Technology, Hellas（希腊研究与技术基金会）
Variational representations of divergences and distances between high-dimensional probability distributions offer significant theoretical insights and practical advantages in numerous research areas. Recently, they have gained popularity in machine learning as a tractable and scalable approach for training probabilistic models and for statistically differentiating between data distributions. Their advantages include: 1) They can be estimated from data as statistical averages. 2) Such representations can leverage the ability of neural networks to efficiently approximate optimal solutions in fun…
I. Sason obtained the tight bounds for symmetric divergence measures are derived by applying the results established by G. L. Gilardoni. In this article, we are going to report two kinds of extensions for the above results, namely classical q-extension and non-commutative extension.
University College London（伦敦大学学院）；Massachusetts Institute of Technology（麻省理工学院）
Generalization error bounds are essential to understanding machine learning algorithms. This paper presents novel expected generalization error upper bounds based on the average joint distribution between the output hypothesis and each input training sample. Multiple generalization error upper bounds based on different information measures are provided, including Wasserstein distance, total variation distance, KL divergence, and Jensen-Shannon divergence. Due to the convexity of the information measures, the proposed bounds in terms of Wasserstein distance and total variation distance are show…
We study concentration inequalities for the Kullback--Leibler (KL) divergence between the empirical distribution and the true distribution. Applying a recursion technique, we improve over the method of types bound uniformly in all regimes of sample size $n$ and alphabet size $k$, and the improvement becomes more significant when $k$ is large. We discuss the applications of our results in obtaining tighter concentration inequalities for $L_1$ deviations of the empirical distribution from the true distribution, and the difference between concentration around the expectation or zero. We also obta…
In this paper we establish lower bounds on information divergence from a distribution to certain important classes of distributions as Gaussian, exponential, Gamma, Poisson, geometric, and binomial. These lower bounds are tight and for several convergence theorems where a rate of convergence can be computed, this rate is determined by the lower bounds proved in this paper. General techniques for getting lower bounds in terms of moments are developed.
This paper provides a numerical framework for computing the achievable rate region of memoryless multiple access channel (MAC) with a continuous alphabet from data. In particular, we use recent results on variational lower bounds on mutual information and KL-divergence to compute the boundaries of the rate region of MAC using a set of functions parameterized by neural networks. Our method relies on a variational lower bound on KL-divergence and an upper bound on KL-divergence based on the f-divergence inequalities. Unlike previous work, which computes an estimate on mutual information, which i…
Harvard University（哈佛大学）；Massachusetts Institute of Technology（麻省理工学院）
The families of $f$-divergences (e.g. the Kullback-Leibler divergence) and Integral Probability Metrics (e.g. total variation distance or maximum mean discrepancies) are widely used to quantify the similarity between probability distributions. In this work, we systematically study the relationship between these two families from the perspective of convex duality. Starting from a tight variational representation of the $f$-divergence, we derive a generalization of the moment-generating function, which we show exactly characterizes the best lower bound of the $f$-divergence as a function of a gi…
This paper proposes a new family of lower and upper bounds on the minimum mean squared error (MMSE). The key idea is to minimize/maximize the MMSE subject to the constraint that the joint distribution of the input-output statistics lies in a Kullback-Leibler divergence ball centered at some Gaussian reference distribution. Both bounds are tight and are attained by Gaussian distributions whose mean is identical to that of the reference distribution and whose covariance matrix is determined by a scalar parameter that can be obtained by finding the root of a monotonic function. The upper bound co…
The problem of estimating frequency moments of a data stream has attracted a lot of attention since the onset of streaming algorithms [AMS99]. While the space complexity for approximately computing the $p^{\rm th}$ moment, for $p\in(0,2]$ has been settled [KNW10], for $p>2$ the exact complexity remains open. For $p>2$ the current best algorithm uses $O(n^{1-2/p}\log n)$ words of space [AKO11,BO10], whereas the lower bound is of $Ω(n^{1-2/p})$ [BJKS04]. In this paper, we show a tight lower bound of $Ω(n^{1-2/p}\log n)$ words for the class of algorithms based on linear sketches, which store only…
Construction of multivariate tight framelets is known to be a challenging problem. Multivariate dual framelets with vanishing moments generalize tight framelets and are not easy to be constructed either. Compactly supported multivariate framelets with directionality or high vanishing moments are of interest and importance in both theory and applications. In this paper we introduce the notion of a quasi-tight framelet, which is a dual framelet, but behaves almost like a tight framelet. Let $ϕ\in L_2(R^d)$ be an arbitrary compactly supported $M$-refinable function such that its underlying low-pa…
We propose an approach to construct a family of two-dimensional compactly supported real-valued symmetric quincunx tight framelets $\{ϕ; ψ_1,ψ_2,ψ_3\}$ in $L_2(R^2)$ with arbitrarily high orders of vanishing moments. Such symmetric quincunx tight framelets are associated with quincunx tight framelet filter banks $\{a;b_1,b_2,b_3\}$ having increasing orders of vanishing moments and enjoying the additional double canonical properties: \[ b_1(k_1,k_2)=(-1)^{1+k_1+k_2} a(1-k_1,-k_2), b_3(k_1,k_2)=(-1)^{1+k_1+k_2} b_2(1-k_1,-k_2). \] For a low-pass filter $a$ which is not a quincunx orthonormal wav…
École Polytechnique（巴黎综合理工学院）；Sony Computer Science Laboratories（索尼计算机科学研究所）；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；University of Geneva（日内瓦大学）
We describe a framework to build distances by measuring the tightness of inequalities, and introduce the notion of proper statistical divergences and improper pseudo-divergences. We then consider the Hölder ordinary and reverse inequalities, and present two novel classes of Hölder divergences and pseudo-divergences that both encapsulate the special case of the Cauchy-Schwarz divergence. We report closed-form formulas for those statistical dissimilarities when considering distributions belonging to the same exponential family provided that the natural parameter space is a cone (e.g., multivaria…
This note outlines the steps for proving that the moments of a randomly-selected subset of a general ETF (complex, with aspect ratio $0<γ<1$) converge to the corresponding MANOVA moments. We bring here an extension for the proof of the 'Kesten-Mckay' moments (real ETF, $γ=1/2$) \cite{magsino2020kesten}. In particular, we establish a recursive computation of the $r$th moment, for $r = 1,2,\ldots$, and verify, using a symbolic program, that the recursion output coincides with the MANOVA moments.
We establish bounds on the KL divergence between two multivariate Gaussian distributions in terms of the Hamming distance between the edge sets of the corresponding graphical models. We show that the KL divergence is bounded below by a constant when the graphs differ by at least one edge; this is essentially the tightest possible bound, since classes of graphs exist for which the edge discrepancy increases but the KL divergence remains bounded above by a constant. As a natural corollary to our KL lower bound, we also establish a sample size requirement for correct model selection via maximum l…
This paper develops systematic approaches to obtain $f$-divergence inequalities, dealing with pairs of probability measures defined on arbitrary alphabets. Functional domination is one such approach, where special emphasis is placed on finding the best possible constant upper bounding a ratio of $f$-divergences. Another approach used for the derivation of bounds among $f$-divergences relies on moment inequalities and the logarithmic-convexity property, which results in tight bounds on the relative entropy and Bhattacharyya distance in terms of $χ^2$ divergences. A rich variety of bounds are sh…
Finite precision approximations of discrete probability distributions are considered, applicable for distribution synthesis, e.g., probabilistic shaping. Two algorithms are presented that find the optimal $M$-type approximation $Q$ of a distribution $P$ in terms of the variational distance $| Q-P|_1$ and the informational divergence $\mathbb{D}(Q| P)$. Bounds on the approximation errors are derived and shown to be asymptotically tight. Several examples illustrate that the variational distance optimal approximation can be quite different from the informational divergence optimal approximation.
Csiszar and Narayan[3] defined the notion of secret key capacity for multiple terminals, characterized it as a linear program with Slepian-Wolf constraints of the related source coding problem of communication for omniscience, and upper bounded it by some information divergence expression from the joint to the product distribution of the private observations. This paper proves that the bound is tight for the important case when all users are active, using the polymatroidal structure[6] underlying the source coding problem. When some users are not active, the bound may not be tight. This paper …
In this paper, tight upper and lower bounds are derived on the weighted sum of minimum mean-squared errors for additive Gaussian noise channels. The bounds are obtained by constraining the input distribution to be close to a Gaussian reference distribution in terms of the Kullback--Leibler divergence. The distributions that attain these bounds are shown to be Gaussian whose covariance matrices are defined implicitly via systems of matrix equations. Furthermore, the estimators that attain the upper bound are shown to be minimax robust against deviations from the assumed input distribution. The …
Universal relations that characterize the fluctuations of nonequilibrium systems are of fundamental importance. The thermodynamic and kinetic uncertainty relations impose upper bounds on the precision of currents solely by total entropy production and dynamical activity, respectively. Recently, a tighter bound that imposes on the precision of currents by both total entropy production and dynamical activity has been derived (referred to as the TKUR). In this paper, we show that the TKUR gives the tightest bound of a class of inequalities that imposes an upper bound on the precision of currents …
RIKEN Hakubi Research Team, RIKEN Pioneering Research Institute (PRI) and RIKEN Center for Quantum Computing (RQC)（RIKEN 博慧研究团队、RIKEN 先锋研究所（PRI）和 RIKEN 量子计算中心）；Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电气与计算机工程系）；Centre for Quantum Technologies, National University of Singapore（新加坡国立大学量子技术中心）
