The use of the logarithm of the variate in the calculation of differential entropy among certain related statistical distributions

Random variate generation using only finitely many unbiased, independently and identically distributed random bits

For any discrete probability distributions with bounded entropy, we can generate exactly a random variate using only a finite expected number of perfect coin flips. A perfect coin flip is the outcome of an unbiased Bernoulli random variable. Coin flips are unbiased, independently and identically distributed in all our work. We survey well-known algorithms for the discrete case such as the one from Knuth and Yao as well as the one from Han and Hoshi. We also discuss briefly about a practical implementation for the algorithm proposed by Knuth and Yao. For the continuous case, only approximations…

Links between the Logarithmic Sobolev Inequality and the convolution inequalities for Entropy and Fisher Information

Relative to the Gaussian measure on $\mathbb{R}^d$, entropy and Fisher information are famously related via Gross' logarithmic Sobolev inequality (LSI). These same functionals also separately satisfy convolution inequalities, as proved by Stam. We establish a dimension-free inequality that interpolates among these relations. Several interesting corollaries follow: (i) the deficit in the LSI satisfies a convolution inequality itself; (ii) the deficit in the LSI controls convergence in the entropic and Fisher information central limit theorems; and (iii) the LSI is stable with respect to HWI jum…

熵的对数分解与带符号测度空间

A Logarithmic Decomposition and a Signed Measure Space for Entropy

本文提出熵的对数分解，在结果空间上构造带符号测度空间，以对数原子刻画信息量，并应用于Gács-Körner公共信息及二元与三元系统的定性区分。

Deformed logarithms. Associated entropic divergences. Applications to linear inverse problems. Inversion algorithms

This work belongs to the framework of inverse problems with linear model. The resolution of this type of problem consists in minimizing (possibly under constraints) a function of discrepancy between the measurements and a physical model of the considered phenomenon. We will here extend this study to divergence functions based on forms of entropy developed in the field of Statistical Physics. These various forms of entropy are based on "the deformed algebra" [9] and in particularly on the notion of "Deformed Logarithm" [15][9].

A short characterization of relative entropy

We prove characterization theorems for relative entropy (also known as Kullback-Leibler divergence), q-logarithmic entropy (also known as Tsallis entropy), and q-logarithmic relative entropy. All three have been characterized axiomatically before, but we show that earlier proofs can be simplified considerably, at the same time relaxing some of the hypotheses.

Sharp Concentration Inequalities for the Centered Relative Entropy

We study the relative entropy between the empirical estimate of a discrete distribution and the true underlying distribution. If the minimum value of the probability mass function exceeds an $α> 0$ (i.e. when the true underlying distribution is bounded sufficiently away from the boundary of the simplex), we prove an upper bound on the moment generating function of the centered relative entropy that matches (up to logarithmic factors in the alphabet size and $α$) the optimal asymptotic rates, subsequently leading to a sharp concentration inequality for the centered relative entropy. As a coroll…