We tighten the Entropy Power Inequality (EPI) when one of the random summands is Gaussian. Our strengthening is closely connected to the concept of strong data processing for Gaussian channels and generalizes the (vector extension of) Costa's EPI. This leads to a new reverse entropy power inequality and, as a corollary, sharpens Stam's inequality relating entropy power and Fisher information. Applications to network information theory are given, including a short self-contained proof of the rate region for the two-encoder quadratic Gaussian source coding problem.
Our argument is based on weak convergence and a technique employed by Geng and Nair for establishing Gaussian optimality via rotational-invariance, which traces its roots to a `doubling trick' that has been successfully used in the study of functional inequalities.
The matrix version of the entropy-power inequality for real or complex coefficients and variables is proved using a transportation argument that easily settles the equality case. An application to blind source extraction is given.
We establish a general class of entropy inequalities that take the concise form of Gaussian comparisons. The main result unifies many classical and recent results, including the Shannon-Stam inequality, the Brunn-Minkowski inequality, the Zamir-Feder inequality, the Brascamp-Lieb and Barthe inequalities, the Anantharam-Jog-Nair inequality, and others.
Costa's entropy power inequality is an important generalization of Shannon's entropy power inequality. Related with Costa's entropy power inequality and a conjecture proposed by McKean in 1966, Cheng-Geng recently conjectured that $D(m,n): (-1)^{m+1}(\partial^m/\partial^m t)H(X_t)\ge0$, where $X_t$ is the $n$-dimensional random variable in Costa's entropy power inequality and $H(X_t)$ the differential entropy of $X_t$. $D(1,n)$ and $D(2,n)$ were proved by Costa as consequences of Costa's entropy power inequality. Cheng-Geng proved $D(3,1)$ and $D(4,1)$. In this paper, we propose a systematical…
Let $\mathsf{N}_{\rm d}\left[X\right]=\frac{1}{2π{\rm e}}{\rm e}^{2\mathsf{H}\left[X\right]}$ denote the entropy power of the discrete random variable $X$ where $\mathsf{H}\left[X\right]$ denotes the discrete entropy of $X$. In this paper, we show that for two independent discrete random variables $X$ and $Y$, the entropy power inequality $\mathsf{N}_{\rm d}\left[X\right]+\mathsf{N}_{\rm d}\left[Y\right]\leq 2 \mathsf{N}_{\rm d}\left[X+Y\right]$ holds and it can be tight. The basic idea behind the proof is to perturb the discrete random variables using suitably designed continuous random varia…
The Discounted Least Information Theory of Entropy (DLITE) is a new information measure that quantifies the amount of entropic difference between two probability distributions. It manifests multiple critical properties both as an information-theoretic quantity and as metric distance. In the report, we provide a proof of the triangular inequality of DLITE's cube root ($\sqrt[3]{DL}$), an important property of a metric, along with alternative proofs for two additional properties.
Using a sharp version of the reverse Young inequality, and a Rényi entropy comparison result due to Fradelizi, Madiman, and Wang, the authors are able to derive Rényi entropy power inequalities for log-concave random vectors when Rényi parameters belong to $(0,1)$. Furthermore, the estimates are shown to be sharp up to absolute constants.
In this paper, we investigate the partition inequality, joint convexity, and Pinsker's inequality, for a divergence that generalizes the Tsallis Relative Entropy and Kullback-Leibler divergence. The generalized divergence is defined in terms of a deformed exponential function, which replaces the Tsallis $q$-exponential. We also constructed a family of probability distributions related to the generalized divergence. We found necessary and sufficient conditions for the partition inequality to be satisfied. A sufficient condition for the joint convexity was established. We proved that the general…
This paper concerns the folklore statement that ``entropy is a lower bound for compression''. More precisely we derive from the entropy theorem a simple proof of a pointwise inequality firstly stated by Ornstein and Shields and which is the almost-sure version of an average inequality firstly stated by Khinchin in 1953. We further give an elementary proof of original Khinchin inequality that can be used as an exercise for Information Theory students and we conclude by giving historical and technical notes of such inequality.
A Massey-like inequality is any useful lower bound on guessing entropy in terms of the computationally scalable Shannon entropy. The asymptotically optimal Massey-like inequality is determined and further refined for finite-support distributions. The impact of these results are highlighted for side-channel attack evaluation where guessing entropy is a key metric. In this context, the obtained bounds are compared to the state of the art.
The entropy power inequality (EPI) and the Brascamp-Lieb inequality (BLI) are fundamental inequalities concerning the differential entropies of linear transformations of random vectors. The EPI provides lower bounds for the differential entropy of linear transformations of random vectors with independent components. The BLI, on the other hand, provides upper bounds on the differential entropy of a random vector in terms of the differential entropies of some of its linear transformations. In this paper, we define a family of entropy functionals, which we show are subadditive. We then establish …
We provide a simple proof of a curious inequality for the binary entropy function, an inequality that has been used in two different contexts. In the 1980's, Boppana used this entropy inequality to prove lower bounds on Boolean formulas. More recently, the inequality was used to achieve major progress on Frankl's union-closed sets conjecture. Our proof of the entropy inequality uses basic differential calculus.
A noisy entropy inequality for boolean functions by Samorodnitsky is applied to binary codes. It is shown that a binary code that achieves capacity on the binary erasure channel admits optimal list size for list decoding on some binary symmetric channels (in a regime where this optimal list size is exponentially large).
Motivated by the entropy computations relevant to the evaluation of decrease in entropy in bit reset operations, the authors investigate the deficit in an entropic inequality involving two independent random variables, one continuous and the other discrete. In the case where the continuous random variable is Gaussian, we derive strong quantitative bounds on the deficit in the inequality. More explicitly it is shown that the decay of the deficit is sub-Gaussian with respect to the reciprocal of the standard deviation of the Gaussian variable. What is more, up to rational terms these results are…
Aim of this short note is to study Shannon's entropy power along entropic interpolations, thus generalizing Costa's concavity theorem. We shall provide two proofs of independent interest: the former by $Γ$-calculus, hence applicable to more abstract frameworks; the latter with an explicit remainder term, reminiscent of [20], allowing us to characterize the case of equality.
We prove a quantitative form of the celebrated Ball's theorem on cube slicing in $\mathbb{R}^n$ and obtain, as a consequence, equality cases in the min-entropy power inequality. Independently, we also give a quantitative form of Khintchine's inequality in the special case $p=1$.
The conditional entropy power inequality is a fundamental inequality in information theory, stating that the conditional entropy of the sum of two conditionally independent vector-valued random variables each with an assigned conditional entropy is minimum when the random variables are Gaussian. We prove the conditional entropy power inequality in the scenario where the conditioning system is quantum. The proof is based on the heat semigroup and on a generalization of the Stam inequality in the presence of quantum conditioning. The entropy power inequality with quantum conditioning will be a k…
In this paper, we proved an exact asymptotically sharp upper bound of the $L^p$ Lebesgue Constant (i.e. the $L^p$ norm of Dirichlet kernel) for $p\ge 2$. As an application, we also verified the implication of a new $\infty $-Rényi entropy power inequality for integer valued random variables.
We prove the quantum conditional Entropy Power Inequality for quantum additive noise channels. This inequality lower bounds the quantum conditional entropy of the output of an additive noise channel in terms of the quantum conditional entropies of the input state and the noise when they are conditionally independent given the memory. We also show that this conditional Entropy Power Inequality is optimal in the sense that we can achieve equality asymptotically by choosing a suitable sequence of Gaussian input states. We apply the conditional Entropy Power Inequality to find an array of informat…
On the analogue of the concavity of entropy power in the Brunn-Minkowski theory
Costa和Cover在{ \it On the similarity of the entropy power inequality and Brunn-Minkowski inequality}（IEEE Trans. Inform. Theory 30（1984），no. 6，837-839）可测集的外部平行体积的$\frac{1}{n}$ -熵作为熵功率的熵的模拟。我们研究了这个猜想，并研究了它与几何不等式的关系。
We establish a lower bound on the entropy of weighted sums of (possibly dependent) random variables $(X_1, X_2, \dots, X_n)$ possessing a symmetric joint distribution. Our lower bound is in terms of the joint entropy of $(X_1, X_2, \dots, X_n)$. We show that for $n \geq 3$, the lower bound is tight if and only if $X_i$'s are i.i.d.\ Gaussian random variables. For $n=2$ there are numerous other cases of equality apart from i.i.d.\ Gaussians, which we completely characterize. Going beyond sums, we also present an inequality for certain linear transformations of $(X_1, \dots, X_n)$. Our primary t…
We prove the conditional Entropy Power Inequality for Gaussian quantum systems. This fundamental inequality determines the minimum quantum conditional von Neumann entropy of the output of the beam-splitter or of the squeezing among all the input states where the two inputs are conditionally independent given the memory and have given quantum conditional entropies. We also prove that, for any couple of values of the quantum conditional entropies of the two inputs, the minimum of the quantum conditional entropy of the output given by the conditional Entropy Power Inequality is asymptotically ach…
The relative $α$-entropy is the Rényi analog of relative entropy and arises prominently in information-theoretic problems. Recent information geometric investigations on this quantity have enabled the generalization of the Cramér-Rao inequality, which provides a lower bound for the variance of an estimator of an escort of the underlying parametric probability distribution. However, this framework remains unexamined in the Bayesian framework. In this paper, we propose a general Riemannian metric based on relative $α$-entropy to obtain a generalized Bayesian Cramér-Rao inequality. This establish…
We study quantum computation relations on unital finite-dimensional CAR $C^{*}$-algebras. We prove an entropy power inequality (EPI) in a fermionic setting, which presumably will permit understanding the capacities in fermionic linear optics. Similar relations to the bosonic case are shown, and alternative proofs of known facts are given. Clifford algebras and the Grassmann representation can thus be used to obtain mathematical results regarding coherent fermion states.
We prove stability estimates for the Shannon-Stam inequality (also known as the entropy-power inequality) for log-concave random vectors in terms of entropy and transportation distance. In particular, we give the first stability estimate for general log-concave random vectors in the following form: for log-concave random vectors $X,Y \in \mathbb{R}^d$, the deficit in the Shannon-Stam inequality is bounded from below by the expression $$ C \left(\mathrm{D}\left(X||G\right) + \mathrm{D}\left(Y||G\right)\right), $$ where $\mathrm{D}\left( \cdot ~ ||G\right)$ denotes the relative entropy with resp…
