arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2202.00956cs.ITcs.CRmath.IT
收藏

互信息替代测量方法的研究

Investigation of Alternative Measures for Mutual Information

Bulut Kuskonmaz, Jaron Skovsted Gundersen, Rafal Wisniewski

摘要:

互信息$I(X;Y)$是信息论中的一个有用的定义,用于估计随机变量$Y$关于随机变量$X$持有多少信息。定义互信息的一种方式是通过比较$X$和$Y$的联合分布与通过KL-发散的边缘的乘积。如果两个分布彼此接近,则几乎没有$X$从$Y$泄漏,因为两个变量接近于独立。在离散设置中,互信息具有$Y$揭示关于$X$的多少比特的良好解释,并且如果$I(X;Y)=H(X)$($X$的香农熵),则$X$被完全揭示。然而,在连续的情况下,我们没有同样的推理。例如,在连续情况下,互信息可以是无限的。这一事实使我们能够尝试不同的度量或分歧来定义互信息。在本文中,我们正在评估不同的度量或分歧,如Kullback-Liebler(KL)分歧,Wasserstein距离,Jensen-Shannon分歧和总变异距离,以形成替代的互信息在连续的情况下。我们部署不同的方法来估计或约束这些指标和分歧,并评估其性能。

英文摘要:

Mutual information $I(X;Y)$ is a useful definition in information theory to estimate how much information the random variable $Y$ holds about the random variable $X$. One way to define the mutual information is by comparing the joint distribution of $X$ and $Y$ with the product of the marginals through the KL-divergence. If the two distributions are close to each other there will be almost no leakage of $X$ from $Y$ since the two variables are close to being independent. In the discrete setting the mutual information has the nice interpretation of how many bits $Y$ reveals about $X$ and if $I(X;Y)=H(X)$ (the Shannon entropy of $X$) then $X$ is completely revealed. However, in the continuous case we do not have the same reasoning. For instance the mutual information can be infinite in the continuous case. This fact enables us to try different metrics or divergences to define the mutual information. In this paper, we are evaluating different metrics or divergences such as Kullback-Liebler (KL) divergence, Wasserstein distance, Jensen-Shannon divergence and total variation distance to form alternatives to the mutual information in the continuous case. We deploy different methods to estimate or bound these metrics and divergences and evaluate their performances.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑