arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.08545cs.ITcs.LGmath.ITmath.OCstat.ML
收藏

核方法的信息论

Information Theory with Kernel Methods

Inria(法国国家信息与自动化研究所);Ecole Normale Supérieure(巴黎高等师范学院);PSL Research University(巴黎文理研究大学)

Francis Bach

摘要:

我们考虑通过再生核希尔伯特空间的协方差算子分析概率分布。我们发现,这些运营商的冯诺依曼熵和相对熵是密切相关的香农熵和相对熵的通常概念,并分享他们的许多属性。它们与来自各种预言机的关于概率分布的有效估计算法结合在一起。我们还考虑产品空间,并表明,张量积核,我们可以定义的概念,互信息和联合熵,然后可以完全独立的特点,但只有部分条件独立。最后,我们将展示这些新的相对熵的概念如何导致新的上界的日志分区函数,可以一起使用凸优化变分推理方法,提供了一个新的家庭的概率推理方法。

英文摘要:

We consider the analysis of probability distributions through their associated covariance operators from reproducing kernel Hilbert spaces. We show that the von Neumann entropy and relative entropy of these operators are intimately related to the usual notions of Shannon entropy and relative entropy, and share many of their properties. They come together with efficient estimation algorithms from various oracles on the probability distributions. We also consider product spaces and show that for tensor product kernels, we can define notions of mutual information and joint entropies, which can then characterize independence perfectly, but only partially conditional independence. We finally show how these new notions of relative entropy lead to new upper-bounds on log partition functions, that can be used together with convex optimization within variational inference methods, providing a new family of probabilistic inference methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑