In this work we present a new method for the estimation of Mutual Information (MI) between random variables. Our approach is based on an original interpretation of the Girsanov theorem, which allows us to use score-based diffusion models to estimate the Kullback Leibler divergence between two densities as a difference between their score functions. As a by-product, our method also enables the estimation of the entropy of random variables. Armed with such building blocks, we present a general recipe to measure MI, which unfolds in two directions: one uses conditional diffusion process, whereas the other uses joint diffusion processes that allow simultaneous modelling of two random variables. Our results, which derive from a thorough experimental protocol over all the variants of our approach, indicate that our method is more accurate than the main alternatives from the literature, especially for challenging distributions. Furthermore, our methods pass MI self-consistency tests, including data processing and additivity under independence, which instead are a pain-point of existing methods.
Diffusion Models and Semi-Supervised Learners Benefit Mutually with Few Labels
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing Key Laboratory of Big Data Management and Analysis Methods（大数据管理与分析方法北京市重点实验室）；Dept. of Comp. Sci. & Tech., BNRist Center, THU-Bosch ML Center, Tsinghua University（清华大学计算机科学与技术系，北京国家信息科学技术研究中心，清华-博世机器学习联合研究中心）
CDGraph: Dual Conditional Social Graph Synthesizing via Diffusion Model
Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所）；Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）；Department of Electrical Engineering, National Taiwan University（国立台湾大学电机工程学系）；Department of Computer Science, TU Delft（代尔夫特理工大学计算机科学系）
Allen Discovery Center at Tufts University（塔夫茨大学阿伦发现中心）；Institute for Theoretical Physics, TU Wien（理论物理研究所，维也纳技术大学）；Wyss Institute for Biologically Inspired Engineering at Harvard University（哈佛大学生物启发工程研究所）
InfoBridge: Mutual Information estimation via Bridge Matching
Applied AI Institute（应用人工智能研究所）；Institute of Numerical Mathematics (RAS)（俄罗斯科学院数值数学研究所）；Moscow Independent Research Institute of Artificial Intelligence（莫斯科独立人工智能研究所）
