We consider the "partial information decomposition" (PID) problem, which aims to decompose the information that a set of source random variables provide about a target random variable into separate redundant, synergistic, union, and unique components. In the first part of this paper, we propose a general framework for constructing a multivariate PID. Our framework is defined in terms of a formal analogy with intersection and union from set theory, along with an ordering relation which specifies when one information source is more informative than another. Our definitions are algebraically and axiomatically motivated, and can be generalized to domains beyond Shannon information theory (such as algorithmic information theory and quantum information theory). In the second part of this paper, we use our general framework to define a PID in terms of the well-known Blackwell order, which has a fundamental operational interpretation. We demonstrate our approach on numerous examples and show that it overcomes many drawbacks associated with previous proposals.
The Partial Entropy Decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal
在多元系统中获得有意义的统计依赖性的定量描述是一个困难的开放性问题。部分信息分解(Partial Information Decomposition,PID)是将目标变量的互信息(Mutual Information,MI)分解为预测变量的不同子集中冗余、唯一和协同的分量。在这里,我们建议将PID的优雅形式主义应用于多变量熵,从而产生部分熵分解(PED)。我们实现了PED的熵冗余度测量的基础上逐点常见的代数,一个自然的定义,这是密切相关的MI的定义。我们展示了这种方法可以揭示二元与三元生成结构的多变量系统是无法区分的经典香农措施。熵的观点还表明,错误信息是协同熵,因此,MI本身包括冗余和协同效应。我们在两个预测PED和MI之间的关系,并推导出两种替代的信息分解,我们说明了几个示例系统。这表明,在熵方面,单变量预测MI是不是一个适当的子集的联合MI,我们建议这个以前未认识到的事实部分解释了为什么获得一致的PID已被证明是困难的。PED还允许单独量化机械冗余(与系统功能相关)与源冗余(由输入之间的依赖关系引起);这是一个现有方法无法解决的重要区别。PED提供的新视角有助于澄清PID方法遇到的一些困难,由此产生的分解为广泛的应用领域的实际数据分析提供了有用的工具。
The mathematical landscape of partial information decomposition: A comprehensive review of properties and measures
Imperial College London(帝国理工学院);University of Cambridge(剑桥大学);Queen Mary University(伦敦玛丽女王大学);University College London(伦敦大学学院);Aix Marseille Université(艾克斯-马赛大学)
Bits and Pieces: Understanding Information Decomposition from Part-whole Relationships and Formal Logic
Campus Institute for Dynamics of Biological Networks, Georg-August University, Goettingen(哥廷根乔治·奥古斯特大学生物网络动力学校园研究所);MEG Unit, Brain Imaging Center, Goethe University, Frankfurt(法兰克福歌德大学脑成像中心脑磁图单元)