Lloyd Allison
This report concerns the information content of a graph, optionally conditional on one or more background, "common knowledge" graphs. It describes an algorithm to estimate this information content, and includes some examples based on chemical compounds.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
Characterizations of Conditional Mutual Independence: Equivalence and Implication
本文研究了条件互独立性的等价性和蕴含性，提出了一种标准形式用于描述其本质特征。
Nearest-Neighbours Estimators for Conditional Mutual Information
针对条件互信息估计数据需求大、受维数灾难限制的问题，本文介绍了Kozachenko-Leonenko框架下仅依赖点间距离、偏差可解析计算的最近邻估计器，并在模拟数据上完成测试。
We propose a new information-theoretic bound on generalization error based on a combination of the error decomposition technique of Bu et al. and the conditional mutual information (CMI) construction of Steinke and Zakynthinou. In a previous work, Haghifam et al. proposed a different bound combining the two aforementioned techniques, which we refer to as the conditional individual mutual information (CIMI) bound. However, in a simple Gaussian setting, both the CMI and the CIMI bounds are order-wise worse than that by Bu et al.. This observation motivated us to propose the new bound, which over…
The conditional mutual information I(X;Y|Z) measures the average information that X and Y contain about each other given Z. This is an important primitive in many learning problems including conditional independence testing, graphical model inference, causal strength estimation and time-series problems. In several applications, it is desirable to have a functional purely of the conditional distribution p_{Y|X,Z} rather than of the joint distribution p_{X,Y,Z}. We define the potential conditional mutual information as the conditional mutual information calculated with a modified joint distribut…
Several recent works in communication systems have proposed to leverage the power of neural networks in the design of encoders and decoders. In this approach, these blocks can be tailored to maximize the transmission rate based on aggregated samples from the channel. Motivated by the fact that, in many communication schemes, the achievable transmission rate is determined by a conditional mutual information term, this paper focuses on neural-based estimators for this information-theoretic quantity. Our results are based on variational bounds for the KL-divergence and, in contrast to some previo…
An Adaptive Neighborhood Partition Full Conditional Mutual Information Maximization Method for Feature Selection
针对特征选择中冗余特征信息利用不足及变量波动影响互信息计算的问题，提出全条件互信息最大化方法（FCMIM）与自适应邻域划分算法（ANP），在17个基准数据集上验证了其优于其他互信息方法的有效性。
This paper presents a novel feature selection method based on the conditional mutual information (CMI). The proposed High Order Conditional Mutual Information Maximization (HOCMIM) incorporates high order dependencies into the feature selection procedure and has a straightforward interpretation due to its bottom-up derivation. The HOCMIM is derived from the CMI's chain expansion and expressed as a maximization optimization problem. The maximization problem is solved using a greedy search procedure, which speeds up the entire feature selection process. The experiments are run on a set of benchm…
Quantum conditional mutual information and channel capacity
提出条件量子通信任务，证明其最优速率由量子条件互信息的一半给出，将经典密钥生成容量扩展到量子领域。
In this work, we analyse how to define a conditional version of Sibson's $α$-Mutual Information. Several such definitions can be advanced and they all lead to different information measures with different (but similar) operational meanings. We will analyse in detail one such definition, compute a closed-form expression for it and endorse it with an operational meaning while also considering some applications. The alternative definitions will also be mentioned and compared.
In the present paper, a simplified two-dimensional magnetic recording (TDMR) channel model is proposed in order to capture the qualitative features of writing and read-back processes of TDMR systems.The proposed channel model incorporates the effects of both linear interference from adjacent bit-cells and signal-dependent noise due to irregular grain boundaries between adjacent bit-cells. The simplicity of the proposed model enables us to derive the closed form of the conditional PDF representing the probabilistic nature of the channel. The conditional PDF is Gaussian distributed and is parame…
Testing (Conditional) Mutual Information
本文研究互信息与条件互信息检验的样本复杂度，给出区分条件独立与条件依赖所需样本数的上界并证明其在若干参数区域紧致，同时提出模拟弱相关样本的新方法和可处理相关样本的等价性检验估计器。
Due to the advantage of achieving a better performance under weak regularization, elastic net has attracted wide attention in statistics, machine learning, bioinformatics, and other fields. In particular, a variation of the elastic net, adaptive elastic net (AEN), integrates the adaptive grouping effect. In this paper, we aim to develop a new algorithm: Adaptive Elastic Net with Conditional Mutual Information (AEN-CMI) that further improves AEN by incorporating conditional mutual information into the gene selection process. We apply this new algorithm to screen significant genes for two kinds …
Estimation of information theoretic quantities such as mutual information and its conditional variant has drawn interest in recent times owing to their multifaceted applications. Newly proposed neural estimators for these quantities have overcome severe drawbacks of classical $k$NN-based estimators in high dimensions. In this work, we focus on conditional mutual information (CMI) estimation by utilizing its formulation as a minmax optimization problem. Such a formulation leads to a joint training procedure similar to that of generative adversarial networks. We find that our proposed estimator …
The analogues of Arimoto's definition of conditional Rényi entropy and Rényi mutual information are explored for abstract alphabets. These quantities, although dependent on the reference measure, have some useful properties similar to those known in the discrete setting. In addition to laying out some such basic properties and the relations to Rényi divergences, the relationships between the families of mutual informations defined by Sibson, Augustin-Csiszár, and Lapidoth-Pfister, as well as the corresponding capacities, are explored.
The information-theoretic framework of Russo and J. Zou (2016) and Xu and Raginsky (2017) provides bounds on the generalization error of a learning algorithm in terms of the mutual information between the algorithm's output and the training sample. In this work, we study the proposal, by Steinke and Zakynthinou (2020), to reason about the generalization error of a learning algorithm by introducing a super sample that contains the training sample as a random subset and computing mutual information conditional on the super sample. We first show that these new bounds based on the conditional mutu…
Conditional independence testing is a fundamental problem underlying causal discovery and a particularly challenging task in the presence of nonlinear and high-dimensional dependencies. Here a fully non-parametric test for continuous data based on conditional mutual information combined with a local permutation scheme is presented. Through a nearest neighbor approach, the test efficiently adapts also to non-smooth distributions due to strongly nonlinear dependencies. Numerical experiments demonstrate that the test reliably simulates the null distribution even for small sample sizes and with hi…
It is well known that the quantum mutual information and its conditional version do not increase under local channels. I this paper we show that the recently established lower semicontinuity of the quantum conditional mutual information implies (in fact, is equivalent to) the lower semicontinuity of the loss of the quantum (conditional) mutual information under local channels considered as a function on the Cartesian product of the set of all states of a composite system and the sets of all local channels (equipped with the strong convergence). Some applications of this property are considered…
In this paper we consider the problem of extracting secret key from an eavesdropped source $p_{XYZ}$ at a rate given by the conditional mutual information. We investigate this question under three different scenarios: (i) Alice ($X$) and Bob ($Y$) are unable to communicate but share common randomness with the eavesdropper Eve ($Z$), (ii) Alice and Bob are allowed one-way public communication, and (iii) Alice and Bob are allowed two-way public communication. Distributions having a key rate of the conditional mutual information are precisely those in which a "helping" Eve offers Alice and Bob no…
The estimation of mutual information (MI) or conditional mutual information (CMI) from a set of samples is a long-standing problem. A recent line of work in this area has leveraged the approximation power of artificial neural networks and has shown improvements over conventional methods. One important challenge in this new approach is the need to obtain, given the original dataset, a different set where the samples are distributed according to a specific product density function. This is particularly challenging when estimating CMI. In this paper, we introduce a new technique, based on k neare…
Differentiable Conditional Mutual Information for Multi-Terminal Linear Gaussian Wireless Networks
针对多终端高斯信道，提出基于线性高斯有向无环图的可微条件互信息闭式表达式，实现端到端梯度优化，适用于加权和速率、安全速率等目标。
Several Representations of $α$-Mutual Information and Interpretations as Privacy Leakage Measures
本文基于Rényi散度和条件Rényi熵，提出了$α$-互信息的新表示，并利用广义均值和增益函数解释其作为隐私泄露度量的意义，同时提出满足条件化降低熵的新型条件Rényi熵。
Estimating conditional mutual information (CMI) is an essential yet challenging step in many machine learning and data mining tasks. Estimating CMI from data that contains both discrete and continuous variables, or even discrete-continuous mixture variables, is a particularly hard problem. In this paper, we show that CMI for such mixture variables, defined based on the Radon-Nikodym derivate, can be written as a sum of entropies, just like CMI for purely discrete or continuous data. Further, we show that CMI can be consistently estimated for discrete-continuous mixture variables by learning an…
Conditional Mutual Information (CMI) is a measure of conditional dependence between random variables X and Y, given another random variable Z. It can be used to quantify conditional dependence among variables in many data-driven inference problems such as graphical models, causal learning, feature selection and time-series analysis. While k-nearest neighbor (kNN) based estimators as well as kernel-based methods have been widely used for CMI estimation, they suffer severely from the curse of dimensionality. In this paper, we leverage advances in classifiers and generative models to design metho…
We provide an information-theoretic framework for studying the generalization properties of machine learning algorithms. Our framework ties together existing approaches, including uniform convergence bounds and recent methods for adaptive data analysis. Specifically, we use Conditional Mutual Information (CMI) to quantify how well the input (i.e., the training data) can be recognized given the output (i.e., the trained model) of the learning algorithm. We show that bounds on CMI can be obtained from VC dimension, compression schemes, differential privacy, and other methods. We then show that b…
We give a novel, unified derivation of conditional PAC-Bayesian and mutual information (MI) generalization bounds. We derive conditional MI bounds as an instance, with special choice of prior, of conditional MAC-Bayesian (Mean Approximately Correct) bounds, itself derived from conditional PAC-Bayesian bounds, where `conditional' means that one can use priors conditioned on a joint training and ghost sample. This allows us to get nontrivial PAC-Bayes and MI-style bounds for general VC classes, something recently shown to be impossible with standard PAC-Bayesian/MI bounds. Second, it allows us t…
Mutual information is fundamentally important for measuring statistical dependence between variables and for quantifying information transfer by signaling and communication mechanisms. It can, however, be challenging to evaluate for physical models of such mechanisms and to estimate reliably from data. Furthermore, its relationship to better known statistical procedures is still poorly understood. Here we explore new connections between mutual information and regression-based dependence measures, $ν^{-1}$, that utilise the determinant of the second-moment matrix of the conditional mean predict…
The conditional quantum mutual information $I(A;B|C)$ of a tripartite state $ρ_{ABC}$ is an information quantity which lies at the center of many problems in quantum information theory. Three of its main properties are that it is non-negative for any tripartite state, that it decreases under local operations applied to systems $A$ and $B$, and that it obeys the duality relation $I(A;B|C)=I(A;B|D)$ for a four-party pure state on systems $ABCD$. The conditional mutual information also underlies the squashed entanglement, an entanglement measure that satisfies all of the axioms desired for an ent…
Estimating Conditional Mutual Information for Dynamic Feature Selection
本文面向动态特征选择，提出以判别式方法估计特征与响应变量的条件互信息，并扩展支持可变预算、非均匀成本、先验信息和部分输入，实验显示其性能稳定优于近期方法。
Information theory provides a fundamental framework for the quantification of information flows through channels, formally Markov kernels. However, quantities such as mutual information and conditional mutual information do not necessarily reflect the causal nature of such flows. We argue that this is often the result of conditioning based on sigma algebras that are not associated with the given channels. We propose a version of the (conditional) mutual information based on families of sigma algebras that are coupled with the underlying channel. This leads to filtrations which allow us to prov…
Explicit Entropic Constructions for Coverage, Facility Location, and Graph Cuts
本文通过显式熵构造证明了实践中常用的子模函数如覆盖、设施选址和图割目标可精确表示为香农熵，建立了组合信息度量与经典信息论的联系。
We characterize the growth of the Sibson and Arimoto mutual informations and $α$-maximal leakage, of any order that is at least unity, between a random variable and a growing set of noisy, conditionally independent and identically-distributed observations of the random variable. Each of these measures increases exponentially fast to a limit that is order- and measure-dependent, with an exponent that is order- and measure-independent.
We consider dynamic versions of the mutual information of lifetime distributions, with focus on past lifetimes, residual lifetimes and mixed lifetimes evaluated at different instants. This allows to study multicomponent systems, by measuring the dependence in conditional lifetimes of two components having possibly different ages. We provide some bounds, and investigate the mutual information of residual lifetimes within the time-transformed exponential model (under both the assumptions of unbounded and truncated lifetimes). Moreover, with reference to the order statistics of a random sample, w…
In this paper, we generalize the fundamental relation between the derivative of the mutual information and the minimum mean squared error (MMSE) to multiuser setups. We prove that the derivative of the mutual information with respect to the signal to noise ratio (SNR) is equal to the MMSE plus a covariance induced due to the interference, quantified by a term with respect to the cross correlation of the multiuser input estimates, the channels and the precoding matrices. We also derive new relations for the gradient of the conditional and non-conditional mutual information with respect to the M…
Reed-Muller (RM) codes and polar codes are generated by the same matrix $G_m= \bigl[\begin{smallmatrix}1 & 0 \\ 1 & 1 \\ \end{smallmatrix}\bigr]^{\otimes m}$ but using different subset of rows. RM codes select simply rows having largest weights. Polar codes select instead rows having the largest conditional mutual information proceeding top to down in $G_m$; while this is a more elaborate and channel-dependent rule, the top-to-down ordering has the advantage of making the conditional mutual information polarize, giving directly a capacity-achieving code on any binary memoryless symmetric chann…
Meta-learning optimizes an inductive bias---typically in the form of the hyperparameters of a base-learning algorithm---by observing data from a finite number of related tasks. This paper presents an information-theoretic bound on the generalization performance of any given meta-learner, which builds on the conditional mutual information (CMI) framework of Steinke and Zakynthinou (2020). In the proposed extension to meta-learning, the CMI bound involves a training \textit{meta-supersample} obtained by first sampling $2N$ independent tasks from the task environment, and then drawing $2M$ indepe…
We introduce the Conditional Mutual Information (CMI) for the estimation of the Markov chain order. For a Markov chain of $K$ symbols, we define CMI of order $m$, $I_c(m)$, as the mutual information of two variables in the chain being $m$ time steps apart, conditioning on the intermediate variables of the chain. We find approximate analytic significance limits based on the estimation bias of CMI and develop a randomization significance test of $I_c(m)$, where the randomized symbol sequences are formed by random permutation of the components of the original symbol sequence. The significance tes…
On information content in certain objects
本文研究自然对象的普通条件复杂度与全条件复杂度之间的差距，首次证明复杂度小于 n 的字符串数量与字典序第一个高复杂度字符串之间的相互全条件复杂度可能很大。
Two-Parameter Rényi Information Quantities with Applications to Privacy Amplification and Soft Covering
本文提出双参数 Rényi 条件熵与互信息，证明其基本性质并统一已有变体，进而刻画了隐私放大与软覆盖问题中的强逆指数。
We propose a new estimator to measure directed dependencies in time series. The dimensionality of data is first reduced using a new non-uniform embedding technique, where the variables are ranked according to a weighted sum of the amount of new information and improvement of the prediction accuracy provided by the variables. Then, using a greedy approach, the most informative subsets are selected in an iterative way. The algorithm terminates, when the highest ranked variable is not able to significantly improve the accuracy of the prediction as compared to that obtained using the existing sele…
Neural Estimation of Pairwise Mutual Information in Masked Discrete Sequence Models
本文提出了一种神经框架，直接从预训练的遮蔽扩散模型（MDMs）的隐藏状态中估计成对条件互信息（MI），利用模型自身条件分布计算的地面真实MI进行监督，从而捕捉模型内部对依赖结构的信念，并在单次前向传递中预测完整的MI矩阵，实现MI引导的并行解码。
A variety of new measures of quantum Renyi mutual information and quantum Renyi conditional entropy have recently been proposed, and some of their mathematical properties explored. Here, we show that the Renyi mutual information attains operational meaning in the context of composite hypothesis testing, when the null hypothesis is a fixed bipartite state and the alternate hypothesis consists of all product states that share one marginal with the null hypothesis. This hypothesis testing problem occurs naturally in channel coding, where it corresponds to testing whether a state is the output of …
We provide a mutual information lower bound that can be used to analyze the effect of training in models with unknown parameters. For large-scale systems, we show that this bound can be calculated using the difference between two derivatives of a conditional entropy function. The bound does not require explicit estimation of the unknown parameters. We provide a step-by-step process for computing the bound, and provide an example application. A comparison with known classical mutual information bounds is provided.
We revisit the problem of asymmetric binary hypothesis testing against a composite alternative hypothesis. We introduce a general framework to treat such problems when the alternative hypothesis adheres to certain axioms. In this case we find the threshold rate, the optimal error and strong converse exponents (at large deviations from the threshold) and the second order asymptotics (at small deviations from the threshold). We apply our results to find operational interpretations of various Renyi information measures. In case the alternative hypothesis is comprised of bipartite product distribu…
Single-shot lossy compression: mutual information bounds
本文提出通过互信息上界来优化数据压缩，证明在保真度约束下最小化互信息可作为数据最小描述长度的合理代理。
Besides the different approaches suggested in the literature, accurate estimation of the order of a Markov chain from a given symbol sequence is an open issue, especially when the order is moderately large. Here, parametric significance tests of conditional mutual information (CMI) of increasing order $m$, $I_c(m)$, on a symbol sequence are conducted for increasing orders $m$ in order to estimate the true order $L$ of the underlying Markov chain. CMI of order $m$ is the mutual information of two variables in the Markov chain being $m$ time steps apart, conditioning on the intermediate variable…
The entropy of a pair of random variables is commonly depicted using a Venn diagram. This representation is potentially misleading, however, since the multivariate mutual information can be negative. This paper presents new measures of multivariate information content that can be accurately depicted using Venn diagrams for any number of random variables. These measures complement the existing measures of multivariate mutual information and are constructed by considering the algebraic structure of information sharing. It is shown that the distinct ways in which a set of marginal observers can s…
Conditional Fisher-Information Central Limit Theorems under Log-Concavity with Information-Theoretic Consequences
本文证明对数凹性下条件Fisher信息中心极限定理，并由此推出条件熵收敛及互信息斜率趋于高斯基准等推论。
We present a method to generate speech from input text and a style vector that is extracted from a reference speech signal in an unsupervised manner, i.e., no style annotation, such as speaker information, is required. Existing unsupervised methods, during training, generate speech by computing style from the corresponding ground truth sample and use a decoder to combine the style vector with the input text. Training the model in such a way leaks content information into the style vector. The decoder can use the leaked content and ignore some of the input text to minimize the reconstruction lo…
Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information
针对知识蒸馏中传统最大对数似然方法估计贝叶斯条件分布精度不足的问题，提出最大条件互信息估计方法训练教师模型，提升学生分类准确率，在少样本、零样本场景下增益更显著。
Generalization in Federated Learning: A Conditional Mutual Information Framework
本文提出基于条件互信息的超客户端框架，为联邦学习两级泛化误差推导多个信息论界，并给出快速速率界及实验验证。
Normalized Conditional Mutual Information Surrogate Loss for Deep Neural Classifiers
本文提出归一化条件互信息作为替代交叉熵损失，通过实验表明其在图像识别和全滑片成像任务中显著提升模型性能。
In this paper, we derive new closed-form expressions for the gradient of the mutual information with respect to arbitrary parameters of the two-user multiple access channel (MAC). The derived relations generalize the fundamental relation between the derivative of the mutual information and the minimum mean squared error (MMSE) to multiuser setups. We prove that the derivative of the mutual information with respect to the signal to noise ratio (SNR) is equal to the MMSE plus a covariance induced due to the interference, quantified by a term with respect to the cross correlation of the multiuser…
We consider a model nondispersive nonlinear optical fiber channel with additive white Gaussian noise at large $\mathrm{SNR}$ (signal-to-noise ratio) in the intermediate power region. Using Feynman path-integral technique we for the first time find the optimal input signal distribution maximizing the channel's per-sample mutual information. The finding of the optimal input signal distribution allows us to improve previously known estimates for the channel capacity. The output signal entropy, conditional entropy, and per-sample mutual information are calculated for Gaussian, half-Gaussian and mo…
We start with Fannes' type and Winter's type tight continuity bounds for the quantum conditional mutual information and their specifications for states of special types. Then we analyse continuity of the Holevo quantity with respect to nonequivalent metrics on the set of discrete ensembles of quantum states. We show that the Holevo quantity is continuous on the set of all ensembles of m states with respect to all the metrics if either m or the dimension of underlying Hilbert space is finite and obtain Fannes' type tight continuity bounds for the Holevo quantity in this case. In general case co…
We characterize mutual information as the unique map on ordered pairs of random variables satisfying a set of axioms similar to those of Faddeev's characterization of the Shannon entropy. There is a new axiom in our characterization however which has no analogue for Shannon entropy, based on the notion of a Markov triangle, which may be thought of as a composition of communication channels for which conditional entropy acts functorially. Our proofs are coordinate-free in the sense that no logarithms appear in our calculations.
Conditional contraction coefficients and their applications to quantum networks
本研究引入条件压缩系数框架，将经典相对熵与互信息压缩的对应关系扩展至量子边信息场景，为量子网络信息压缩分析提供统一框架，并拓展了相关边界与极限。
Quantum information measures such as the entropy and the mutual information find applications in physics, e.g., as correlation measures. Generalizing such measures based on the Rényi entropies is expected to enhance their scope in applications. We prescribe Rényi generalizations for any quantum information measure which consists of a linear combination of von Neumann entropies with coefficients chosen from the set {-1,0,1}. As examples, we describe Rényi generalizations of the conditional quantum mutual information, some quantum multipartite information measures, and the topological entangleme…
We derive a tight lower bound on equivocation (conditional entropy), or equivalently a tight upper bound on mutual information between a signal variable and channel outputs. The bound is in terms of the joint distribution of the signals and maximum a posteriori decodes (most probable signals given channel output). As part of our derivation, we describe the key properties of the distribution of signals, channel outputs and decodes, that minimizes equivocation and maximizes mutual information. This work addresses a problem in data analysis, where mutual information between signals and decodes is…
We describe a compression-based distance for genomic sequences. Instead of using the usual conjoint information content, as in the classical Normalized Compression Distance (NCD), it uses the conditional information content. To compute this Normalized Conditional Compression Distance (NCCD), we need a normal conditional compressor, that we built using a mixture of static and dynamic finite-context models. Using this approach, we measured chromosomal distances between Hominidae primates and also between Muroidea (rat and mouse), observing several insights of evolution that so far have not been …
Information estimates such as the ``direct method'' of Strong et al. (1998) sidestep the difficult problem of estimating the joint distribution of response and stimulus by instead estimating the difference between the marginal and conditional entropies of the response. While this is an effective estimation strategy, it tempts the practitioner to ignore the role of the stimulus and the meaning of mutual information. We show here that, as the number of trials increases indefinitely, the direct (or ``plug-in'') estimate of marginal entropy converges (with probability 1) to the entropy of the time…
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。