理解基于互信息界的总相关估计的局限性
Understanding the limitation of Total Correlation Estimation Based on Mutual Information Bounds
AI总结：
该研究指出基于互信息界的总相关（TC）估计因重要性采样存在偏差放大问题，提出基于监督学习的校正模型，有效提升了TC估计精度并消除估计方差。
摘要：
总相关（TC）是衡量多维随机变量中各边缘分布间相关性的关键指标，常被用作表示学习中的归纳偏置。已有研究表明，可通过分解利用互信息（MI）界来估计TC值。但我们通过理论推导与定性实验发现，由于分解过程中使用了重要性采样，当采样中的提议分布与目标分布差异较大时，基于MI界估计的TC值的偏差会被放大。为缓解估计偏差问题，我们提出了一种基于监督学习的TC估计校正模型，该模型以基于MI界的TC估计器的训练迭代损失序列作为输入特征，输出真实TC值。实验表明，我们提出的方法能够提升TC估计的准确度，并消除TC估计过程中产生的方差。
英文摘要：
The total correlation(TC) is a crucial index to measure the correlation between marginal distribution in multidimensional random variables, and it is frequently applied as an inductive bias in representation learning. Previous research has shown that the TC value can be estimated using mutual information boundaries through decomposition. However, we found through theoretical derivation and qualitative experiments that due to the use of importance sampling in the decomposition process, the bias of TC value estimated based on MI bounds will be amplified when the proposal distribution in the sampling differs significantly from the target distribution. To reduce estimation bias issues, we propose a TC estimation correction model based on supervised learning, which uses the training iteration loss sequence of the TC estimator based on MI bounds as input features to output the true TC value. Experiments show that our proposed method can improve the accuracy of TC estimation and eliminate the variance generated by the TC estimation process.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
