arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.04712cs.ITcs.LGmath.IT
收藏

高维平滑熵估计:基于降维方法

High-Dimensional Smoothed Entropy Estimation via Dimensionality Reduction

IBM Research(IBM研究院);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);UIUC(伊利诺伊大学厄巴纳-香槟分校)

Kristjan Greenewald, Brian Kingsbury, Yuancheng Yu

AI总结:

本文提出基于主成分分析降维的高维平滑熵估计方法,克服了指数级样本复杂度,适用于低维嵌入高维结构,并验证了在DNN互信息估计中的有效性。

摘要:

我们研究了在高斯卷积下克服微分熵估计中指数级样本复杂度的问题。具体而言,我们考虑通过 $X$ 的 $n$ 个独立同分布样本来估计微分熵 $h(X+Z)$,其中 $X$ 和 $Z$ 是独立的 $D$ 维随机变量,$X$ 为具有有界二阶矩的次高斯分布,且 $Z\sim\mathcal{N}(0,\sigma^2I_D)$。在绝对误差损失下,上述问题的参数估计率为 $\frac{c^D}{\sqrt{n}}$,该速率在数据维度 $D$ 上呈指数增长,在实际应用中往往带来问题。我们通过在主成分分析(PCA)降维后再进行熵估计,克服了这一指数级样本复杂度,并证明了当 PCA 的未解释方差趋近于零时,渐近误差开销也随之消失。这意味着对于嵌入在高维空间中的固有低维结构,我们的方法可实现近乎最优的性能,这些结构包括深度神经网络(DNN)的隐藏层输出,可用于估计 DNN 中的互信息(MI)。我们提供了数值结果,验证了我们的 PCA 方法在高斯数据和螺旋数据上的表现。我们还将该方法应用于分析神经网络层间的信息流(参见信息瓶颈),测量了用于 MNIST 分类的含噪全连接网络和含噪卷积神经网络(CNN)中的互信息。

英文摘要:

We study the problem of overcoming exponential sample complexity in differential entropy estimation under Gaussian convolutions. Specifically, we consider the estimation of the differential entropy $h(X+Z)$ via $n$ independently and identically distributed samples of $X$, where $X$ and $Z$ are independent $D$-dimensional random variables with $X$ sub-Gaussian with bounded second moment and $Z\sim\mathcal{N}(0,σ^2I_D)$. Under the absolute-error loss, the above problem has a parametric estimation rate of $\frac{c^D}{\sqrt{n}}$, which is exponential in data dimension $D$ and often problematic for applications. We overcome this exponential sample complexity by projecting $X$ to a low-dimensional space via principal component analysis (PCA) before the entropy estimation, and show that the asymptotic error overhead vanishes as the unexplained variance of the PCA vanishes. This implies near-optimal performance for inherently low-dimensional structures embedded in high-dimensional spaces, including hidden-layer outputs of deep neural networks (DNN), which can be used to estimate mutual information (MI) in DNNs. We provide numerical results verifying the performance of our PCA approach on Gaussian and spiral data. We also apply our method to analysis of information flow through neural network layers (c.f. information bottleneck), with results measuring mutual information in a noisy fully connected network and a noisy convolutional neural network (CNN) for MNIST classification.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑