arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.04823cs.CVcs.AIcs.ITcs.LGmath.IT
收藏

使用抗噪声扩散谱熵评估训练过程中的神经网络表示

Assessing Neural Network Representations During Training Using Noise-Resilient Diffusion Spectral Entropy

Yale University(耶鲁大学);Université de Montréal(蒙特利尔大学);Mila – Quebec AI Institute(米拉-魁北克人工智能研究所);Meta AI Research (FAIR)(Meta人工智能研究院(基础人工智能研究团队))

Danqi Liao, Chen Liu, Benjamin W. Christensen, Alexander Tong, Guillaume Huguet, Guy Wolf, Maximilian Nickel, Ian Adelstein, Smita Krishnaswamy

AI总结:

本文提出抗噪声的扩散谱熵 DSE 与扩散谱互信息 DSMI,用于刻画高维神经表示,并揭示训练、过拟合及泛化中的表示演化规律。

摘要:

神经网络中的熵和互信息能够提供关于学习过程的丰富信息,但事实证明,它们在高维空间中难以可靠计算。确实,在含噪且高维的数据中,环境维度下的传统估计会趋近于一个固定的熵值,并且计算成本高得令人难以承受。为解决这些问题,我们利用数据几何来访问底层流形,并可靠地计算这些信息论度量。具体而言,我们定义了数据集神经表示中的扩散谱熵(DSE),以及表示数据的不同变量之间的扩散谱互信息(DSMI)。首先,我们在高维模拟数据中表明,它们构成了针对内蕴维度和关系强度的抗噪声度量,并且优于经典 Shannon 熵、非参数估计以及互信息神经估计(MINE)。随后,我们研究了分类网络在监督学习、自监督或过拟合情况下表示的演化过程。我们观察到:(1)神经表示的 DSE 在训练期间增加;(2)在可泛化学习期间,与类别标签的 DSMI 增加,但在过拟合期间保持停滞;(3)与输入信号的 DSMI 表现出不同趋势:在 MNIST 上增加,而在 CIFAR-10 和 STL-10 上减少。最后,我们表明 DSE 可用于指导更好的网络初始化,并且 DSMI 可用于预测 ImageNet 上 962 个模型的下游分类准确率。官方实现可在 https://github.com/ChenLiu-1996/DiffusionSpectralEntropy 获取。

英文摘要:

Entropy and mutual information in neural networks provide rich information on the learning process, but they have proven difficult to compute reliably in high dimensions. Indeed, in noisy and high-dimensional data, traditional estimates in ambient dimensions approach a fixed entropy and are prohibitively hard to compute. To address these issues, we leverage data geometry to access the underlying manifold and reliably compute these information-theoretic measures. Specifically, we define diffusion spectral entropy (DSE) in neural representations of a dataset as well as diffusion spectral mutual information (DSMI) between different variables representing data. First, we show that they form noise-resistant measures of intrinsic dimensionality and relationship strength in high-dimensional simulated data that outperform classic Shannon entropy, nonparametric estimation, and mutual information neural estimation (MINE). We then study the evolution of representations in classification networks with supervised learning, self-supervision, or overfitting. We observe that (1) DSE of neural representations increases during training; (2) DSMI with the class label increases during generalizable learning but stays stagnant during overfitting; (3) DSMI with the input signal shows differing trends: on MNIST it increases, while on CIFAR-10 and STL-10 it decreases. Finally, we show that DSE can be used to guide better network initialization and that DSMI can be used to predict downstream classification accuracy across 962 models on ImageNet. The official implementation is available at https://github.com/ChenLiu-1996/DiffusionSpectralEntropy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑