粗粒化隐藏表示：通过映射熵进行无监督神经元选择
Coarse-Graining Hidden Representations: Unsupervised Neuron Selection via Mapping Entropy
- University of Trento（特伦托大学）
- INFN-TIFPA, Trento Institute for Fundamental Physics and Applications（INFN-TIFPA 特伦托基础物理与应用研究所）
- Radboud University（拉德堡德大学）
机构由 AI 辅助整理，请以论文原文为准。
中文总结 AI 辅助
该研究提出通过最小化映射熵（ME）的无监督准则选择过参数化神经网络的关键神经元，在师生网络、非线性高斯过程任务及翻译增强MNIST上，ME选择的子网络性能优于同等规模随机子集。
中文摘要 AI 辅助
过参数化神经网络所携带的隐藏单元数量远超任务名义上所需的数量，由此引出一个问题：哪些神经元是必不可少的，以及这种区分是否能在无需标签或梯度的情况下，仅从表示本身中清晰辨别。我们将神经元选择问题转化为通过保留隐藏层的一部分神经元来对其进行粗粒化的问题，并通过映射熵（Mapping Entropy，ME）对每个候选选择进行评分。该量度衡量了丢弃部分网络神经元所固有导致的判别力损失，使ME最小的选择被视为具有特别信息价值的选择。该准则完全是无监督的，因为它仅依赖于隐藏激活统计量。在师生网络中，ME优化可恢复与教师一致的最小表示，并按隐藏层的剩余变异性比例保留额外单元；在一项非线性高斯过程任务中，它选择了连贯的功能类映射，其偏好类别会在训练过程中发生转移。在该任务以及经翻译增强的MNIST上，ME选择的子网络表现优于同等规模的随机子集，在强压缩情况下最为明显——这将构型可区分性与预测性能关联了起来。
英文摘要
Overparameterized neural networks carry far more hidden units than a task nominally requires, raising the question of which neurons are essential and whether that distinction is legible in the representation itself, without labels or gradients. We cast neuron selection as the problem of coarse-graining the hidden layer by retaining a subset of its neurons, and score each putative selection by the mapping entropy (ME). This quantity measures the loss of discriminatory power inherent in discarding part of the network neurons, and the selection that minimises the ME is taken as particularly informative. This criterion is fully unsupervised, in that it depends only on hidden-activation statistics. In teacher-student networks, ME optimisation recovers the minimal teacher-consistent representation and retains extra units in proportion to the hidden layer's residual variability; in a non-linear Gaussian process task, it selects coherent functional-class mappings whose preferred class shifts across training. On this task and on translation-augmented MNIST, ME-selected subnetworks outperform random subsets of equal size, most clearly under strong compression - linking configurational distinguishability to predictive performance.
