arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1905.12784cs.LGstat.ML
收藏

深度神经网络中数据表示的内在维数

Intrinsic dimension of data representations in deep neural networks

International School for Advanced Studies(国际高等研究学院);Technical University of Munich(慕尼黑工业大学)

Alessio Ansuini, Alessandro Laio, Jakob H. Macke, Davide Zoccolan

摘要:

深度神经网络在多个处理层上逐步转换其输入。这些网络学习到的表示的几何特性是什么?在这里,我们研究数据表示的内在维度(ID),即描述表示所需的最小参数数量。我们发现,在一个经过训练的网络中,ID比每层中的单元数量小几个数量级。在层之间,ID首先增加,然后在最后的层中逐渐减小。值得注意的是,最后一个隐藏层的ID预测了测试集的分类精度。这些结果也不能通过线性维度估计(例如,用主成分分析),也不是在已经被人工线性化的表示中。它们既不存在于未经训练的网络中,也不存在于在随机标签上训练的网络中。这表明,能够泛化的神经网络是那些将数据转换为低维但不一定是平坦流形的网络。

英文摘要:

Deep neural networks progressively transform their inputs across multiple processing layers. What are the geometrical properties of the representations learned by these networks? Here we study the intrinsic dimensionality (ID) of data-representations, i.e. the minimal number of parameters needed to describe a representation. We find that, in a trained network, the ID is orders of magnitude smaller than the number of units in each layer. Across layers, the ID first increases and then progressively decreases in the final layers. Remarkably, the ID of the last hidden layer predicts classification accuracy on the test set. These results can neither be found by linear dimensionality estimates (e.g., with principal component analysis), nor in representations that had been artificially linearized. They are neither found in untrained networks, nor in networks that are trained on randomized labels. This suggests that neural networks that can generalize are those that transform the data into low-dimensional, but not necessarily flat manifolds.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑