深度学生机的空间异构学习
Spatially heterogeneous learning by a deep student machine
AI总结：
通过教师-学生设定的统计力学方法研究深度神经网络的监督学习，发现在稠密极限下DNN学习具有空间异构性，且过参数化不削弱泛化能力，减小宽度或数据有效维度可提升泛化能力。
摘要：
具有大量可调参数的深度神经网络（DNN）在很大程度上仍然是黑盒。为了阐明DNN的隐藏层，我们通过一种称为教师-学生设定的统计力学方法，研究了由宽度为$N$、深度为$L$、包含$NL$个具有$c$个输入的感知机组成的DNN的监督学习。我们考虑了一个学生机集合，这些学生机精确再现了由教师机提供的$M$组$N$维输入/输出关系。我们表明，在我们所谓的“稠密极限”下，该问题变得精确可解：$N \gg c \gg 1$且$M \gg 1$，固定$α=M/c$，使用(H. Yoshino, (2020))中开发的副本方法。我们还通过执行简单的贪婪MC模拟对模型进行了数值研究。模拟表明，DNN的学习在网络空间中是相当异构的：教师机和学生机的配置在更靠近输入/输出边界的层内相关性更高，而中心区域由于过参数化，相关性要低得多，这与理论预测在定性上是一致的。我们在理论和数值上评估了不同深度$L$的DNN的泛化误差。值得注意的是，理论和模拟都表明，仅在中心与教师机弱相关的学生机的泛化能力，即使在系统变得严重过参数化的深度极限$L \gg 1$下也不会消失。我们还通过将隐藏流形模型(S. Goldt et. al., (2020))纳入我们的模型，考虑了数据有效维度$D(\leq N)$的影响。理论表明，减小宽度$N$或减小数据的有效维度$D$都会增强对稠密极限的环路修正。模拟表明这两者都会导致泛化能力的显著提升。
英文摘要：
Deep neural networks (DNN) with a huge number of adjustable parameters remain largely black boxes. To shed light on the hidden layers of DNN, we study supervised learning by a DNN of width $N$ and depth $L$ consisting of $NL$ perceptrons with $c$ inputs by a statistical mechanics approach called the teacher-student setting. We consider an ensemble of student machines that exactly reproduce $M$ sets of $N$ dimensional input/output relations provided by a teacher machine. We show that the problem becomes exactly solvable in what we call as 'dense limit': $N \gg c \gg 1$ and $M \gg 1$ with fixed $α=M/c$ using the replica method developed in (H. Yoshino, (2020)). We also study the model numerically performing simple greedy MC simulations. Simulations reveal that learning by the DNN is quite heterogeneous in the network space: configurations of the teacher and the student machines are more correlated within the layers closer to the input/output boundaries while the central region remains much less correlated due to the over-parametrization in qualitative agreement with the theoretical prediction. We evaluate the generalization-error of the DNN with various depth $L$ both theoretically and numerically. Remarkably both the theory and simulation suggest generalization-ability of the student machines, which are only weakly correlated with the teacher in the center, does not vanish even in the deep limit $L \gg 1$ where the system becomes heavily over-parametrized. We also consider the impact of effective dimension $D(\leq N)$ of data by incorporating the hidden manifold model (S. Goldt et. al., (2020)) into our model. The theory implies that the loop corrections to the dense limit become enhanced by either decreasing the width $N$ or decreasing the effective dimension $D$ of the data. Simulation suggests both lead to significant improvements in generalization-ability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
