软委员会机器超可实现在线学习动力学中的软模式
Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machines
摘要：
通过随机梯度下降训练的过参数化深度神经网络成功地执行了许多具有实际意义的任务。过度参数化的一个方面是学生网络可能比数据生成过程具有更大的表现力。在学生-教师场景的上下文中，这对应于所谓的过度可实现的情况，其中学生网络具有比教师更多的隐藏单元。对于在线学习的两层软委员会机器在过可实现的情况下，我们发现，完美的学习方法发生在幂律的时尚，而不是指数，在可实现的情况下。如果教师和学生输出被适当地重新缩放，则所有学生节点学习并复制其中一个教师节点。
英文摘要：
Over-parametrized deep neural networks trained by stochastic gradient descent are successful in performing many tasks of practical relevance. One aspect of over-parametrization is the possibility that the student network has a larger expressivity than the data generating process. In the context of a student-teacher scenario, this corresponds to the so-called over-realizable case, where the student network has a larger number of hidden units than the teacher. For on-line learning of a two-layer soft committee machine in the over-realizable case, we find that the approach to perfect learning occurs in a power-law fashion rather than exponentially as in the realizable case. All student nodes learn and replicate one of the teacher nodes if teacher and student outputs are suitably rescaled.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
