arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1809.06848cs.LGcs.AIstat.ML
收藏

深度神经网络的学习动力学

On the Learning Dynamics of Deep Neural Networks

Microsoft Research Montreal(微软研究院蒙特利尔);Universite de Montreal, MILA(蒙特利尔大学,MILA)

Remi Tachet, Mohammad Pezeshki, Samira Shabanian, Aaron Courville, Yoshua Bengio

摘要:

虽然近年来已经取得了很多进展,但深度非线性神经网络中的学习动力学直到今天仍然在很大程度上被误解。在这项工作中,我们研究了二进制分类的情况下,并证明了在这样的网络学习的各种性质的强假设下,如线性可分性的数据。扩展现有的结果从线性的情况下,我们证实了经验观察证明,分类误差也遵循一个S形的非线性架构。我们表明,适当的初始化,学习阐述了平行的独立模式,参数空间的某些区域可能会导致训练失败。我们还证明了数据集中的输入范数和特征频率导致了不同的收敛速度,这可能会对深度神经网络的泛化能力有所启发。我们提供了交叉熵和铰链损失的学习动态之间的比较,这可能有助于理解生成对抗网络训练的最新进展。最后,我们确定了一种现象,即我们对梯度饥饿进行了洗礼,其中数据集中最频繁的特征阻止了其他不太频繁但同样信息丰富的特征的学习。

英文摘要:

While a lot of progress has been made in recent years, the dynamics of learning in deep nonlinear neural networks remain to this day largely misunderstood. In this work, we study the case of binary classification and prove various properties of learning in such networks under strong assumptions such as linear separability of the data. Extending existing results from the linear case, we confirm empirical observations by proving that the classification error also follows a sigmoidal shape in nonlinear architectures. We show that given proper initialization, learning expounds parallel independent modes and that certain regions of parameter space might lead to failed training. We also demonstrate that input norm and features' frequency in the dataset lead to distinct convergence speeds which might shed some light on the generalization capabilities of deep neural networks. We provide a comparison between the dynamics of learning with cross-entropy and hinge losses, which could prove useful to understand recent progress in the training of generative adversarial networks. Finally, we identify a phenomenon that we baptize gradient starvation where the most frequent features in a dataset prevent the learning of other less frequent but equally informative features.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑