arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.02649stat.MLcs.LGq-bio.NC
收藏

广义门控线性网络上梯度下降的隐偏差

The Implicit Bias of Gradient Descent on Generalized Gated Linear Networks

Columbia University(哥伦比亚大学);Flatiron Institute(弗莱托恩研究所)

Samuel Lippl, L. F. Abbott, SueYeon Chung

摘要:

了解深度神经网络梯度下降训练的渐近行为对于揭示归纳偏差和提高网络性能至关重要。我们推导了一类数学上易于处理的深度非线性神经网络-门控线性网络(GLN)的无限时间训练极限,并将这些结果推广到由一般齐次多项式描述的门控网络。我们研究我们的结果的影响,首先集中在两层GLN。然后,我们将我们的理论预测应用于在MNIST上训练的GLN,并展示了架构约束和梯度下降的隐式偏差如何影响性能。最后,我们证明了我们的理论捕获了ReLU网络的大部分归纳偏差。通过明确归纳偏见,我们的框架准备通知更有效的,生物学上合理的,和强大的学习算法的发展。

英文摘要:

Understanding the asymptotic behavior of gradient-descent training of deep neural networks is essential for revealing inductive biases and improving network performance. We derive the infinite-time training limit of a mathematically tractable class of deep nonlinear neural networks, gated linear networks (GLNs), and generalize these results to gated networks described by general homogeneous polynomials. We study the implications of our results, focusing first on two-layer GLNs. We then apply our theoretical predictions to GLNs trained on MNIST and show how architectural constraints and the implicit bias of gradient descent affect performance. Finally, we show that our theory captures a substantial portion of the inductive bias of ReLU networks. By making the inductive bias explicit, our framework is poised to inform the development of more efficient, biologically plausible, and robust learning algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑