arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.15958cs.LGmath.DS
收藏

关于三层神经网络动力学的研究:初始凝聚现象

On the dynamics of three-layer neural networks: initial condensation

Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)

Zheng-An Chen, Tao Luo

AI总结:

本文研究三层神经网络训练中的初始凝聚现象,通过严格理论分析建立有效动力学爆破性质并给出凝聚发生的充分条件,同时揭示其与深度矩阵分解低秩偏置的关联。

摘要:

实证与理论研究表明,两层神经网络的输入权重在以较小值初始化时,会收敛到孤立的取向。这一被称为凝聚的现象表明,梯度下降方法在训练过程中倾向于自发降低神经网络的复杂度。在本工作中,我们阐明三层神经网络训练中出现的凝聚现象背后的机制,并将其与两层神经网络的训练区分开来。通过严格的理论分析,我们建立了有效动力学的爆破性质,并给出了凝聚发生的充分条件,这些发现得到了实验结果的证实。此外,我们探讨了凝聚与深度矩阵分解中观察到的低秩偏置之间的关联。

英文摘要:

Empirical and theoretical works show that the input weights of two-layer neural networks, when initialized with small values, converge towards isolated orientations. This phenomenon, referred to as condensation, indicates that the gradient descent methods tend to spontaneously reduce the complexity of neural networks during the training process. In this work, we elucidate the mechanisms behind the condensation phenomena occurring in the training of three-layer neural networks and distinguish it from the training of two-layer neural networks. Through rigorous theoretical analysis, we establish the blow-up property of effective dynamics and present a sufficient condition for the occurrence of condensation, findings that are substantiated by experimental results. Additionally, we explore the association between condensation and the low-rank bias observed in deep matrix factorization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑