基于电动力学的神经网络的收敛结果
Convergence Results for Neural Networks via Electrodynamics
摘要:
我们研究了一个深度为2的神经网络是否可以使用梯度下降来学习另一个深度为2的网络。假设一个线性输出节点,我们表明,梯度下降是否收敛到目标函数的问题是等价于以下问题电动力学:鉴于$k$固定质子在$\mathbb{R}^d,$和$k$电子,每个移动由于从质子的吸引力和排斥力从其余的电子,是否在平衡所有的电子将匹配的质子,排列。在标准电力下,这遵循经典的恩肖定理。在我们的设置中,力由激活函数和输入分布决定。在此等价性的基础上,我们证明了存在一个激活函数,使得梯度下降学习目标网络中的至少一个隐藏节点。通过迭代,我们证明了梯度下降可以用来一次一个节点地学习整个网络。
英文摘要:
We study whether a depth two neural network can learn another depth two network using gradient descent. Assuming a linear output node, we show that the question of whether gradient descent converges to the target function is equivalent to the following question in electrodynamics: Given $k$ fixed protons in $\mathbb{R}^d,$ and $k$ electrons, each moving due to the attractive force from the protons and repulsive force from the remaining electrons, whether at equilibrium all the electrons will be matched up with the protons, up to a permutation. Under the standard electrical force, this follows from the classic Earnshaw's theorem. In our setting, the force is determined by the activation function and the input distribution. Building on this equivalence, we prove the existence of an activation function such that gradient descent learns at least one of the hidden nodes in the target network. Iterating, we show that gradient descent can be used to learn the entire network one node at a time.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。