arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.14064cs.LG
收藏

用过参数化递归神经网络学习低维状态空间

Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets

Tel Aviv University(特拉维夫大学);Google Research(谷歌研究院)

Edo Cohen-Karlik, Itamar Menuhin-Gruman, Raja Giryes, Nadav Cohen, Amir Globerson

摘要:

深度学习中的过度参数化通常指的是这样一种设置,即经过训练的神经网络(NN)具有以多种方式拟合训练数据的代表性能力,其中一些泛化能力很好,而另一些则不然。在递归神经网络(RNN)的情况下,存在额外的过度参数化层,从某种意义上说,模型可能会表现出许多解决方案,这些解决方案可以很好地概括训练中看到的序列长度,其中一些可以外推到更长的序列,而另一些则不能。大量的工作研究了梯度下降(GD)的趋势,以适应过参数化的神经网络的解决方案,推广良好。另一方面,它倾向于用外推的解决方案来拟合过参数化的RNN,这一点直到最近才被发现,而且还远没有被理解。在本文中,我们分析了GD的外推性能时,应用到过参数化的线性RNN。与最近的论点表明对短期记忆的内隐偏见相反,我们提供了学习低维状态空间的理论证据,它也可以模拟长期记忆。我们的结果依赖于一个动态的表征,这表明GD(小步长和接近零的初始化)努力保持一定形式的平衡,以及在统计的时刻问题的背景下开发的工具(恢复概率分布从其时刻)。实验证实了我们的理论,通过学习线性和非线性RNN的低维状态空间来证明外推。

英文摘要:

Overparameterization in deep learning typically refers to settings where a trained neural network (NN) has representational capacity to fit the training data in many ways, some of which generalize well, while others do not. In the case of Recurrent Neural Networks (RNNs), there exists an additional layer of overparameterization, in the sense that a model may exhibit many solutions that generalize well for sequence lengths seen in training, some of which extrapolate to longer sequences, while others do not. Numerous works have studied the tendency of Gradient Descent (GD) to fit overparameterized NNs with solutions that generalize well. On the other hand, its tendency to fit overparameterized RNNs with solutions that extrapolate has been discovered only recently and is far less understood. In this paper, we analyze the extrapolation properties of GD when applied to overparameterized linear RNNs. In contrast to recent arguments suggesting an implicit bias towards short-term memory, we provide theoretical evidence for learning low-dimensional state spaces, which can also model long-term memory. Our result relies on a dynamical characterization which shows that GD (with small step size and near-zero initialization) strives to maintain a certain form of balancedness, as well as on tools developed in the context of the moment problem from statistics (recovery of a probability distribution from its moments). Experiments corroborate our theory, demonstrating extrapolation via learning low-dimensional state spaces with both linear and non-linear RNNs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑