RNN学习过程中随机性和结构之间的相互作用
The interplay between randomness and structure during learning in RNNs
摘要：
在低维任务上训练的递归神经网络（RNN）已被广泛用于功能生物网络的建模。然而，通过学习找到的解决方案和初始连接的效果并没有得到很好的理解。在这里，我们研究了在神经科学文献的启发下，使用梯度下降对不同任务进行训练的RNN。我们发现，经常性的连接的变化可以描述低秩矩阵，尽管学习算法的无约束性质。为了确定低秩结构的起源，我们转向一个易于分析的设置：在简化的任务上训练线性RNN。我们展示了低维任务结构如何导致连接的低秩变化。这种低秩结构使我们能够解释和量化在随机初始连接存在下的加速学习现象。总而言之，我们的研究从学习过程和由此产生的网络结构两个方面为理解经过训练的RNN开辟了一个新的视角。
英文摘要：
Recurrent neural networks (RNNs) trained on low-dimensional tasks have been widely used to model functional biological networks. However, the solutions found by learning and the effect of initial connectivity are not well understood. Here, we examine RNNs trained using gradient descent on different tasks inspired by the neuroscience literature. We find that the changes in recurrent connectivity can be described by low-rank matrices, despite the unconstrained nature of the learning algorithm. To identify the origin of the low-rank structure, we turn to an analytically tractable setting: training a linear RNN on a simplified task. We show how the low-dimensional task structure leads to low-rank changes to connectivity. This low-rank structure allows us to explain and quantify the phenomenon of accelerated learning in the presence of random initial connectivity. Altogether, our study opens a new perspective to understanding trained RNNs in terms of both the learning process and the resulting network structure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
