训练反馈网络全局稳定性的几何分析
A geometrical analysis of global stability in trained feedback networks
摘要:
递归神经网络由于其实现复杂计算的能力而在神经科学和机器学习的背景下被广泛研究。虽然在过去的几年里,在设计有效的学习算法方面取得了实质性的进展,但仍然缺乏对训练的递归网络的充分理解。具体来说,允许计算从底层经常性动态中出现的机制在很大程度上是未知的。在这里,我们专注于一个简单的,但未充分探索的计算设置:一个反馈架构训练相关联的静态输出到一个静态输入。作为一个出发点,我们推导出一个近似的分析描述的全球动态训练网络,假设不相关的连接权重的反馈和随机散装。由此产生的平均场理论表明,该任务承认几类解决方案,这意味着不同的稳定性。不同类别的特征在于相对于输入向量的读出的几何布置,定义在由网络人口跨越的高维空间。我们发现这种近似理论方法可以用来理解标准训练技术如何在有限大小的反馈网络中实现输入-输出任务。特别是,我们的简化描述捕获了目标解的局部和全局稳定性,从而预测了训练性能。
英文摘要:
Recurrent neural networks have been extensively studied in the context of neuroscience and machine learning due to their ability to implement complex computations. While substantial progress in designing effective learning algorithms has been achieved in the last years, a full understanding of trained recurrent networks is still lacking. Specifically, the mechanisms that allow computations to emerge from the underlying recurrent dynamics are largely unknown. Here we focus on a simple, yet underexplored computational setup: a feedback architecture trained to associate a stationary output to a stationary input. As a starting point, we derive an approximate analytical description of global dynamics in trained networks which assumes uncorrelated connectivity weights in the feedback and in the random bulk. The resulting mean-field theory suggests that the task admits several classes of solutions, which imply different stability properties. Different classes are characterized in terms of the geometrical arrangement of the readout with respect to the input vectors, defined in the high-dimensional space spanned by the network population. We find that such approximate theoretical approach can be used to understand how standard training techniques implement the input-output task in finite-size feedback networks. In particular, our simplified description captures the local and the global stability properties of the target solution, and thus predicts training performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。