循环神经网络的门控理论
Theory of gating in recurrent neural networks
摘要：
循环神经网络（RNN）是一种强大的动态模型，广泛应用于机器学习（ML）和神经科学。之前的理论工作主要集中在具有加性相互作用的RNN上。然而，门控-即乘法-相互作用在真实神经元中无处不在，也是ML中性能最好的RNN的核心特征。在这里，我们表明，门控提供了灵活的控制集体动力学的两个显着特征：i）时间尺度和ii）维度。门控制时间尺度导致一个新的，边缘稳定的状态，其中网络作为一个灵活的积分器。与以前的方法不同，门控允许这个重要的功能，而无需参数微调或特殊的对称性。门还提供了一种灵活的、依赖于上下文的机制来重置记忆轨迹，从而补充了记忆功能。调制维度的门可以诱导一种新的、不连续的混沌过渡，其中输入将稳定系统推向强混沌活动，与输入的典型稳定效果相反。在这种过渡中，与加性RNN不同，临界点的扩散（拓扑复杂性）与混沌动力学的出现（动力学复杂性）解耦。丰富的动力学在相图中进行了总结，从而为ML从业者提供了一个原则性参数初始化选择的地图。
英文摘要：
Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) and neuroscience. Prior theoretical work has focused on RNNs with additive interactions. However, gating - i.e. multiplicative - interactions are ubiquitous in real neurons and also the central feature of the best-performing RNNs in ML. Here, we show that gating offers flexible control of two salient features of the collective dynamics: i) timescales and ii) dimensionality. The gate controlling timescales leads to a novel, marginally stable state, where the network functions as a flexible integrator. Unlike previous approaches, gating permits this important function without parameter fine-tuning or special symmetries. Gates also provide a flexible, context-dependent mechanism to reset the memory trace, thus complementing the memory function. The gate modulating the dimensionality can induce a novel, discontinuous chaotic transition, where inputs push a stable system to strong chaotic activity, in contrast to the typically stabilizing effect of inputs. At this transition, unlike additive RNNs, the proliferation of critical points (topological complexity) is decoupled from the appearance of chaotic dynamics (dynamical complexity). The rich dynamics are summarized in phase diagrams, thus providing a map for principled parameter initialization choices to ML practitioners.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
