arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.02545q-bio.NCnlin.CDphysics.bio-ph
收藏

信息处理循环神经网络中的动态相位和共振现象

Dynamical Phases and Resonance Phenomena in Information-Processing Recurrent Neural Networks

Claus Metzner, Patrick Krauss

摘要:

循环神经网络(RNN)是一种复杂的动态系统,能够在没有任何驱动输入的情况下持续活动。自由运行的RNN的长期行为,由周期性,混沌和固定点吸引子描述,由神经连接权重的统计数据控制,例如非零连接的密度$d$,或兴奋性和抑制性连接之间的平衡$b$。然而,出于信息处理的目的,RNN需要接收外部输入信号,并且不清楚哪个动态机制对于这种信息输入是最佳的。我们使用的平均相关性$C$和瞬时输入向量和下一个系统状态向量之间的互信息$I$作为信息输入的定量措施,并分析其依赖于网络的平衡和密度。值得注意的是,这两个得到的相图$C(b,d)$和$I(b,d)$是高度一致的,指出了动力系统和复杂系统的信息处理方法之间的联系。信息进口是最大的不是在“混沌边缘”,这是最适合于计算,但令人惊讶的是,在低密度的混沌政权和混沌和固定点政权之间的边界。此外,我们发现了一种全新类型的共振现象,称为“输入共振”(IR),其中信息输入显示出最大值,即对RNN及其输入之间的耦合强度的峰值依赖性。IR补充了递归共振(RR),其中连续系统状态的相关性和互信息对于添加到系统的一定幅度的噪声达到峰值。IR和RR都可以用来优化人工神经网络中的信息处理,也可能在生物神经系统中发挥关键作用。

英文摘要:

Recurrent neural networks (RNNs) are complex dynamical systems, capable of ongoing activity without any driving input. The long-term behavior of free-running RNNs, described by periodic, chaotic and fixed point attractors, is controlled by the statistics of the neural connection weights, such as the density $d$ of non-zero connections, or the balance $b$ between excitatory and inhibitory connections. However, for information processing purposes, RNNs need to receive external input signals, and it is not clear which of the dynamical regimes is optimal for this information import. We use both the average correlations $C$ and the mutual information $I$ between the momentary input vector and the next system state vector as quantitative measures of information import and analyze their dependence on the balance and density of the network. Remarkably, both resulting phase diagrams $C(b,d)$ and $I(b,d)$ are highly consistent, pointing to a link between the dynamical systems and the information-processing approach to complex systems. Information import is maximal not at the 'edge of chaos', which is optimally suited for computation, but surprisingly in the low-density chaotic regime and at the border between the chaotic and fixed point regime. Moreover, we find a completely new type of resonance phenomenon, called 'Import Resonance' (IR), where the information import shows a maximum, i.e. a peak-like dependence on the coupling strength between the RNN and its input. IR complements Recurrence Resonance (RR), where correlation and mutual information of successive system states peak for a certain amplitude of noise added to the system. Both IR and RR can be exploited to optimize information processing in artificial neural networks and might also play a crucial role in biological neural systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑