回声状态网络中传递函数非线性的研究
Exploring Transfer Function Nonlinearity in Echo State Networks
摘要：
超线性和次线性突触前和树突整合被认为是生物神经元的非线性计算能力的原因，强调非线性整合的作用，而不是非线性输出阈值。传递函数的非线性如何、为什么以及在多大程度上有助于生物启发的神经网络模型还没有完全理解。在这里，我们研究这些问题的回声状态网络（ESN）的上下文中。ESN是一种简单的神经网络架构，其中固定的递归网络由输入信号驱动，并且输出由读出层从网络状态的测量生成。ESN结构在某些信号处理任务上具有高效的训练和良好的性能，例如系统辨识和时间序列预测。ESN的性能进行了分析，在网络结构和输入偏置的连接模式。然而，在网络中的传递函数的影响还没有得到系统的研究。在这里，我们使用的方法tanh的泰勒展开的一个常用的传递函数，双曲正切函数，系统地研究增加非线性的传递函数的记忆，非线性容量和信号处理性能的ESN的效果。有趣的是，我们发现二次近似足以捕获ESN与tanh函数的计算能力。本研究的结果适用于ESN的软件和硬件实现。
英文摘要：
Supralinear and sublinear pre-synaptic and dendritic integration is considered to be responsible for nonlinear computation power of biological neurons, emphasizing the role of nonlinear integration as opposed to nonlinear output thresholding. How, why, and to what degree the transfer function nonlinearity helps biologically inspired neural network models is not fully understood. Here, we study these questions in the context of echo state networks (ESN). ESN is a simple neural network architecture in which a fixed recurrent network is driven with an input signal, and the output is generated by a readout layer from the measurements of the network states. ESN architecture enjoys efficient training and good performance on certain signal-processing tasks, such as system identification and time series prediction. ESN performance has been analyzed with respect to the connectivity pattern in the network structure and the input bias. However, the effects of the transfer function in the network have not been studied systematically. Here, we use an approach tanh on the Taylor expansion of a frequently used transfer function, the hyperbolic tangent function, to systematically study the effect of increasing nonlinearity of the transfer function on the memory, nonlinear capacity, and signal processing performance of ESN. Interestingly, we find that a quadratic approximation is enough to capture the computational power of ESN with tanh function. The results of this study apply to both software and hardware implementation of ESN.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
