基于尺度Cayley变换的正交循环神经网络
Orthogonal Recurrent Neural Networks with Scaled Cayley Transform
摘要：
循环神经网络（RNN）被设计用于处理连续数据，但会受到梯度消失或爆炸的影响。最近关于酉循环神经网络（uRNN）的工作已经被用来解决这个问题，在某些情况下，超过了长短期记忆网络（LSTM）的能力。我们提出了一个更简单和新颖的更新计划，以保持正交递归权重矩阵，而不使用复值矩阵。这是通过使用Cayley变换用反对称矩阵进行参数化来完成的。这样的参数化不能表示具有负一特征值的矩阵，但是通过由一和负一组成的对角矩阵缩放递归权重矩阵来克服这种限制。所提出的训练方案涉及直接的梯度计算和更新步骤。在几个实验中，所提出的缩放凯莱正交递归神经网络（scoRNN）比其他酉RNN具有更少的可训练参数，取得了更好的结果。
英文摘要：
Recurrent Neural Networks (RNNs) are designed to handle sequential data but suffer from vanishing or exploding gradients. Recent work on Unitary Recurrent Neural Networks (uRNNs) have been used to address this issue and in some cases, exceed the capabilities of Long Short-Term Memory networks (LSTMs). We propose a simpler and novel update scheme to maintain orthogonal recurrent weight matrices without using complex valued matrices. This is done by parametrizing with a skew-symmetric matrix using the Cayley transform. Such a parametrization is unable to represent matrices with negative one eigenvalues, but this limitation is overcome by scaling the recurrent weight matrix by a diagonal matrix consisting of ones and negative ones. The proposed training scheme involves a straightforward gradient calculation and update step. In several experiments, the proposed scaled Cayley orthogonal recurrent neural network (scoRNN) achieves superior results with fewer trainable parameters than other unitary RNNs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
