实时的递归学习的最优Kronecker和逼近
Optimal Kronecker-Sum Approximation of Real Time Recurrent Learning
摘要：
递归神经网络（RNN）的核心目标之一是学习序列数据中的长期依赖关系。然而，最流行的训练方法，通过时间截断反向传播（TBPTT），明确禁止学习超出截断范围的依赖关系。相比之下，在线训练算法实时的递归学习（RTRL）提供了未截断的梯度，具有不切实际的大计算成本的缺点。最近公布的方法通过提供RTRL的噪声近似来降低这些成本。提出了一种新的RTRL的近似算法--最优Kronecker和近似（OK）。我们证明了OK是最佳的一类近似RTRL，其中包括所有的方法发表至今。此外，我们表明，OK具有经验上可忽略的噪声：与以前的算法不同，它在现实世界的任务（字符级Penn TreeBank）中匹配TBPTT，并且可以利用在线参数更新在合成字符串记忆任务中优于TBPTT。代码在github上可用。
英文摘要：
One of the central goals of Recurrent Neural Networks (RNNs) is to learn long-term dependencies in sequential data. Nevertheless, the most popular training method, Truncated Backpropagation through Time (TBPTT), categorically forbids learning dependencies beyond the truncation horizon. In contrast, the online training algorithm Real Time Recurrent Learning (RTRL) provides untruncated gradients, with the disadvantage of impractically large computational costs. Recently published approaches reduce these costs by providing noisy approximations of RTRL. We present a new approximation algorithm of RTRL, Optimal Kronecker-Sum Approximation (OK). We prove that OK is optimal for a class of approximations of RTRL, which includes all approaches published so far. Additionally, we show that OK has empirically negligible noise: Unlike previous algorithms it matches TBPTT in a real world task (character-level Penn TreeBank) and can exploit online parameter updates to outperform TBPTT in a synthetic string memorization task. Code availiable on github.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
