arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.19115cs.LG
收藏

面向非线性RNN的可扩展且稳定的并行化实现

Towards Scalable and Stable Parallelization of Nonlinear RNNs

Stanford University(斯坦福大学);Wu Tsai Neurosciences Institute(吴蔡神经科学研究所);GE Healthcare(GE医疗);Liquid AI

Xavier Gonzalez, Andrew Warrington, Jimmy T. H. Smith, Scott W. Linderman

AI总结:

针对非线性RNN并行方法DEER复杂度高、数值不稳定的问题,提出拟牛顿近似降复杂度、基于ELK关联结合并行卡尔曼平滑提升稳定性的方案,实现更大规模更稳定的非线性RNN并行求值。

摘要:

Transformer和线性状态空间模型可在现代硬件上并行计算,但非线性循环神经网络(RNN)的求值看似是一个固有串行的问题。不过,Lim等人于2024年提出了名为DEER的方法,通过将状态建模为不动点问题的解来并行计算非线性RNN。他们推导了牛顿法的并行形式来求解该不动点问题,相比串行求值实现了显著加速。但DEER的计算复杂度随状态大小呈三次方增长,且算法可能存在数值不稳定的问题。我们通过两项创新解决这些局限:为降低计算复杂度,我们采用拟牛顿近似,证明其收敛性与牛顿法相当,且内存占用更低、速度更快;为提升DEER的稳定性,我们利用Levenberg-Marquardt算法与卡尔曼平滑之间的关联(我们称之为ELK),该关联既能稳定牛顿法,又能借助高效的并行化卡尔曼平滑算法保持性能。通过多项实验,我们证明这些创新可实现更大规模、更稳定的非线性RNN并行求值。

英文摘要:

Transformers and linear state space models can be evaluated in parallel on modern hardware, but evaluating nonlinear RNNs appears to be an inherently sequential problem. Recently, however, Lim et al. '24 developed an approach called DEER, which evaluates nonlinear RNNs in parallel by posing the states as the solution to a fixed-point problem. They derived a parallel form of Newton's method to solve the fixed-point problem and achieved significant speedups over sequential evaluation. However, the computational complexity of DEER is cubic in the state size, and the algorithm can suffer from numerical instability. We address these limitations with two novel contributions. To reduce the computational complexity, we apply quasi-Newton approximations and show they converge comparably to Newton, use less memory, and are faster. To stabilize DEER, we leverage a connection between the Levenberg-Marquardt algorithm and Kalman smoothing, which we call ELK. This connection allows us to stabilize Newton's method while using efficient parallelized Kalman smoothing algorithms to retain performance. Through several experiments, we show that these innovations allow for parallel evaluation of nonlinear RNNs at larger scales and with greater stability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑