arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2110.13985cs.LGcs.AI
收藏

将递归、卷积和连续时间模型与线性状态空间层相结合

Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers

Stanford University(斯坦福大学);University at Buffalo, SUNY(纽约州立大学布法罗分校)

Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, Christopher Ré

摘要:

递归神经网络(RNN)、时间卷积和神经微分方程(NDE)是用于时间序列数据的深度学习模型的流行家族,每种模型在建模能力和计算效率方面都具有独特的优势和权衡。我们引入了一个简单的序列模型的灵感控制系统,概括这些方法,同时解决其缺点。线性状态空间层(LSSL)通过简单地模拟线性连续时间状态空间表示$\dot{x} = Ax + Bu, y = Cx + Du$来映射序列$u \mapsto y$。理论上,我们表明,LSSL模型是密切相关的上述三个家庭的模型,并继承他们的优势。例如,他们将卷积推广到连续时间,解释常见的RNN算法,并分享NDE的特征,如时间尺度适应。然后,我们结合和推广最近的连续时间记忆理论,引入一个可训练的结构矩阵子集$A$,赋予LSSL与长程记忆。从经验上讲,将LSSL层堆叠到一个简单的深度神经网络中,可以在序列图像分类、现实世界的医疗回归任务和语音中获得跨时间序列基准的最新结果。在长度为16000个序列的困难语音分类任务中,LSSL的性能比以前的方法高出24个准确点,甚至比在100倍短序列上使用手工制作的特征的基线更好。

英文摘要:

Recurrent neural networks (RNNs), temporal convolutions, and neural differential equations (NDEs) are popular families of deep learning models for time-series data, each with unique strengths and tradeoffs in modeling power and computational efficiency. We introduce a simple sequence model inspired by control systems that generalizes these approaches while addressing their shortcomings. The Linear State-Space Layer (LSSL) maps a sequence $u \mapsto y$ by simply simulating a linear continuous-time state-space representation $\dot{x} = Ax + Bu, y = Cx + Du$. Theoretically, we show that LSSL models are closely related to the three aforementioned families of models and inherit their strengths. For example, they generalize convolutions to continuous-time, explain common RNN heuristics, and share features of NDEs such as time-scale adaptation. We then incorporate and generalize recent theory on continuous-time memorization to introduce a trainable subset of structured matrices $A$ that endow LSSLs with long-range memory. Empirically, stacking LSSL layers into a simple deep neural network obtains state-of-the-art results across time series benchmarks for long dependencies in sequential image classification, real-world healthcare regression tasks, and speech. On a difficult speech classification task with length-16000 sequences, LSSL outperforms prior approaches by 24 accuracy points, and even outperforms baselines that use hand-crafted features on 100x shorter sequences.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑