arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.04882cs.NEcs.AIcs.LGcs.SDeess.AS
收藏

LMUFormer:具有勒让德记忆单元的低复杂度且强大的脉冲模型

LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Units

University of Southern California(南加州大学);Amazon Inc.(亚马逊公司)

Zeyu Liu, Gourav Datta, Anni Li, Peter Anthony Beerel

AI总结:

提出 LMUFormer,一种基于勒让德记忆单元的低复杂度脉冲模型,通过卷积补丁嵌入和通道混合器增强,在 SCv2 数据集上以 53 倍参数和 65 倍 FLOPs 减少达到与 Transformer 相当的性能,并支持流式推理。

摘要:

Transformer 模型在许多应用中表现出高精度,但具有高复杂度和缺乏序列处理能力,使其不适合许多资源受限的边缘流式应用。因此,许多研究者提出将 Transformer 模型重构为 RNN 模块,通过显式状态修改自注意力计算。然而,这些方法通常会导致显著的性能下降。最终目标是开发一个具有以下特性的模型:并行训练、流式推理和低成本推理,以及最先进的性能。在本文中,我们提出了实现这一目标的新方向。我们展示了如何通过架构修改推动循环模型的性能接近 Transformer 模型,同时保留其序列处理能力。具体来说,受勒让德记忆单元(LMU)在序列学习任务中最近成功的启发,我们提出了 LMUFormer,它通过卷积补丁嵌入和卷积通道混合器增强 LMU。此外,我们提出了该架构的脉冲版本,该版本在补丁嵌入和通道混合模块中引入了状态的优势,同时降低了计算复杂度。我们在多个序列数据集上评估了我们的架构。与 ANN 领域内 SCv2 数据集上的最先进 Transformer 模型相比,我们的 LMUFormer 表现出相当的性能,同时参数减少了 53 倍,FLOPs 减少了 65 倍。此外,由于我们的模型在实时数据处理方面的能力,我们实现了序列长度减少 32.03%,同时性能下降微乎其微。我们的代码公开在 https://github.com/zeyuliu1037/LMUFormer.git。

英文摘要:

Transformer models have demonstrated high accuracy in numerous applications but have high complexity and lack sequential processing capability making them ill-suited for many streaming applications at the edge where devices are heavily resource-constrained. Thus motivated, many researchers have proposed reformulating the transformer models as RNN modules which modify the self-attention computation with explicit states. However, these approaches often incur significant performance degradation. The ultimate goal is to develop a model that has the following properties: parallel training, streaming and low-cost inference, and SOTA performance. In this paper, we propose a new direction to achieve this goal. We show how architectural modifications to a recurrent model can help push its performance toward Transformer models while retaining its sequential processing capability. Specifically, inspired by the recent success of Legendre Memory Units (LMU) in sequence learning tasks, we propose LMUFormer, which augments the LMU with convolutional patch embedding and convolutional channel mixer. Moreover, we present a spiking version of this architecture, which introduces the benefit of states within the patch embedding and channel mixer modules while simultaneously reducing the computing complexity. We evaluated our architectures on multiple sequence datasets. In comparison to SOTA transformer-based models within the ANN domain on the SCv2 dataset, our LMUFormer demonstrates comparable performance while necessitating a remarkable 53 times reduction in parameters and a substantial 65 times decrement in FLOPs. Additionally, owing to our model's proficiency in real-time data processing, we can achieve a 32.03% reduction in sequence length, all while incurring an inconsequential decline in performance. Our code is publicly available at https://github.com/zeyuliu1037/LMUFormer.git.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑