arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.17445cs.LGcs.AIcs.CL
收藏

将指数平滑融入MLP:一种简单但有效的序列模型

Incorporating Exponential Smoothing into MLP: A Simple but Effective Sequence Model

Peking University(北京大学)

Jiqun Chu, Zuoquan Lin

AI总结:

本文提出将指数平滑直接融入逐元素MLP构建简单序列模型,在参数增加不到1%的情况下,在LRA基准上取得了与S4相当的效果,证明了简单SSM的潜力。

摘要:

对序列数据中的长程依赖进行建模是序列学习的关键步骤。最近开发的模型Structured State Space (S4)在长程序列建模中展现出显著效果。然而,S4的成功究竟归因于其复杂的参数化和HiPPO初始化,还是仅仅因为State Space Models (SSMs),目前尚不清楚。为了进一步探究深度SSMs的潜力,我们从指数平滑(ETS)这一简单的SSM出发,通过将其直接融入逐元素MLP,提出了一种堆叠架构。我们为简单的ETS增加了额外参数和复数域,以降低归纳偏置。尽管增加的参数量不到逐元素MLP的1%,我们的模型在LRA基准上取得了与S4相当的结果。

英文摘要:

Modeling long-range dependencies in sequential data is a crucial step in sequence learning. A recently developed model, the Structured State Space (S4), demonstrated significant effectiveness in modeling long-range sequences. However, It is unclear whether the success of S4 can be attributed to its intricate parameterization and HiPPO initialization or simply due to State Space Models (SSMs). To further investigate the potential of the deep SSMs, we start with exponential smoothing (ETS), a simple SSM, and propose a stacked architecture by directly incorporating it into an element-wise MLP. We augment simple ETS with additional parameters and complex field to reduce the inductive bias. Despite increasing less than 1\% of parameters of element-wise MLP, our models achieve comparable results to S4 on the LRA benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑