arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.09298cs.LGcs.AIcs.CVstat.ML
收藏

是什么让卷积模型在长序列建模中表现出色?

What Makes Convolutional Models Great on Long Sequence Modeling?

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Princeton University(普林斯顿大学);Microsoft Research(微软研究院)

Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, Debadeepta Dey

摘要:

卷积模型在多个领域得到了广泛的应用。然而,大多数现有模型仅使用局部卷积,使得模型无法有效地处理长程依赖。注意力通过聚集全局信息克服了这个问题,但也使计算复杂性与序列长度成二次方。最近,Gu等人[2021]受状态空间模型的启发,提出了一种称为S4的模型。S4可以有效地实现为一个全局卷积模型,其内核大小等于输入序列长度。S4可以建模比Transformers长得多的序列,并在几个远程任务上实现了SoTA的显著增益。尽管S4在经验上取得了成功,但它也参与其中。它需要复杂的参数化和初始化方案。因此,S4不太直观,也很难使用。在这里,我们的目标是揭开S4的神秘面纱,并提取有助于S4作为全局卷积模型取得成功的基本原则。我们专注于卷积核的结构,并确定S4所享有的两个关键但直观的原则,这些原则足以构成有效的全局卷积模型:1)卷积核的参数化需要有效,因为参数的数量应该随序列长度呈次线性扩展。2)核需要满足衰减结构,即与较近的邻居卷积的权重大于较远的邻居。基于这两个原则,我们提出了一个简单而有效的卷积模型,称为结构化全局卷积(SGConv)。SGConv在几个任务上表现出强大的经验性能:1)由于速度更快,SGConv在长距离竞技场和语音命令数据集上超过了S4。2)当将SGConv插入标准语言和视觉模型时,它显示出提高效率和性能的潜力。

英文摘要:

Convolutional models have been widely used in multiple domains. However, most existing models only use local convolution, making the model unable to handle long-range dependency efficiently. Attention overcomes this problem by aggregating global information but also makes the computational complexity quadratic to the sequence length. Recently, Gu et al. [2021] proposed a model called S4 inspired by the state space model. S4 can be efficiently implemented as a global convolutional model whose kernel size equals the input sequence length. S4 can model much longer sequences than Transformers and achieve significant gains over SoTA on several long-range tasks. Despite its empirical success, S4 is involved. It requires sophisticated parameterization and initialization schemes. As a result, S4 is less intuitive and hard to use. Here we aim to demystify S4 and extract basic principles that contribute to the success of S4 as a global convolutional model. We focus on the structure of the convolution kernel and identify two critical but intuitive principles enjoyed by S4 that are sufficient to make up an effective global convolutional model: 1) The parameterization of the convolutional kernel needs to be efficient in the sense that the number of parameters should scale sub-linearly with sequence length. 2) The kernel needs to satisfy a decaying structure that the weights for convolving with closer neighbors are larger than the more distant ones. Based on the two principles, we propose a simple yet effective convolutional model called Structured Global Convolution (SGConv). SGConv exhibits strong empirical performance over several tasks: 1) With faster speed, SGConv surpasses S4 on Long Range Arena and Speech Command datasets. 2) When plugging SGConv into standard language and vision models, it shows the potential to improve both efficiency and performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑