SeRpEnt：面向表达性状态空间模型的选择性重采样
SeRpEnt: Selective Resampling for Expressive State Space Models
AI总结：
本文提出SeRpEnt架构，通过选择性重采样机制按信息内容压缩序列，在Long Range Arena等基准上验证了其有效性，优于传统状态空间模型。
摘要：
状态空间模型（SSMs）近来在深度学习序列建模领域崭露头角，尤其是作为Transformer的替代方案。其成功源于避免了基于注意力模型的两个众所周知缺点：相对于序列长度的二次复杂度和无法建模长距离依赖。SSM变体Mamba通过使用状态参数的选择性机制，在没有任何注意力形式的情况下展现了与Transformer相当的性能。然而，选择性仅通过经验评估，其有效性原因仍不清楚。在本工作中，我们展示了选择性与序列处理的关系。我们的分析表明，Mamba中的选择性时间间隔充当信息的线性逼近器。然后，我们提出了SeRpEnt架构，这是一种进一步利用选择性以信息感知方式压缩序列的SSM。它采用一种重采样机制，根据元素的信息内容聚合元素。我们在Long Range Arena基准和其他语言建模任务上的实证结果显示了SeRpEnt重采样机制的优势。
英文摘要：
State Space Models (SSMs) have recently enjoyed a rise to prominence in the field of deep learning for sequence modeling, especially as an alternative to Transformers. Their success stems from avoiding two well-known drawbacks of attention-based models: quadratic complexity with respect to the sequence length and inability to model long-range dependencies. The SSM variant Mamba has demonstrated performance comparable to Transformers without any form of attention, thanks to the use of a selective mechanism for the state parameters. Selectivity, however, is only evaluated empirically and the reasons of its effectiveness remain unclear. In this work, we show how selectivity is related to the sequence processing. Our analysis shows that selective time intervals in Mamba act as linear approximators of information. Then, we propose our SeRpEnt architecture, a SSM that further exploits selectivity to compress sequences in an information-aware fashion. It employs a resampling mechanism that aggregates elements based on their information content. Our empirical results in the Long Range Arena benchmark and other language modeling tasks show benefits of the SeRpEnt's resampling mechanism.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
