面向高效序列建模的稀疏模块化激活
Sparse Modular Activation for Efficient Sequence Modeling
AI总结：
针对现有序列混合模型注意力应用静态统一、质量-效率权衡不佳的问题，提出可微的稀疏模块化激活机制SMA，构建架构SeqBoat，实现线性推理复杂度与更优权衡，在多任务上取得线性复杂度混合模型的新SOTA。
摘要：
近期融合线性状态空间模型（Linear State Space Models, SSMs）与自注意力机制的混合模型已在一系列序列建模任务中展现出优异效果。但现有方法会对输入序列中的所有元素静态、统一地应用注意力模块，导致质量-效率权衡未达最优。为解决这一局限，我们提出稀疏模块化激活（Sparse Modular Activation, SMA）——这是一种通用机制，能让神经网络以可微的方式为序列元素稀疏、动态地激活子模块。通过允许每个元素跳过未被激活的子模块，SMA可在训练和推理阶段降低神经网络的计算与内存消耗。为验证SMA在序列建模中的有效性，我们设计了一种新型神经网络架构SeqBoat，它利用SMA，基于从SSM学到的状态表示来稀疏激活门控注意力单元（Gated Attention Unit, GAU）。通过限制GAU仅对被激活的输入执行局部注意力，SeqBoat可实现线性推理复杂度，同时拥有理论上无限的注意力跨度，且相比基于分块的模型能提供更优的质量-效率权衡。我们在长序列建模、语音分类和语言建模等广泛任务上开展了实验，结果表明SeqBoat在具有线性复杂度的混合模型中取得了新的最优结果，并通过学到的稀疏激活模式揭示了每项任务所需的注意力量级。我们的代码已公开于https://github.com/renll/SeqBoat。
英文摘要：
Recent hybrid models combining Linear State Space Models (SSMs) with self-attention mechanisms have demonstrated impressive results across a range of sequence modeling tasks. However, current approaches apply attention modules statically and uniformly to all elements in the input sequences, leading to sub-optimal quality-efficiency trade-offs. To address this limitation, we introduce Sparse Modular Activation (SMA), a general mechanism enabling neural networks to sparsely and dynamically activate sub-modules for sequence elements in a differentiable manner. Through allowing each element to skip non-activated sub-modules, SMA reduces computation and memory consumption of neural networks at both training and inference stages. To validate the effectiveness of SMA on sequence modeling, we design a novel neural architecture, SeqBoat, which employs SMA to sparsely activate a Gated Attention Unit (GAU) based on the state representations learned from an SSM. By constraining the GAU to only conduct local attention on the activated inputs, SeqBoat can achieve linear inference complexity with theoretically infinite attention span, and provide substantially better quality-efficiency trade-off than the chunking-based models. With experiments on a wide range of tasks, including long sequence modeling, speech classification and language modeling, SeqBoat brings new state-of-the-art results among hybrid models with linear complexity, and reveals the amount of attention needed for each task through the learned sparse activation patterns. Our code is publicly available at https://github.com/renll/SeqBoat.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
