Longhorn:状态空间模型是摊销在线学习者
Longhorn: State Space Models are Amortized Online Learners
AI总结:
本文从在线学习目标推导状态转移规则,提出深度 SSM 架构 Longhorn,在语言与视觉等任务上超过 Mamba,并显著提升样本效率和长上下文外推能力。
摘要:
现代大型语言模型建立在通过下一 token 预测进行序列建模的基础之上。尽管 Transformer 仍是序列建模的主导架构,但其解码复杂度随序列长度呈二次增长,这构成了一个主要限制。状态空间模型(SSM)提供了一种具有竞争力的替代方案,可在线性解码效率下运行,同时在训练期间保持并行性。然而,大多数现有 SSM 依赖于线性递归设计,这些设计看起来有些临时性。在本工作中,我们从在线学习的视角探索 SSM 设计,将 SSM 概念化为面向特定在线学习问题的元模块。该方法把 SSM 设计与制定精确的在线学习目标联系起来,其状态转移规则由求解这些目标导出。基于这一洞见,我们提出一种新颖的深度 SSM 架构 Longhorn,其更新类似于求解在线联想回忆问题的闭式解。实验结果表明,Longhorn 在标准序列建模基准、语言建模和视觉任务上优于包括 Mamba 模型在内的最先进 SSM。具体而言,与 Mamba 相比,Longhorn 的样本效率提升 1.8 倍,并能在推理时外推至最长 16 倍的上下文。
英文摘要:
Modern large language models are built on sequence modeling via next-token prediction. While the Transformer remains the dominant architecture for sequence modeling, its quadratic decoding complexity in sequence length poses a major limitation. State-space models (SSMs) present a competitive alternative, offering linear decoding efficiency while maintaining parallelism during training. However, most existing SSMs rely on linear recurrence designs that appear somewhat ad hoc. In this work, we explore SSM design through the lens of online learning, conceptualizing SSMs as meta-modules for specific online learning problems. This approach links SSM design to formulating precise online learning objectives, with state transition rules derived from solving these objectives. Based on this insight, we introduce a novel deep SSM architecture, Longhorn, whose update resembles the closed-form solution for solving the online associative recall problem. Our experimental results show that Longhorn outperforms state-of-the-art SSMs, including the Mamba model, on standard sequence modeling benchmarks, language modeling, and vision tasks. Specifically, Longhorn achieves a 1.8x improvement in sample efficiency compared to Mamba, and can extrapolate over contexts that are up to 16x longer during inference.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。