发表机构
School of Computer Science, Sichuan University; School of Artificial Intelligence, Sichuan University（四川大学计算机学院; 四川大学人工智能学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对现有线性Transformer主流方向状态空间模型的输入适应性与内存容量局限，论文提出带记忆专家混合体和时间感知线性注意力的MixFormer，在长序列生成任务上实现性能提升，为下一代网络基础设施提供更可持续的计算主干。
AI 中文摘要
状态空间模型（State Space Models，SSMs）作为线性Transformer的主流研究方向，旨在比标准Transformer实现更高的长上下文建模效率。然而，现有SSMs存在输入适应性有限、内存容量受限的问题，在对超长序列建模时会导致信息丢失。为解决这些局限，我们提出MixFormer，一种整合了记忆专家混合体（Mixture-of-Memory-Experts，MoE）机制的新型线性Transformer。具体而言，该模型通过多个协作的记忆专家维持差异化的记忆状态，并采用新型时间感知线性注意力（Time-Aware Linear Attention，TALA）机制，该机制利用可学习的指数衰减函数和位置偏差来动态更新记忆。此设计使模型能够选择性强化重要历史信息，同时有效缓解记忆稀释，大幅提升长程依赖建模能力。在长序列文本与图像生成任务上的实验表明，MixFormer不仅取得了显著的性能提升，还为下一代网络基础设施提供了更可持续的计算主干。
英文摘要
State Space Models (SSMs), as a mainstream research direction of linear Transformers, aim to achieve higher efficiency than standard Transformers in long-context modeling. However, existing SSMs suffer from limited input adaptivity and constrained memory capacity, leading to information loss when modeling ultra-long sequences. To address these limitations, we propose MixFormer, a novel linear Transformer that integrates a Mixture-of-Memory-Experts (MoE) mechanism. Specifically, the model maintains differentiated memory states through multiple collaborating memory experts and employs a novel Time-Aware Linear Attention (TALA) mechanism, which leverages learnable exponential decay functions and positional biases to dynamically update memory. This design enables the model to selectively reinforce important historical information while effectively mitigating memory dilution, substantially improving long-range dependency modeling. Experiments on long-sequence text and image generation tasks demonstrate that MixFormer not only achieves significant performance gains but also provides a more sustainable computational backbone for the next generation of web infrastructure.
Comments17 pages, 9 figures, and 4 tables
