带分层记忆的Mamba：解决长序列建模中的表示瓶颈
Mamba with Hierarchical Memory: Solving Representation Bottleneck in Long Sequence Modeling
浏览论文内容
中文总结 AI 辅助
该研究提出分层记忆Mamba（HMM），通过分层记忆机制解决Mamba类循环线性注意力模型的长序列表示瓶颈，在两项任务上提升性能且参数和训练开销增加极少。
中文摘要 AI 辅助
循环线性注意力模型（RLAs）如Mamba，提供了高效的线性时间序列建模，可作为Transformer的替代方案，但其固定容量的循环状态限制了长序列建模。受人类分层记忆启发，我们提出分层记忆Mamba（HMM）以解决这一局限。在预训练的Mamba主干基础上，HMM集成了轻量工作记忆，从主干隐藏状态嵌入的快速感官记忆中提取慢速段落级语义（PLS）；PLS随后被压缩为持久长期记忆，用于任务相关检索。语义信息的分层处理克服了RLAs的表示瓶颈，并通过参数学习赋予HMM跨任务泛化能力，这是其他增强长上下文的Mamba变体未具备的。在密码检索（Passkey Retrieval）和LongBench-E任务上的评估显示，与强大的Mamba基模型相比，HMM将检索成功率提升了34.3%至37.1%，推理准确率提升了1.6%至14.2%，同时仅增加2%的额外参数，且训练开销极小。
英文摘要
Recurrent linear attention models (RLAs) such as Mamba offer efficient linear-time sequence modeling as an alternative to Transformers, yet their fixed-capacity recurrent states limit long-sequence modeling. Drawing inspiration from hierarchical human memory, we propose Hierarchical Memory Mamba (HMM) to address this limitation. Building upon a pre-trained Mamba backbone, HMM integrates a lightweight working memory that extracts slow paragraph-level semantics (PLS) from the fast sensory memory embedded in the backbone's hidden states. The PLS is subsequently compressed into persistent long-term memory for task-relevant retrieval. The hierarchical processing of semantic information overcomes the representation bottleneck of RLAs and endows HMM cross-task generalization through parametric learning, which is not observed in other long-context enhanced Mamba variants. Evaluations on Passkey Retrieval and LongBench-E tasks demonstrate that HMM improves retrieval success by 34.3--37.1% and reasoning accuracy by 1.6--14.2% over strong Mamba-based models, while adding only 2% extra parameters and with minimal training overhead.
发表机构
- The Hong Kong University of Science and Technology（香港科技大学）
- University of Oxford（牛津大学）
- The Chinese University of Hong Kong（香港中文大学）
- City University of Hong Kong（香港城市大学）
- Hainan Bielefeld University of Applied Sciences（海南比勒费尔德应用科学大学）
- BrainGalaxy
机构由 AI 辅助整理，请以论文原文为准。
