将Mamba作为持续学习器：元学习选择性状态空间模型用于持续学习
Learning Mamba as a Continual Learner: Meta-Learning Selective State Space Models for Continual Learning
AI总结：
该研究提出MambaCL，将Mamba作为固定大小状态序列模型，通过选择性正则化元训练，在元持续学习任务中实现了良好性能与强泛化性，展现了固定状态学习器用于高效持续学习的潜力。
摘要：
持续学习（CL）在不存储或重新训练所有已见样本的前提下，从非平稳数据流中学习。元持续学习（MCL）将CL视为序列预测问题，并将持续学习器本身元学习为序列模型，Transformer是自然的选择。然而，尽管Transformer学习器性能尚可，但它依赖线性增长的键值缓存来存储所有过去的表示，这与CL不存储所有已见样本的目标相冲突。具有固定大小状态的序列模型，如线性注意力模型和状态空间模型（SSM），从定义上符合CL的要求；但由于过去的样本被压缩并混合在有限状态中，此类学习器更难进行元学习，早期基于核的模型在MCL上表现不佳。因此，我们旨在获得具有固定大小状态的有效持续学习器，并使其适用于MCL。通过为MCL构建选择性SSM，我们提出了MambaCL，实践中以Mamba为主要模型。我们通过提出的选择性正则化来元训练该学习器，该正则化监督压缩状态内的隐式关联，适用于所有固定状态模型。此外，我们对Mamba和其他固定状态模型在各种MCL场景中的表现进行了系统实证研究，涵盖有效性、泛化性和模型设计。结果显示出良好的性能和强泛化性，证明了固定状态学习器在高效持续学习与适应方面的潜力。
英文摘要：
Continual learning (CL) learns from a non-stationary data stream without storing or re-training on all seen samples. Meta-continual learning (MCL) casts CL as sequence prediction and meta-learns the continual learner itself as a sequence model, with Transformers as natural choices. However, despite decent performance, a Transformer learner relies on a linearly growing key-value cache to store all past representations, conflicting with CL's objective of not storing all seen samples. Sequence models with a constant-size state, e.g., linear-attention models and state-space models (SSMs), match CL's requirement by definition; however, with past samples compressed and mixed in the bounded state, such learners are harder to meta-learn, and earlier kernel-based ones performed poorly on MCL. We thus aim to obtain effective continual learners with constant-size states and make them work for MCL. By formulating the selective SSM for MCL, we propose MambaCL, with Mamba as the main model in practice. We meta-train the learner with a proposed selectivity regularization, which supervises the implicit associations within the compressed state and applies across constant-state models. Furthermore, we conduct a systematic empirical study of how Mamba and other constant-state models behave across various MCL scenarios, covering effectiveness, generalization, and model designs. The results highlight promising performance and strong generalization, demonstrating the potential of constant-state learners for efficient continual learning and adaptation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
