循序渐进：通过增量混合秩-1联想记忆专家实现持续学习
Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts
AI总结：
针对持续学习中专家粒度粗糙导致的冗余、干扰和遗忘问题，提出MoRAM方法，将秩-1适配器作为细粒度专家和联想记忆单元，通过自激活机制实现增量扩展，显著提升塑性-稳定性权衡和泛化能力。
摘要：
持续学习（CL）与大型预训练模型旨在增量获取知识而不发生灾难性遗忘。现有的基于LoRA的混合专家（MoE）方法通过添加孤立的新专家并冻结旧专家来扩展容量，但仍存在冗余、干扰、路由模糊以及由此导致的遗忘问题。我们研究了源于粗粒度专家粒度的问题。粗粒度专家（例如高秩LoRA）编码低专一性信息，导致专家重复/干扰以及随着专家积累而路由退化/混乱。在这项工作中，我们提出了MoRAM（混合秩-1联想记忆）。基于权重矩阵作为线性联想记忆的观点，MoRAM将CL实现为可重用原子秩-1专家作为记忆的增量扩展。每个秩-1适配器充当细粒度MoE专家或联想记忆单元。通过将秩-1专家视为键值记忆对，我们消除了显式的MoE-LoRA路由器，采用自激活机制，其中每个记忆原子通过其内在键评估其相关性。因此，推理过程成为对增量累积的学习快照记忆的内容可寻址检索和回忆。在CLIP和LLM上的大量实验表明，MoRAM显著优于最先进的方法，实现了更好的塑性-稳定性权衡、更强的泛化能力和更少的遗忘。项目页面：https://artificer-ai-lab.github.io/MoRAM/。
英文摘要：
Continual learning (CL) with large pre-trained models aims to incrementally acquire knowledge without catastrophic forgetting. Existing LoRA-based Mixture-of-Experts (MoE) methods expand capacity by adding isolated new experts while freezing old ones, but still suffer from redundancy, interference, routing ambiguity, and consequent forgetting. We investigate the issues stemming from coarse-grained expert granularity. Coarse-grained experts (e.g., high-rank LoRA) encode low-specialty information, leading to expert duplication/interference and routing degradation/confusion as experts accumulate. In this work, we propose MoRAM (Mixture of Rank-1 Associative Memory). Grounded in the view that weight matrices act as linear associative memories, MoRAM achieves CL as incremental expansion of reusable atomic rank-1 experts as memory. Each rank-1 adapter acts as a fine-grained MoE expert or an associative memory unit. By viewing rank-1 experts as key-value memory pairs, we eliminate explicit MoE-LoRA routers with self-activation, where each memory atom evaluates its relevance via its intrinsic key. The inference process thus becomes a content-addressable retrieval and recall over the incrementally accumulated memory of learning snapshots. Extensive experiments on CLIP and LLMs show that MoRAM significantly outperforms state-of-the-art methods, achieving a better plasticity-stability trade-off, stronger generalization, and reduced forgetting. Project Page: https://artificer-ai-lab.github.io/MoRAM/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
