发表机构
Arkadhi Research（阿卡迪研究机构）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究提出持续学习架构CMP，采用稀疏关系编码、两层竞争记忆与局部更新，在语言建模实验中较参数匹配的Transformer基线表现出更低的灾难性遗忘，为持续学习提供了新方向。
AI 中文摘要
我们提出了CMP（认知记忆基元，Cognitive Memory Primitive），这是一种持续学习架构，它将输入表示为稀疏关系编码，存储在两层竞争记忆中，并通过局部更新进行学习，无需通过其特征生成系统进行端到端的反向传播。我们研究了稀疏表示、局部学习和持久记忆的结合是否能比传统的基于反向传播的持续学习方法减少灾难性遗忘。在受控的域增量字节级语言建模协议上，CMP表现出比参数匹配的Transformer（采用在线弹性权重整合EWC训练）低得多的反向迁移。在包含三个随机种子的15个域重复实验中，CMP表现出稳定的遗忘行为，而单独的头对头比较和域顺序分析显示，在报告的实验设置下，CMP的遗忘始终低于所评估的Transformer基线。我们报告了这些发现，同时还报告了CMP相对于Transformer存在显著的单域准确率差距、在视觉基准上的零结果，以及CMP无法与独立的准确率提升机制结合的情况，这体现了我们报告正负结果的承诺。这些结果表明，稀疏表示、局部学习和持久记忆的结合是持续学习的有前景方向，同时推动进一步研究学习规则、表示和架构设计在缓解灾难性遗忘中的各自作用。
英文摘要
We introduce CMP (Cognitive Memory Primitive), a continual-learning architecture that repre?sents inputs as sparse relational codes, stores them in a two-tier competitive memory, and learns through local updates without end-to-end backpropagation through its feature-generating system. We investigate whether combining sparse representations, local learning, and persistent memory can reduce catastrophic forgetting relative to conventional backpropagation-based continual?learning approaches. On a controlled domain-incremental byte-level language modeling protocol, CMP demonstrates substantially lower backward transfer than a parameter-matched Trans?former trained with online Elastic Weight Consolidation (EWC). Across a three-seed replicated 15-domain experiment, CMP exhibits stable forgetting behavior, while separate head-to-head comparisons and domain-order analyses show consistently lower forgetting than the evaluated Transformer baseline under the reported experimental settings. We report these findings alongside a substantial single-domain accuracy gap relative to the Transformer, a null result on a vision benchmark, and a documented failure to combine CMP with an independent accuracy-improving mechanism, reflecting our commitment to reporting both positive and negative outcomes. These results suggest that the combination of sparse representations, local learning, and persistent memory is a promising direction for continual learning, while motivating further investigation into the respective roles of learning rules, representations, and architectural design in mitigating catastrophic forgetting.
Comments28 pages, 11 figures
