通过丘脑路由皮层柱实现语言模型的高效持续学习
Efficient Continual Learning in Language Models via Thalamically Routed Cortical Columns
AI总结：
TRC$^{2}$通过丘脑路由和海马路径实现持续学习，提升任务边界建模质量并减少累积遗忘，优于多种基线模型。
摘要：
大型语言模型在实际部署中必须适应不断变化的数据、用户行为和任务混合，而无需擦除之前获得的能力。在实践中，这仍然具有挑战性：顺序更新导致灾难性遗忘，而许多稳定方法依赖于外部过程，这些过程成本高、脆弱且难以扩展。我们提出了TRC$^{2}$（丘脑路由皮层柱），一种仅解码器的架构，使持续学习成为骨干自身的属性。TRC$^{2}$结合了堆叠的皮层柱和丘脑调节路径，用于选择性列间通信，以及海马路径用于事件选择性检索、延迟惊喜写作和由重放驱动的巩固。这种设计本地化了快速可塑性，同时保持了较慢的稳定计算路径。我们进一步引入了因果记忆更新方案和在线重放控制器，该控制器根据测量的遗忘调整巩固强度。在C4、WikiText-103和GSM8K的任务序列语言建模流中，TRC$^{2}$在任务边界建模质量上持续改进，并相对于Transformer、Mamba、MoE、DeepSeek和在相同管道下训练的持续学习基线显著减少了累积遗忘。消融实验表明，丘脑和海马组件对保留增益至关重要，而完整模型在吞吐量和训练成本方面仍具有竞争力。
英文摘要：
Large language models deployed in the wild must adapt to evolving data, user behavior, and task mixtures without erasing previously acquired capabilities. In practice, this remains difficult: sequential updates induce catastrophic forgetting, while many stabilization methods rely on external procedures that are costly, brittle, or difficult to scale. We present TRC$^{2}$ (Thalamically Routed Cortical Columns), a decoder-only architecture that makes continual learning a property of the backbone itself. TRC$^{2}$ combines stacked cortical columns with a thalamic modulatory pathway for selective inter-column communication and a hippocampal pathway for event selective retrieval, delayed surprise-based writing, and replay-driven consolidation. This design localizes fast plasticity while preserving a slower stable computation pathway. We further introduce a causal memory-update scheme and an online replay controller that adjusts consolidation strength from measured forgetting. Across a task-sequential language-modeling stream over C4, WikiText-103, and GSM8K, TRC$^{2}$ consistently improves task-boundary modeling quality and substantially reduces cumulative forgetting relative to Transformer, Mamba, MoE, DeepSeek and continual learning baselines trained under the same pipeline. Ablations show that the thalamic and hippocampal components are central to the retention gains, while the full model remains competitive in throughput and training cost.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
