CP-MoE：一致性保留的混合专家用于持续学习
CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
AI总结：
本文提出CP-MoE，一种基于瞬时专家的持续学习框架，通过一致性保留的路由偏置和瞬时专家引导的正则化机制，减少参数干扰和遗忘，同时保留跨任务知识转移。
摘要：
持续学习在大语言模型（LLMs）和视觉-语言模型（VLMs）中仍面临灾难性遗忘的严重障碍。尽管混合专家（MoE）架构提供了扩展的有效途径，但现有的基于LoRA的MoE持续学习方法仍面临根本性的权衡：要么过于激进地隔离专家，限制任务间的知识转移，要么允许任务特定的更新覆盖重要的现有参数，导致严重的遗忘。为此，我们提出了CP-MoE，一种持续学习框架，围绕瞬时专家构建，该专家捕捉早期任务特定的更新并引导其整合到稳定的专家中。CP-MoE引入了一种一致性保留的路由偏置，利用瞬时专家估计与稳定专家的表示相似性，并引导路由向更兼容的专家选择方向；还引入了一种瞬时专家引导的正则化机制，该机制在合并过程中选择性地保护重要历史参数。这些组件共同减少了参数干扰和遗忘，同时保留了跨任务的知识转移。我们在基于LLM和VLM的MoE模型上验证了CP-MoE，既在单模态又在多模态持续学习基准上进行了测试。在SuperNI基准上，涵盖多样化的序列语言任务，CP-MoE实现了最先进的性能，并在未见任务上表现出更强的零样本迁移能力。在VQA v2数据集上，它能有效扩展到多模态视觉推理，一致地减少遗忘，并优于强大的MoE基线。
英文摘要：
Catastrophic forgetting remains a major obstacle to continual learning in large language models (LLMs) and vision--language models (VLMs). Although Mixture-of-Experts (MoE) architectures offer an efficient path to scaling, existing LoRA-based MoE continual learning methods still face a fundamental trade-off: they either isolate experts too aggressively, limiting knowledge transfer across tasks, or allow task-specific updates to overwrite important existing parameters, leading to severe forgetting. To address this, we propose CP-MoE, a continual learning framework built around a transient expert that captures early task-specific updates and guides their integration into stable experts. CP-MoE introduces a consistency-preserving routing bias, which uses the transient expert to estimate representation similarity with stable experts and steer routing towards more compatible expert selection, and a transient expert-guided regularisation mechanism, which selectively protects important historical parameters during merging. Together, these components reduce parameter interference and forgetting while preserving cross-task knowledge transfer. We validate CP-MoE on both unimodal and multimodal continual learning benchmarks with LLM-based and VLM-based MoE models. On SuperNI benchmark, spanning diverse sequential language tasks, CP-MoE achieves state-of-the-art performance and stronger zero-shot transfer to unseen tasks. On VQA v2 dataset, it scales effectively to multimodal visual reasoning, consistently reduces forgetting, and outperforms strong MoE baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
