用于大语言持续学习的子专家混合模型
Mixtures of SubExperts for Large Language Continual Learning
AI总结：
研究大语言持续学习中稳定性-可塑性困境及可扩展性问题，提出子专家混合模型MoSEs，通过模块化稀疏性和组合路由，平衡困境，实现知识隔离、重组扩展及亚线性增长，实验验证其性能优于基线，为持续学习基础模型提供关键归纳偏置。
摘要：
在大语言模型中实现终身学习需要解决稳定性-可塑性困境（即模型必须纳入新知识而不覆盖先前的表示），同时在参数增长受限的情况下保持可扩展性。现有方法无法满足这三个要求；共享参数方法存在灾难性干扰，而任务隔离扩展排除了知识转移且线性扩展。我们提出子专家混合模型（MoSEs），它将模型能力分解为可复用的组合原语，通过轻量级子专家和学习到的子路由函数增强Transformer层，可动态选择和组合模块的稀疏子集。通过三个支柱平衡困境：在稀疏激活模块中隔离知识实现稳定性；通过路由驱动的重组和选择性扩展实现可塑性；通过有效容量的亚线性增长实现可扩展性。路由机制实现了组合泛化。在TRACE和SuperNI上的实验验证显示，MoSEs减少遗忘、改善前向转移且参数效率更高，建立了新的帕累托前沿，在保持严格参数预算的同时达到了当前最优性能。结果表明模块化稀疏性和组合路由是构建持续学习且不饱和的基础模型的关键归纳偏置。
英文摘要：
Enabling lifelong learning in LLMs demands resolving the stability-plasticity dilemma (i.e., models must incorporate new knowledge without overwriting prior representations) while maintaining scalability under bounded parameter growth. Existing PEFT methods fail to satisfy this triad; shared-parameter approaches suffer from catastrophic interference, while task-isolated expansions preclude knowledge transfer and scale linearly. We propose Mixtures of SubExperts (MoSEs), a modular and sparse framework that factorizes model capacity into reusable, compositional primitives. MoSEs augment transformer layers with lightweight SubExperts and a learned sub-routing function that dynamically selects and composes a sparse subset of modules conditioned on task inputs. This induces a structured decomposition of the parameter space where knowledge is localized yet accessible, mitigating interference while preserving reuse. Specifically, MoSEs balance the dilemma via three pillars: (i) stability by isolating knowledge within sparsely activated modules, (ii) plasticity through routing-driven recombination and selective expansion, and (iii) scalability via sublinear growth in effective capacity. Notably, the routing mechanism enables compositional generalization, allowing new tasks to be represented as combinations of previously acquired sub-functions. We empirically validate MoSEs on TRACE and SuperNI, showing reduced forgetting, improved forward transfer, and better parameter efficiency over strong PEFT baselines. MoSEs establish a new Pareto frontier, achieving state-of-the-art performance while maintaining strict parameter budgets. Our results suggest that modular sparsity and compositional routing are key inductive biases for building foundation models that continually learn without saturation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
