稀疏子空间到专家共享的任务无关持续学习
Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning
AI总结：
提出SETA框架，通过将参数分解为任务特定专家和共享专家的稀疏子空间，结合自适应弹性锚定和路由感知正则化，解决LLM持续学习中的塑性-稳定性困境，在多个基准上优于现有方法。
摘要：
大型语言模型（LLM）中的持续学习受到塑性-稳定性困境的阻碍，获取新能力往往导致先前知识的灾难性遗忘。现有方法通常统一对待参数，未能区分特定任务知识和共享能力。我们提出了用于任务无关持续学习的稀疏专家混合（SETA）框架，该框架通过将参数自适应稀疏子空间分解为任务特定专家模块来解决塑性-稳定性冲突。与标准更新（其中任务竞争相同参数）不同，SETA将知识分离为独特专家（旨在隔离任务特定模式）和共享专家（负责捕获共同特征）。这种结构通过自适应弹性锚定和路由感知正则化来维护，该正则化在权重和路由级别共同保护共享知识，并使统一的门控网络能够在推理过程中自动检索正确的专家组合。在多种领域特定基准上的大量实验表明，相对于最先进的持续学习基线，SETA实现了具有竞争力或更优的整体性能，特别是在LLaMA-2 7B和Qwen3-4B上，早期任务知识的保持和反向迁移能力尤为突出。
英文摘要：
Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities. We introduce Mixture of Sparse Experts for Task Agnostic Continual Learning (SETA), a framework that resolves the plasticity-stability conflict through adaptive sparse subspace decomposition into task-specific expert modules. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into unique experts, designed to isolate task-specific patterns, and shared experts, responsible for capturing common features. This structure is maintained through adaptive elastic anchoring and a routing-aware regularization that jointly protect shared knowledge at both the weight and routing levels and enable a unified gating network to automatically retrieve the correct expert combination during inference. Extensive experiments across diverse domain-specific benchmarks demonstrate that SETA achieves competitive or superior overall performance relative to state-of-the-art continual learning baselines, with particularly strong retention of early-task knowledge and improved backward transfer on LLaMA-2 7B and Qwen3-4B.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
