面向流形感知的自适应持续模型合并方法
Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution
AI总结：
本文提出MADE-IT方法，通过流形几何引导专家表示，解决持续模型合并中的参数饱和与冗余问题，实现自适应专家进化与隐式路由，提升模型在长周期任务序列中的准确性和鲁棒性。
摘要：
持续模型合并（CMM）通过顺序整合任务特定模型到统一架构，无需大量再训练。然而现有方法面临根本的饱和-冗余困境：基于骨干网络的方法在固定容量内遭遇参数饱和和表征干扰，而混合专家（MoE）变体则通过无差别扩展导致专家冗余和路由瓶颈。为解决这些挑战，我们提出MADE-IT（流形感知动态专家进化与隐式路由），通过基于流形几何的内在专家表示协调专家管理和激活。我们引入基于投影的子空间亲和度度量与分布感知的自适应阈值机制，引导自主专家进化，平衡多样性与架构简洁性。此外，为绕过参数化门控网络，我们设计一种数据自由且训练自由的隐式路由机制，通过特征子空间对齐激活专家。广泛实验表明，MADE-IT在长周期和洗牌任务序列中优于强基线，在准确性和鲁棒性方面表现优异，同时显著剪枝冗余专家，特别是在通用模块和早期层。
英文摘要：
Continual Model Merging (CMM) sequentially integrates task-specific models into a unified architecture without intensive retraining. However, existing CMM methods are hindered by a fundamental saturation-redundancy dilemma: backbone-centric approaches face parameter saturation and representation interference within fixed capacities, whereas Mixture-of-Experts (MoE) variants resort to indiscriminate expansion, incurring expert redundancy and a routing bottleneck reliant on additional data-driven optimization. To resolve these challenges, we propose MADE-IT (Manifold-Aware Dynamic Expert Evolution and Implicit rouTing), an adaptive CMM method that orchestrates expert management and activation by grounding intrinsic expert representations in manifold geometry. We introduce a projection-based subspace affinity metric coupled with a distribution-aware adaptive threshold mechanism to guide autonomous expert evolution, harmonizing diversity with architectural parsimony. Furthermore, to bypass parameterized gating networks, we design a data-free and training-free implicit routing mechanism that activates experts via feature-subspace alignment. Extensive experiments demonstrate that MADE-IT consistently outperforms strong baselines in accuracy and robustness across long-horizon and shuffled task sequences, while significantly pruning redundant experts, particularly within generic modules and early layers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
