MINGLE：用于测试时持续模型合并的零空间门控低秩专家混合模型
MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging
AI总结：
针对持续模型合并中存在的灾难性遗忘与测试分布适应性不足问题，提出了TTCMM任务与MINGLE框架，结合低秩专家混合、零空间约束门控及自适应松弛策略，在基准上平均超越此前最优方法7-9%。
摘要：
持续模型合并可在无法获取原始训练数据的情况下，依次整合独立微调的模型，为持续学习提供了一种可扩展且高效的解决方案。然而，现有方法面临两大关键挑战：任务间的参数干扰会导致灾难性遗忘，且对不断变化的测试分布的适应性有限。为解决这些问题，我们提出了测试时持续模型合并（Test-Time Continual Model Merging，TTCMM）任务，该任务在推理阶段利用少量无标注测试样本缓解参数冲突并应对分布偏移。我们提出了一种面向TTCMM的新型框架MINGLE。MINGLE采用由参数高效的低秩专家构成的专家混合架构，在动态合并模型以缓解冲突的同时，增强了对不断变化的测试分布的适应性。为进一步减少遗忘，我们提出了零空间约束门控（Null-Space Constrained Gating）机制，将门控更新限制在与先前任务表示正交的子空间中，从而抑制旧任务上的激活并保留过往知识。我们进一步引入了自适应松弛策略（Adaptive Relaxation Strategy），根据测试时自适应过程中观测到的干扰信号动态调整约束强度，在稳定性与适应性之间取得平衡。在标准持续合并基准上的大量实验表明，MINGLE实现了鲁棒的泛化能力，显著降低了遗忘程度，在不同任务顺序下的平均性能始终超越此前的最优方法7-9%。我们的代码可在以下链接获取：https://github.com/zihuanqiu/MINGLE
英文摘要：
Continual model merging integrates independently fine-tuned models sequentially without access to the original training data, offering a scalable and efficient solution for continual learning. However, existing methods face two critical challenges: parameter interference among tasks, which leads to catastrophic forgetting, and limited adaptability to evolving test distributions. To address these issues, we introduce the task of Test-Time Continual Model Merging (TTCMM), which leverages a small set of unlabeled test samples during inference to alleviate parameter conflicts and handle distribution shifts. We propose MINGLE, a novel framework for TTCMM. MINGLE employs a mixture-of-experts architecture with parameter-efficient, low-rank experts, which enhances adaptability to evolving test distributions while dynamically merging models to mitigate conflicts. To further reduce forgetting, we propose Null-Space Constrained Gating, which restricts gating updates to subspaces orthogonal to prior task representations, thereby suppressing activations on old tasks and preserving past knowledge. We further introduce an Adaptive Relaxation Strategy that adjusts constraint strength dynamically based on interference signals observed during test-time adaptation, striking a balance between stability and adaptability. Extensive experiments on standard continual merging benchmarks demonstrate that MINGLE achieves robust generalization, significantly reduces forgetting, and consistently surpasses previous state-of-the-art methods by 7-9% on average across diverse task orders. Our code is available at: https://github.com/zihuanqiu/MINGLE
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
