CABS+：基于冲突感知稀疏化与自适应权重分配的高效可扩展模型融合方法

中文摘要 AI 辅助

模型融合作为一种无需额外重新训练即可构建统一多任务模型的有前景范式，近来受到广泛关注。然而，跨任务的参数冲突与知识干扰往往会降低融合模型的性能。现有工作提出了冲突感知平衡稀疏化（Conflict-Aware and Balanced Sparsification, CABS），该方法通过结构化剪枝与顺序掩码减少参数干扰，但CABS依赖网格搜索确定缩放系数，导致时间复杂度呈指数级，且其优化目标可能被高性能任务主导，进而导致整体性能未达最优。为解决这些局限，我们扩展了CABS并提出CABS+。具体而言，自适应权重分配（Adaptive Weight Allocation, AWA）通过无梯度搜索方案优化融合系数以降低时间复杂度，同时非对称适应度函数可促进跨任务更全面的性能提升。此外，我们对影响模型融合性能的关键因素开展了系统实证研究，并提出相对协同得分（Relative Synergy Score, RSS）以量化模型可融合性并指导模型选择。我们在覆盖大语言模型、小规模语言模型与视觉模型的5种模型及27个数据集上，将CABS+与CABS、AdaMerging、WUDIMerging等先进模型融合方法进行对比。大量实验验证了CABS+的有效性与效率：与AdaMerging和WUDIMerging相比，CABS+的整体性能分别提升16.97%和12.93%，在不同任务数量与模型架构下表现出更强的稳定性与鲁棒性，使用的GPU内存不足AdaMerging的25%，且融合时间较WUDIMerging实现近4倍加速。

英文摘要

Model merging has recently attracted significant attention as a promising paradigm for constructing unified multi-task models without requiring additional retraining. However, parameter conflicts and knowledge interference across tasks often degrade merged-model performance. Prior work introduced Conflict-Aware and Balanced Sparsification (CABS), which reduces parameter interference through structured pruning and sequential masking. However, CABS relies on grid search to determine scaling coefficients, resulting in exponential time complexity, while its optimization objective can be dominated by high-performance tasks, leading to suboptimal overall performance. To address these limitations, we extend CABS and propose CABS+. Specifically, Adaptive Weight Allocation (AWA) optimizes merging coefficients via a gradient-free search scheme to reduce time complexity, while an asymmetric fitness function promotes more comprehensive performance gains across tasks. Moreover, we conduct a systematic empirical study of key factors influencing model merging performance and propose Relative Synergy Score (RSS) to quantify model mergeability and guide model selection. We compare CABS+ with state-of-the-art model merging methods, including CABS, AdaMerging, and WUDIMerging, across 27 datasets and 5 models covering large language, small-scale language, and vision models. Extensive experiments verify the effectiveness and efficiency of CABS+. Compared with AdaMerging and WUDIMerging, CABS+ improves overall performance by 16.97% and 12.93%, respectively, exhibits stronger stability and robustness across varying task numbers and model architectures, uses less than 25% of the GPU memory required by AdaMerging, and achieves nearly a 4x speedup in merging time over WUDIMerging.

发表机构

• Beihang University（北京航空航天大学）
• Hangzhou Innovation Institute of Beihang University（北京航空航天大学杭州创新研究院）
• National University of Singapore（新加坡国立大学）

机构由 AI 辅助整理，请以论文原文为准。