arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08218cs.LGcs.AI
收藏

稀疏性感知的模型融合进化

Sparsity-Aware Evolution for Model Merging

DIRO & Institut Courtois, Université de Montréal(迪罗与库尔托伊研究所,蒙特利尔大学);Mila – Quebec AI Institute(魁北克人工智能研究所);Université Sorbonne Paris Nord, LIPN, CNRS(索邦巴黎北大学,LIPN,CNRS);Université Paris Cité, LLF, CNRS(巴黎城市大学,LLF,CNRS);Canada CIFAR AI Chair(加拿大CIFAR人工智能 chair)

Huan Zhang, Yanjian Zhang, Guillaume Wisniewski, Nadi Tomeh, Bang Liu

AI总结:

本文提出了一种稀疏性感知的进化框架,通过迭代剪枝-融合循环提升模型融合的可靠性,且易于集成。

摘要:

我们提出了一种稀疏性感知的进化(SAE)框架,用于模型融合,该框架涉及迭代的剪枝-融合循环,以作为新颖的突变操作符。我们将稀疏性约束纳入评分函数中,这会引导进化过程倾向于更稀疏的模型,除了其他传统性能评分。有趣的是,稀疏性竞争的副产品会在进化过程中引入额外的局部吸引力和相互作用:如果一个竞争者有更多零元素,另一个竞争者的非零元素将占据这些位置,尽管较稀疏的竞争者在其他位置上输给更稀疏的竞争者。所提出的流程在多种大规模大语言模型(LLM)基准上进行了评估。实验表明,我们的方法可以提高模型融合的可靠性,跨多个基准,并且由于其简单性和与大多数现有方法的正交性,易于集成。

英文摘要:

We propose a sparsity-aware evolutionary (SAE) framework for model merging that involves iterative pruning-merging cycles to act as a novel mutation operator. We incorporate the sparsity constraints into the score function, which steers the evolutionary process to favor more sparse models, in addition to other conventional performance scores. Interestingly, the by-product of \textit{competition} for sparsity introduces an extra local \textit{attraction} and interplay into the evolutionary process: if one competitor has more zero elements, the other competitor's non-zero elements will occupy those positions, even though the less sparse competitor loses to the more sparse competitor in other positions. The proposed pipeline is evaluated on a variety of large-scale LLM benchmarks. Experiments demonstrate that our approach can improve model merging reliability across multiple benchmarks, and is easy to incorporate due to its simplicity and being orthogonal to most existing approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑