发表机构
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究将模型合并的参数选择转化为黑箱多目标优化问题，提出MOBO-Merge框架，经实验验证其在多数场景下优于随机搜索，为模型合并提供了高效的搜索方案。
AI 中文摘要
模型合并直接在权重空间中组合已训练好的模型，相比额外微调提供了一种计算高效的替代方案。然而，选择合并参数十分困难，因为下游评估成本高昂、梯度不可用，且源模型的能力可能存在冲突。我们将合并参数选择表述为一个黑箱多目标优化问题，并提出了MOBO-Merge，这是一种与合并算子无关的框架，在有限的评估预算下使用多目标贝叶斯优化来近似帕累托前沿。我们在两模型指令-数学和三模型指令-数学-代码设置中，使用Linear、SLERP、TIES和分块合并算子评估了Qwen3-4B和Llama-3.1-8B。在保留的基准划分上，MOBO-Merge在12次报告的比较中有11次获得了比随机搜索更高的平均超体积。对于一维Linear插值，增益较小，但对于若干TIES、分块和三目标搜索，增益明显更大。没有任何合并算子是全局最优的：在四组家族-设置组合中，TIES在三组中表现最佳，而Block-Linear 4x在Llama三模型合并中性能最强。这些结果表明，多目标贝叶斯优化作为表达性合并参数化的搜索层具有重要价值。
英文摘要
Model merging combines trained models directly in weight space, offering a compute-efficient alternative to additional fine-tuning. Selecting merge parameters is nevertheless difficult because downstream evaluations are expensive, gradients are unavailable, and source capabilities can conflict. We formulate merge-parameter selection as a black-box multi-objective optimization problem and introduce MOBO-Merge, a merge-operator agnostic framework that uses multi-objective Bayesian optimization to approximate the Pareto front under a limited evaluation budget. We evaluate Qwen3-4B and Llama-3.1-8B in two-model instruction-math and three-model instruction-math-code settings using Linear, SLERP, TIES, and block-wise merge operators. On held-out benchmark partitions, MOBO-Merge obtains higher mean hypervolume than random search in 11 of 12 reported comparisons. The gain is small for one-dimensional Linear interpolation but substantially larger for several TIES, block-wise, and three-objective searches. No merge operator is uniformly best: TIES leads in three of four family-setting combinations, whereas Block-Linear 4x is strongest for the Llama three-model merge. These results show that multi-objective Bayesian optimization is valuable as a search layer for expressive merge parameterizations.
