arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.22285cs.LG
收藏

揭秘可合并性:预测模型合并成功的可解释属性

Demystifying Mergeability: Interpretable Properties to Predict Model Merging Success

University of California, Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学)

Luca Zhou, Bo Zhao, Rose Yu, Emanuele Rodolà

AI总结:

通过架构无关框架和L1正则化线性优化,发现合并成功取决于合并方法和伙伴任务,梯度对齐是最基本的兼容性信号。

摘要:

模型合并结合了分别微调模型的知识,但驱动其成功的因素仍知之甚少。虽然最近的工作将可合并性视为模型的内在属性,但我们通过一个架构无关的框架表明,它从根本上取决于合并方法和伙伴任务。使用基于一组可解释的成对度量(例如,梯度L_2距离)的L1正则化线性优化,我们发现了与五种合并方法合并后归一化准确率相关的属性。我们发现成功驱动因素存在架构和方法特定的变化(平均前5个度量重叠64.0%;符号一致性79.3%),某些方法,特别是TIES,表现出与广泛共识不同的独特“指纹”。然而,至关重要的是,梯度对齐度量始终作为兼容性的最基本信号出现。这些发现为理解可合并性提供了诊断基础,并激发了未来合并感知的微调策略。

英文摘要:

Model merging combines knowledge from separately fine-tuned models, yet the factors driving its success remain poorly understood. While recent work treats mergeability as an intrinsic property of the models, we show with an architecture-agnostic framework that it fundamentally depends on both the merging method and the partner tasks. Using L1-regularized linear optimization over a set of interpretable pairwise metrics (e.g., gradient L_2 distance), we uncover properties correlating with post-merge normalized accuracy across five merging methods. We find architecture- and method-specific variation in success drivers (64.0% average top-5 metric overlap; 79.3% sign agreement), with certain methods, notably TIES, exhibiting distinct ``fingerprints'' that diverge from the broader consensus. Crucially, however, gradient alignment metrics consistently emerge as the most fundamental signals of compatibility. These findings provide a diagnostic foundation for understanding mergeability and motivate future merge-aware fine-tuning strategies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑