一种通过行为和内部机制评估合并方法的流程
A Pipeline to Assess Merging Methods via Behavior and Internals
AI总结：
本文提出了一种评估合并方法的流程，通过行为和内部机制的综合分析，揭示合并对模型性能和语言能力的影响。
摘要：
合并方法通过结合多个语言模型（LMs）的权重来利用其能力，例如用于领域适应。虽然现有研究仅从行为角度研究合并模型，我们提供了首次全面的视角，通过评估和连接其行为和内部机制。我们提出了一种新的评估流程，首先合并多个父LMs，然后基于其在下游任务（如MMLU）上的行为以及内部编码的语言能力来评估合并后的模型，与初始模型进行比较。我们通过评估Qwen2.5家族中经过指令微调、数学和代码适应的LMs的合并来展示此流程。我们的结果表明，合并方法对行为和内部机制的影响不同。虽然合并后的模型性能通常介于两个父模型之间，但其关于语言现象的编码信息，特别是形态和句法方面，可以超越父模型。此外，我们发现这种行为与内部评估之间存在弱排名相关性。通过我们的流程和初步结果，我们强调需要更全面的模型合并方法评估，以获得对其能力和可靠性的真实理解，而不仅仅是潜在的表面行为进步。
英文摘要：
Merging methods combine the weights of multiple language models (LMs) to leverage their capacities, such as for domain adaptation. While existing studies investigate merged models from a solely behavioral perspective, we offer the first comprehensive view by assessing and connecting their behavior and internals. We present a novel evaluation pipeline that first merges multiple parent LMs, and then evaluates the merged models in comparison to the initial ones based on their behavior on downstream tasks, like MMLU, and the internal encoded linguistic competence. We showcase this pipeline by assessing the merging of instruction fine-tuned with math- and code-adapted LMs from the Qwen2.5 family. Our results show that merging methods impacts behavior and internals differently. While the performance of merged models is typically between that of the two parent models, their encoded information about linguistic phenomena, particularly in morphology and syntax, can surpass the parent models. Moreover, we find weak ranking correlation between this behavior and internal evaluation. With our pipeline and initial results, we emphasize the need for more comprehensive evaluations of model merging methods to gain a faithful understanding of their capabilities and reliability, beyond potential superficial behavioral advances.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
