一项关于任务级模型合并崩溃的经验研究与理论解释
An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse
AI总结：
本文研究了任务级模型合并中的崩溃现象，揭示了任务表示不兼容性与合并失败之间的强相关性，并通过理论分析建立了任务合并性的基本限制。
摘要：
模型合并将从同一基础模型独立微调的LLM进行统一，使能够重用和整合并行开发的努力而无需重新训练。然而，在实践中我们发现合并并不总是成功：某些任务专家模型在合并后会出现灾难性的性能下降。我们称这种失败模式为合并崩溃。直观上，崩溃发生在不同任务学习的表示或参数调整从根本上不兼容时，使得合并迫使破坏性干扰而非协同作用。在本文中，我们识别并表征了任务级合并崩溃现象，其中某些任务组合在所有合并方法中一致触发巨大的性能下降。通过广泛的实验和统计分析，我们证明了任务之间的表示不兼容性与合并崩溃之间存在强相关性，而参数空间冲突度量显示相关性极低，挑战了模型合并文献中的传统观念。我们通过速率-失真理论提供对这一现象的理论解释，通过维度依赖的界限建立任务合并性的基本限制，无论方法如何。
英文摘要：
Model merging unifies independently fine-tuned LLMs from the same base, enabling reuse and integration of parallel development efforts without retraining. However, in practice we observe that merging does not always succeed: certain combinations of task-specialist models suffer from catastrophic performance degradation after merging. We refer to this failure mode as merging collapse. Intuitively, collapse arises when the learned representations or parameter adjustments for different tasks are fundamentally incompatible, so that merging forces destructive interference rather than synergy. In this paper, we identify and characterize the phenomenon of task-level merging collapse, where certain task combinations consistently trigger huge performance degradation across all merging methods. Through extensive experiments and statistical analysis, we demonstrate that representational incompatibility between tasks is strongly correlated with merging collapse, while parameter-space conflict metrics show minimal correlation, challenging conventional wisdom in model merging literature. We provide a theoretical explanation on this phenomenon through rate-distortion theory with a dimension-dependent bound, establishing fundamental limits on task mergeability regardless of methodology.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
