arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2505.20144cs.CLcs.LG
收藏

SeMe:通过语义对齐实现免训练语言模型合并

SeMe: Training-Free Language Model Merging via Semantic Alignment

Monash University(莫纳什大学);Technical University of Munich(慕尼黑工业大学);Chongqing University(重庆大学)

Jian Gu, Aldeida Aleti, Chunyang Chen, Hongyu Zhang

AI总结:

SeMe提出一种无数据、免训练的语义对齐方法,在逐层级别合并语言模型,保留行为并稳定内部知识,优于现有方法,为知识感知模型合并建立新范式。

摘要:

尽管语言模型(LMs)在多种任务上展现出卓越的能力,但没有任何单一模型能够始终优于其他模型,因此需要高效的方法来结合它们的优势,而无需昂贵的重新训练。现有的模型合并技术,如参数平均和任务引导融合,往往依赖于数据相关的计算,或无法保留内部知识,限制了其鲁棒性和可扩展性。我们提出了SeMe(基于语义的合并),一种新颖的、无数据且免训练的方法,利用潜在语义对齐在细粒度、逐层级别上合并语言模型。与以往工作不同,SeMe不仅保留了模型行为,还明确地稳定了内部知识,解决了语言模型融合中的一个关键空白。通过在多种架构和任务上的广泛实验,我们证明SeMe在性能和效率上均优于现有方法,同时消除了对外部数据的依赖。我们的工作为知识感知的模型合并建立了新范式,并提供了对语言模型语义结构的洞察,为更可扩展和可解释的模型组合铺平了道路。

英文摘要:

Despite the remarkable capabilities of Language Models (LMs) across diverse tasks, no single model consistently outperforms others, necessitating efficient methods to combine their strengths without expensive retraining. Existing model merging techniques, such as parameter averaging and task-guided fusion, often rely on data-dependent computations or fail to preserve internal knowledge, limiting their robustness and scalability. We introduce SeMe (Semantic-based Merging), a novel, data-free, and training-free approach that leverages latent semantic alignment to merge LMs at a fine-grained, layer-wise level. Unlike prior work, SeMe not only preserves model behaviors but also explicitly stabilizes internal knowledge, addressing a critical gap in LM fusion. Through extensive experiments across diverse architectures and tasks, we demonstrate that SeMe outperforms existing methods in both performance and efficiency while eliminating reliance on external data. Our work establishes a new paradigm for knowledge-aware model merging and provides insights into the semantic structure of LMs, paving the way for more scalable and interpretable model composition.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑