arXiv 2602.054952026-02-22cs.CLcs.AI

传输与合并：面向大语言模型的跨架构合并

Transport and Merge: Cross-Architecture Merging for Large Language Models

Chenhang Cui, Binyun Yang, Fei Shen, Yuxin Chen, Jingnan Zheng, Xiang Wang, An Zhang, Tat-Seng Chua

AI总结：

本文提出基于最优传输的跨架构合并框架，实现从高资源模型到低资源模型的有效知识转移。

摘要：

大语言模型（LLMs）通过扩展模型容量和训练数据实现强大的能力，但许多实际部署依赖于从小型模型训练或适应低资源数据。这种差距促使需要机制将知识从大型、高资源模型转移到小型、低资源目标。尽管模型合并提供了一种有效的转移机制，但大多数现有方法假设架构兼容的模型，因此无法直接从大型高资源LLMs转移到异构低资源目标。在本工作中，我们提出了一种基于最优传输（OT）的跨架构合并框架，该框架对激活进行对齐，以推断异构模型之间的跨神经元对应关系。所得到的传输计划随后用于指导直接的权重空间融合，使仅使用少量输入即可实现从高资源到低资源的有效转移。在低资源语言和专业领域进行的广泛实验显示，相对于目标模型，有持续的改进。

英文摘要：

Large language models (LLMs) achieve strong capabilities by scaling model capacity and training data, yet many real-world deployments rely on smaller models trained or adapted from low-resource data. This gap motivates the need for mechanisms to transfer knowledge from large, high-resource models to smaller, low-resource targets. While model merging provides an effective transfer mechanism, most existing approaches assume architecture-compatible models and therefore cannot directly transfer knowledge from large high-resource LLMs to heterogeneous low-resource targets. In this work, we propose a cross-architecture merging framework based on optimal transport (OT) that aligns activations to infer cross-neuron correspondences between heterogeneous models. The resulting transport plans are then used to guide direct weight-space fusion, enabling effective high-resource to low-resource transfer using only a small set of inputs. Extensive experiments across low-resource languages and specialized domains demonstrate consistent improvements over target models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文