arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.10491cs.CL
收藏

大型语言模型的知识融合

Knowledge Fusion of Large Language Models

Sun Yat-sen University(中山大学);Tencent AI Lab(腾讯AI实验室)

Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, Shuming Shi

AI总结:

本文提出LLMs知识融合方法,利用源模型生成分布将多架构LLM能力迁移至单一目标模型,并在Llama-2、MPT和OpenLLaMA上验证其可提升推理、常识和代码生成性能。

摘要:

虽然从零开始训练大型语言模型(LLMs)可以生成具有不同功能和优势的模型,但这会带来巨大的成本,并可能导致冗余的能力。另一种经济高效且引人注目的方法是将现有的预训练LLMs合并成一个更强大的模型。然而,由于这些LLMs的架构各不相同,直接混合它们的权重是不切实际的。在本文中,我们引入了LLMs知识融合的概念,旨在结合现有LLMs的能力并将其迁移到单个LLM中。通过利用源LLMs的生成分布,我们将其集体知识和独特优势外部化,从而有可能将目标模型的能力提升到超越任何单个源LLM的水平。我们使用三种具有不同架构的流行LLMs——Llama-2、MPT和OpenLLaMA——在各种基准和任务上验证了我们的方法。我们的发现证实,LLMs的融合可以在推理、常识和代码生成等一系列能力上提高目标模型的性能。我们的代码、模型权重和数据已在https://github.com/fanqiwan/FuseLLM公开。

英文摘要:

While training large language models (LLMs) from scratch can generate models with distinct functionalities and strengths, it comes at significant costs and may result in redundant capabilities. Alternatively, a cost-effective and compelling approach is to merge existing pre-trained LLMs into a more potent model. However, due to the varying architectures of these LLMs, directly blending their weights is impractical. In this paper, we introduce the notion of knowledge fusion for LLMs, aimed at combining the capabilities of existing LLMs and transferring them into a single LLM. By leveraging the generative distributions of source LLMs, we externalize their collective knowledge and unique strengths, thereby potentially elevating the capabilities of the target model beyond those of any individual source LLM. We validate our approach using three popular LLMs with different architectures--Llama-2, MPT, and OpenLLaMA--across various benchmarks and tasks. Our findings confirm that the fusion of LLMs can improve the performance of the target model across a range of capabilities such as reasoning, commonsense, and code generation. Our code, model weights, and data are public at \url{https://github.com/fanqiwan/FuseLLM}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑