通过融合高度专业化的语言模型同时精通文本、代码与数学
Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models
AI总结：
本文提出UltraFuser框架，通过token级门控机制融合语言、编程和数学三个高度专业化的专家模型，并构建包含约30万条指令的UltraChat 2数据集，采用两阶段训练策略实现模型对文本、代码和数学三大领域的同时精通。
摘要：
自然语言、编程代码和数学符号的底层数据分布差异巨大，这为大型语言模型（LLMs）在上述三个领域同时实现高性能带来了复杂的挑战。LLM在特定领域达到极高的熟练度通常需要使用相关语料库进行大量训练，这往往伴随着其他领域性能的牺牲。在本文中，我们提出直接融合已经高度专业化的模型。提出的融合框架UltraFuser包含三个分别在语言、编程和数学上已充分训练的不同专家模型。引入了token级别的门控机制来混合各专家模型的输出。设计了伴随平衡采样的两阶段训练策略以确保稳定性。为了有效训练融合模型，我们进一步构建了高质量的监督指令微调数据集UltraChat 2，其中包含文本、代码和数学内容。该数据集包含约300,000条指令，涵盖了每个领域中的广泛主题。实验表明，我们的模型能够同时掌握这三个关键领域。
英文摘要：
Underlying data distributions of natural language, programming code, and mathematical symbols vary vastly, presenting a complex challenge for large language models (LLMs) that strive to achieve high performance across all three domains simultaneously. Achieving a very high level of proficiency for an LLM within a specific domain often requires extensive training with relevant corpora, which is typically accompanied by a sacrifice in performance in other domains. In this paper, we propose to fuse models that are already highly-specialized directly. The proposed fusing framework, UltraFuser, consists of three distinct specialists that are already sufficiently trained on language, coding, and mathematics. A token-level gating mechanism is introduced to blend the specialists' outputs. A two-stage training strategy accompanied by balanced sampling is designed to ensure stability. To effectively train the fused model, we further construct a high-quality supervised instruction tuning dataset, UltraChat 2, which includes text, code, and mathematical content. This dataset comprises approximately 300,000 instructions and covers a wide range of topics in each domain. Experiments show that our model could simultaneously achieve mastery of the three crucial domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
