arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.06624cs.CLcs.AIcs.LG
收藏

在Llama-3 70B上进行训练实践:最优额外语言混合比例的选择

A Practice of Post-Training on Llama-3 70B with Optimal Selection of Additional Language Mixture Ratio

Geely AI Lab(吉利人工智能实验室);Beihang University(北航)

Ningyuan Xi, Yetao Wu, Kun Fan, Teng Chen, Qingqing Gu, Luo Ji

AI总结:

本文通过在Llama-3 8B和70B上进行持续预训练,研究额外语言混合比例与学习率的最优相关性,提升中文能力及数学、编程等领域的表现,并在实际聊天系统中部署70B模型取得良好效果。

摘要:

大型语言模型(LLM)通常需要持续预训练(CPT)以获得不熟悉的语言技能或适应新领域。CPT的高昂训练成本往往要求谨慎选择关键超参数,如额外语言或领域语料的混合比例。然而,目前尚无系统研究将最优混合比例与实际模型性能之间的差距以及实验扩展定律与全模型大小实际部署之间的差距联系起来。本文在Llama-3 8B和70B上进行CPT以增强其中文能力。我们研究了8B规模上额外语言混合比例(ALMR)与学习率(LR)之间的最优相关性,这直接表明了最优实验设置。通过全面的超参数选择和后续微调,模型不仅在中文相关基准上得到提升,还在数学、编程和情感智能等特定领域有所提升。我们部署了最终的70B LLM到实际聊天系统中,并获得了令人满意的性能。

英文摘要:

Large Language Models (LLM) often need to be Continual Pre-Trained (CPT) to obtain unfamiliar language skills or adapt to new domains. The huge training cost of CPT often asks for cautious choice of key hyper-parameters such as the mixture ratio of extra language or domain corpus. However, there is no systematic study that bridges the gap between the optimal mixture ratio and the actual model performance, and the gap between experimental scaling law and the actual deployment in the full model size. In this paper, we perform CPT on Llama-3 8B and 70B to enhance its Chinese ability. We study the optimal correlation between the Additional Language Mixture Ratio (ALMR) and the Learning Rate (LR) on the 8B size which directly indicates the optimal experimental setup. By thorough choice of hyper-parameter, and subsequent fine-tuning, the model capability is improved not only on the Chinese-related benchmark but also in some specific domains including math, coding, and emotional intelligence. We deploy the final 70B version of LLM on a real-life chat system which obtains satisfying performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑