arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.06273cs.CL
收藏

FuxiTranyu:一个使用均衡数据训练的多语言大语言模型

FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Data

College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部)

Haoran Sun, Renren Jin, Shaoyang Xu, Leiyu Pan, Supryadi, Menglong Cui, Jiangcun Du, Yikun Lei, Lei Yang, Ling Shi, Juesi Xiao, Shaolin Zhu, Deyi Xiong

AI总结:

本文提出开源多语言大模型FuxiTranyu,通过覆盖43种自然语言和16种编程语言的6000亿均衡token从头训练,并提供SFT与DPO版本,在多语言基准上表现具有竞争力。

摘要:

大语言模型(LLM)已在广泛的任务中展现出强大能力。然而,许多LLM在高资源语言与低资源语言之间表现出显著的性能差异。为缓解这一挑战,我们提出FuxiTranyu,这是一个开源多语言LLM,旨在满足研究界对均衡且高性能多语言能力的需求。基础模型FuxiTranyu-8B拥有80亿个参数,并在精心均衡的多语言数据上从头开始训练,这些数据包含6000亿个token,覆盖43种自然语言和16种编程语言。我们还开发了两个指令微调模型:FuxiTranyu-8B-SFT在多样化的多语言指令数据集上进行微调;FuxiTranyu-8B-DPO则进一步在偏好数据集上使用DPO进行优化,以增强对齐能力。在一系列广泛的多语言基准测试上的大量实验表明,FuxiTranyu相较现有多语言LLM(例如BLOOM-7B、PolyLM-13B和Mistral-7B-Instruct)具有具有竞争力的性能。神经元与表征可解释性分析均表明,FuxiTranyu在不同语言之间获得了一致的多语言表征。为促进多语言LLM的进一步研究,我们在HuggingFace(见https://huggingface.co/TJUNLP/FuxiTranyu-8B)和Github(见https://github.com/tjunlp-lab/FuxiTranyu)发布基础模型和指令微调FuxiTranyu模型,以及58个预训练检查点。

英文摘要:

Large language models (LLMs) have demonstrated prowess in a wide range of tasks. However, many LLMs exhibit significant performance discrepancies between high- and low-resource languages. To mitigate this challenge, we present FuxiTranyu, an open-source multilingual LLM, which is designed to satisfy the need of the research community for balanced and high-performing multilingual capabilities. The base model, FuxiTranyu-8B, features 8 billion parameters and is trained from scratch on meticulously balanced multilingual data that contains 600 billion tokens covering 43 natural languages and 16 programming languages. We also develop two instruction-tuned models: FuxiTranyu-8B-SFT which is fine-tuned on a diverse multilingual instruction dataset, and FuxiTranyu-8B-DPO which is further refined with DPO on a preference dataset for enhanced alignment ability. Extensive experiments on a wide range of multilingual benchmarks demonstrate the competitive performance of FuxiTranyu against existing multilingual LLMs, e.g., BLOOM-7B, PolyLM-13B, and Mistral-7B-Instruct. Both neuron and representation interpretability analyses reveal that FuxiTranyu achieves consistent multilingual representations across languages. To promote further research into multilingual LLMs, we release both the base and instruction-tuned FuxiTranyu models together with 58 pre-training checkpoints at HuggingFace (see https://huggingface.co/TJUNLP/FuxiTranyu-8B) and Github (see https://github.com/tjunlp-lab/FuxiTranyu).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑