arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.04588cs.CL
收藏

UltraLink:一个开源的知识增强多语言监督微调数据集

UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset

Tsinghua University(清华大学);Beijing Language and Culture University(北京语言大学);Northeastern University(东北大学);Beijing University of Posts and Telecommunications(北京邮电大学)

Haoyu Wang, Shuo Wang, Yukun Yan, Xujia Wang, Zhiyu Yang, Yuzhuang Xu, Zhenghao Liu, Liner Yang, Ning Ding, Xu Han, Zhiyuan Liu, Maosong Sun

AI总结:

本文构建了开源多语言监督微调数据集UltraLink,通过知识增强方法提升LLMs的语言特定能力,并利用跨语言迁移能力精简语言无关数据,使基于其训练的UltraLink-LM在多任务上优于基线。

摘要:

开源大型语言模型(LLMs)在各个领域获得了显著的能力。然而,大多数研究主要集中在英语上,对多语言能力的探索十分有限。因此,在本研究中,我们构建了一个开源的多语言监督微调数据集。与以往简单翻译英语指令的工作不同,我们同时考虑了LLMs的语言特定能力和语言无关能力。首先,我们引入了一种基于知识的数据增强方法,以激发LLMs更多的语言特定知识,提升其为来自不同国家的用户服务的能力。此外,我们发现现代LLMs具备强大的跨语言迁移能力,因此无需用不同语言重复学习相同内容。由此,我们可以在不降低任何性能的情况下大幅精简语言无关的监督微调(SFT)数据,使多语言SFT更加高效。最终生成的UltraLink数据集包含跨五种语言(即En、Zh、Ru、Fr、Es)的约100万个样本,且所提出的数据构建方法可轻松扩展至其他语言。基于UltraLink训练的UltraLink-LM在许多任务上优于多个代表性基线。

英文摘要:

Open-source large language models (LLMs) have gained significant strength across diverse fields. Nevertheless, the majority of studies primarily concentrate on English, with only limited exploration into the realm of multilingual abilities. In this work, we therefore construct an open-source multilingual supervised fine-tuning dataset. Different from previous works that simply translate English instructions, we consider both the language-specific and language-agnostic abilities of LLMs. Firstly, we introduce a knowledge-grounded data augmentation approach to elicit more language-specific knowledge of LLMs, improving their ability to serve users from different countries. Moreover, we find modern LLMs possess strong cross-lingual transfer capabilities, thus repeatedly learning identical content in various languages is not necessary. Consequently, we can substantially prune the language-agnostic supervised fine-tuning (SFT) data without any performance degradation, making multilingual SFT more efficient. The resulting UltraLink dataset comprises approximately 1 million samples across five languages (i.e., En, Zh, Ru, Fr, Es), and the proposed data construction method can be easily extended to other languages. UltraLink-LM, which is trained on UltraLink, outperforms several representative baselines across many tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑