PolyLM：一个开源多语言大语言模型
PolyLM: An Open Source Polyglot Large Language Model
AI总结：
本文提出开源多语言大语言模型PolyLM，通过双语数据、课程学习和多语言自指令微调提升非英语能力，并在多语言任务上优于LLaMA、BLOOM等开源模型。
摘要：
大语言模型（LLM）在遵循自然语言指令进行理解、推理和生成方面展现出卓越能力。然而，LLM的开发主要集中于英语等高资源语言，从而限制了其在其他语言中的适用性与研究。因此，我们提出PolyLM，这是一个在6400亿（B）个token上训练的多语言LLM，提供1.7B和13B两种模型规模。为增强其多语言能力，我们：1）将双语数据整合到训练数据中；2）采用课程学习策略，在预训练过程中将非英语数据的比例从第一阶段的30%提高到最终阶段的60%。此外，我们提出一种多语言自指令方法，自动生成132.7K条多样化的多语言指令用于模型微调。为评估模型性能，我们汇集了若干现有多语言任务，包括多语言理解、问答、生成和翻译。大量实验表明，PolyLM在多语言任务上超过LLaMA和BLOOM等其他开源模型，同时在英语上保持相当的性能。我们的模型以及指令数据和多语言基准可在以下网址获取：\url{https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation}。
英文摘要：
Large language models (LLMs) demonstrate remarkable ability to comprehend, reason, and generate following nature language instructions. However, the development of LLMs has been primarily focused on high-resource languages, such as English, thereby limiting their applicability and research in other languages. Consequently, we present PolyLM, a multilingual LLM trained on 640 billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its multilingual capabilities, we 1) integrate bilingual data into training data; and 2) adopt a curriculum learning strategy that increases the proportion of non-English data from 30% in the first stage to 60% in the final stage during pre-training. Further, we propose a multilingual self-instruct method which automatically generates 132.7K diverse multilingual instructions for model fine-tuning. To assess the model's performance, we collect several existing multilingual tasks, including multilingual understanding, question answering, generation, and translation. Extensive experiments show that PolyLM surpasses other open-source models such as LLaMA and BLOOM on multilingual tasks while maintaining comparable performance in English. Our models, alone with the instruction data and multilingual benchmark, are available at: \url{https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
