EMMA-500: 提升大规模多语言适应性的大语言模型
EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models
AI总结：
EMMA-500通过持续预训练提升多语言性能，尤其改善低资源语言的覆盖与跨语言迁移能力。
摘要：
在本工作中，我们介绍了EMMA-500，这是一个在546种语言文本上继续训练的大型多语言语言模型，旨在提升多语言性能，重点在于改善低资源语言的语言覆盖。为了促进持续预训练，我们编纂了MaLA语料库，一个包含跨领域精选数据集的综合性多语言数据集。利用该语料库，我们对Llama 2 7B模型进行了广泛的持续预训练，生成了EMMA-500，该模型在广泛的任务基准上表现出稳健的性能，包括全面的多语言任务集。我们的结果表明，持续预训练在扩展大型语言模型的语言能力方面是有效的，特别是对于代表性不足的语言，展示了在跨语言迁移、任务泛化和语言适应性方面的显著提升。我们发布了MaLA语料库、EMMA-500模型权重、脚本和模型生成。
英文摘要：
In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, focusing on improving language coverage for low-resource languages. To facilitate continual pre-training, we compile the MaLA corpus, a comprehensive multilingual dataset enriched with curated datasets across diverse domains. Leveraging this corpus, we conduct extensive continual pre-training of the Llama 2 7B model, resulting in EMMA-500, which demonstrates robust performance across a wide collection of benchmarks, including a comprehensive set of multilingual tasks. Our results highlight the effectiveness of continual pre-training in expanding large language models' language capacity, particularly for underrepresented languages, demonstrating significant gains in cross-lingual transfer, task generalization, and language adaptability. We release the MaLA corpus, EMMA-500 model weights, scripts, and model generations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
