Lugha-Llama：面向非洲语言适配大语言模型
Lugha-Llama: Adapting Large Language Models for African Languages
AI总结：
本文提出Lugha-Llama，通过将非洲语言精选数据与高质量英语教育文本混合训练来适配大语言模型，在IrokoBench和AfriQA等基准上显著超越同规模基线模型，并分析了英语数据对性能提升的关键作用。
摘要：
大语言模型（LLMs）在广泛的自然语言应用中取得了令人瞩目的成果。然而，它们往往难以识别低资源语言，尤其是非洲语言，这些语言在大型训练语料库中代表性不足。在本文中，我们考虑如何将大语言模型适配到低资源的非洲语言。我们发现，将来自非洲语言的精选数据与高质量的英语教育文本相结合，可以形成一种训练混合数据，从而显著提升模型在这些语言上的性能。在具有挑战性的IrokoBench数据集上，我们的模型在同等规模的基线模型中持续取得最佳性能，尤其是在知识密集型多项选择题（AfriMMLU）上表现突出。此外，在跨语言问答基准AfriQA上，我们的模型比基础模型高出超过10%。为了更好地理解英语数据在训练过程中的作用，我们将一个包含2亿token的子集翻译成斯瓦希里语，并进行分析，结果表明这些数据的内容是带来强劲性能的主要原因。我们发布了模型和数据，以鼓励未来在非洲语言方面的研究。
英文摘要：
Large language models (LLMs) have achieved impressive results in a wide range of natural language applications. However, they often struggle to recognize low-resource languages, in particular African languages, which are not well represented in large training corpora. In this paper, we consider how to adapt LLMs to low-resource African languages. We find that combining curated data from African languages with high-quality English educational texts results in a training mix that substantially improves the model's performance on these languages. On the challenging IrokoBench dataset, our models consistently achieve the best performance amongst similarly sized baselines, particularly on knowledge-intensive multiple-choice questions (AfriMMLU). Additionally, on the cross-lingual question answering benchmark AfriQA, our models outperform the base model by over 10%. To better understand the role of English data during training, we translate a subset of 200M tokens into Swahili language and perform an analysis which reveals that the content of these data is primarily responsible for the strong performance. We release our models and data to encourage future research on African languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
