AfriqueLLM: 数据混合与模型架构如何影响非洲语言的持续预训练
AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages
AI总结：
本文通过持续预训练26B tokens，在20种非洲语言上构建了AfriqueLLM模型套件，系统研究了数据组成和模型架构对下游性能的影响，发现数据组成是主要驱动因素，且架构选择比模型规模更重要。
摘要：
大型语言模型（LLM）日益多语言化，但开源模型相对于专有系统仍表现不佳，在非洲语言上差距最为显著。持续预训练（CPT）提供了一条实用的语言适应途径，但在数学推理等要求较高的能力上的改进通常仍然有限。这种限制部分源于许多低资源语言语料库的领域覆盖不均和缺乏任务相关知识。我们提出了\ exttt{AfriqueLLM}，一个通过CPT在26B tokens上适应20种非洲语言的开源LLM套件。我们对跨越不同规模和架构的五个基础模型（包括Llama 3.1、Gemma 3和Qwen 3）进行了全面的实证研究，并系统分析了CPT数据组成如何影响下游性能。特别是，我们改变了包含数学、代码和合成翻译数据的混合比例，并在多种多语言基准上评估了所得模型。我们的结果将数据组成确定为CPT收益的主要驱动因素。添加数学、代码和合成翻译数据带来了持续改进，包括在面向推理的评估上。在固定架构内，较大的模型通常能提升性能，但在不同模型家族之间比较时，架构选择主导了规模因素。此外，基础模型强大的多语言性能并不能可靠地预测CPT后的结果；稳健的架构结合任务对齐的数据提供了更可靠的方案。最后，我们最好的模型改善了长上下文性能，包括文档级翻译。模型和代码已在[Huggingface](https://this URL)和[Github](https://this URL)上发布。
英文摘要：
Large language models (LLMs) are increasingly multilingual, yet open models continue to underperform relative to proprietary systems, with the gap most pronounced for African languages. Continued pre-training (CPT) offers a practical route to language adaptation, but improvements on demanding capabilities such as mathematical reasoning often remain limited. This limitation is driven in part by the uneven domain coverage and missing task-relevant knowledge that characterize many low-resource language corpora. We present \texttt{AfriqueLLM}, a suite of open LLMs adapted to 20 African languages through CPT on 26B tokens. We perform a comprehensive empirical study across five base models spanning sizes and architectures, including Llama 3.1, Gemma 3, and Qwen 3, and systematically analyze how CPT data composition shapes downstream performance. In particular, we vary mixtures that include math, code, and synthetic translated data, and evaluate the resulting models on a range of multilingual benchmarks. Our results identify data composition as the primary driver of CPT gains. Adding math, code, and synthetic translated data yields consistent improvements, including on reasoning-oriented evaluations. Within a fixed architecture, larger models typically improve performance, but architectural choices dominate scale when comparing across model families. Moreover, strong multilingual performance in the base model does not reliably predict post-CPT outcomes; robust architectures coupled with task-aligned data provide a more dependable recipe. Finally, our best models improve long-context performance, including document-level translation. Models and code have been released on [Huggingface](https://huggingface.co/collections/McGill-NLP/afriquellm) and [Github](https://github.com/McGill-NLP/AfriqueLLM).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
