打破语言障碍：大规模跨语言持续预训练
Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale
AI总结：
该研究通过 40M 至 5B 参数的 40 组模型实验，分析跨语言持续预训练的收敛、缩放律、最优资源分配及迁移影响因素。
摘要：
近年来，大语言模型（LLM）在迈向通用人工智能方面取得了重大进展。然而，从头训练这些模型需要大量计算资源和海量文本数据。本文探索了一种为新语言构建 LLM 的替代方法：从现有预训练 LLM 进行持续预训练（CPT），而不是使用随机初始化的参数。基于对参数量从 40M 到 5B 的 40 种模型规模开展的平行实验，我们发现：1）CPT 收敛更快，并以可扩展的方式节省大量资源；2）CPT 遵循由 Hoffmann 等人（2022）推导的扩展缩放律，该定律包含一个联合数据-参数缩放项；3）根据我们估计的缩放因子，CPT 的计算最优数据-参数分配存在显著差异；4）大规模迁移的有效性受训练时长和语言属性影响，但对数据回放具有鲁棒性；数据回放是一种有效缓解 CPT 中灾难性遗忘的方法。我们希望这些发现能为研究界深入理解大规模 LLM 的可迁移性提供参考。
英文摘要：
In recent years, Large Language Models (LLMs) have made significant strides towards Artificial General Intelligence. However, training these models from scratch requires substantial computational resources and vast amounts of text data. In this paper, we explore an alternative approach to constructing an LLM for a new language by continually pretraining (CPT) from existing pretrained LLMs, instead of using randomly initialized parameters. Based on parallel experiments on 40 model sizes ranging from 40M to 5B parameters, we find that 1) CPT converges faster and saves significant resources in a scalable manner; 2) CPT adheres to an extended scaling law derived from Hoffmann et al. (2022) with a joint data-parameter scaling term; 3) The compute-optimal data-parameter allocation for CPT markedly differs based on our estimated scaling factors; 4) The effectiveness of transfer at scale is influenced by training duration and linguistic properties, while robust to data replaying, a method that effectively mitigates catastrophic forgetting in CPT. We hope our findings provide deeper insights into the transferability of LLMs at scale for the research community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
