探索多语言和谐：关于大语言模型预训练中多语言数据分配的研究
Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining
AI总结：
本文提出 Climb 框架，通过跨语言交互感知的语言比例和两步优化方法，系统优化大语言模型预训练中的多语言数据分配，显著提升多语言性能并达到领先水平。
摘要：
大语言模型（LLMs）已成为全球广泛应用的核心，推动了全球对高效多语言能力前所未有的需求。实现稳健多语言性能的关键在于训练语料库中语言比例的合理分配。然而，由于复杂的跨语言交互和对数据集规模的敏感性，确定最优语言比例极具挑战性。本文提出 Climb（Cross-Lingual Interaction-aware Multilingual Balancing，跨语言交互感知的多语言均衡框架），这是一个旨在系统优化多语言数据分配的新框架。Climb 的核心是引入了一种跨语言交互感知的语言比例，通过捕捉语言间的依赖关系，明确量化每种语言的有效分配。基于该比例，Climb 提出了一种有原则的两步优化流程——首先均衡各语言的边际收益，然后最大化所得语言分配向量的规模——显著简化了本质上复杂的多语言优化问题。大量实验证实，Climb 能够在多种多语言设置下准确测量跨语言交互。使用 Climb 推导的比例训练的 LLMs 持续取得最先进的多语言性能，甚至能与使用更多 token 训练的开源 LLMs 达到具有竞争力的性能。
英文摘要：
Large language models (LLMs) have become integral to a wide range of applications worldwide, driving an unprecedented global demand for effective multilingual capabilities. Central to achieving robust multilingual performance is the strategic allocation of language proportions within training corpora. However, determining optimal language ratios is highly challenging due to intricate cross-lingual interactions and sensitivity to dataset scale. This paper introduces Climb (Cross-Lingual Interaction-aware Multilingual Balancing), a novel framework designed to systematically optimize multilingual data allocation. At its core, Climb introduces a cross-lingual interaction-aware language ratio, explicitly quantifying each language's effective allocation by capturing inter-language dependencies. Leveraging this ratio, Climb proposes a principled two-step optimization procedure--first equalizing marginal benefits across languages, then maximizing the magnitude of the resulting language allocation vectors--significantly simplifying the inherently complex multilingual optimization problem. Extensive experiments confirm that Climb can accurately measure cross-lingual interactions across various multilingual settings. LLMs trained with Climb-derived proportions consistently achieve state-of-the-art multilingual performance, even achieving competitive performance with open-sourced LLMs trained with more tokens.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
