混合语言文档在多语言大语言模型预训练中的作用
The Role of Mixed-Language Documents for Multilingual Large Language Model Pretraining
AI总结：
研究发现平行数据对翻译性能至关重要，而语码转换数据对跨语言理解和推理影响较小。
摘要：
多语言大语言模型在预训练过程中尽管主要使用单语数据，却仍能实现出色的跨语言性能。虽然双语数据在预训练语料中被广泛认为能够实现这些能力，但其具体贡献仍不明确。我们通过在受控条件下从头开始预训练模型来研究这一问题，将标准网络语料与去除所有多语言文档的单语版本进行比较。尽管只占语料的2%，去除双语数据导致翻译性能在BLEU评分上下降56%，而在跨语言问答和一般推理任务上的表现保持稳定，训练曲线与基线大致重合。为了理解这种不对称性，我们根据不同语言中内容的语义相关性将双语数据分为平行（14%）、语码转换（72%）和杂项文档（14%）。然后我们通过将平行或语码转换数据重新引入单语语料中进行细致的消融实验。我们的实验发现，平行数据几乎完全恢复了翻译性能（91%的未过滤基线），而语码转换贡献很小。其他跨语言任务对这两种类型的影响都较小。这些发现表明，翻译严重依赖于平行数据中的系统性词级对齐，而跨语言理解和推理似乎即使在没有双语数据的情况下也能实现。
英文摘要：
Multilingual large language models achieve impressive cross-lingual performance despite largely monolingual pretraining. While bilingual data in pretraining corpora is widely believed to enable these abilities, details of its contributions remain unclear. We investigate this question by pretraining models from scratch under controlled conditions, comparing the standard web corpus with a monolingual-only version that removes all multilingual documents. Despite constituting only 2% of the corpus, removing bilingual data causes translation performance to drop 56% in BLEU, while behaviour on cross-lingual QA and general reasoning tasks remains stable, with training curves largely overlapping the baseline. To understand this asymmetry, we categorize bilingual data into parallel (14%), code-switching (72%), and miscellaneous documents (14%) based on the semantic relevance of content in different languages. We then conduct granular ablations by reintroducing parallel or code-switching data into the monolingual-only corpus. Our experiments reveal that parallel data almost fully restores translation performance (91% of the unfiltered baseline), whereas code-switching contributes minimally. Other cross-lingual tasks remain largely unaffected by either type. These findings reveal that translation critically depends on systematic token-level alignments from parallel data, whereas cross-lingual understanding and reasoning appear to be achievable even without bilingual data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
