从非对齐到对齐：利用多路平行语料库扩展多语言大语言模型
From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora
AI总结：
本研究构建了覆盖113种语言的高质量多路平行语料库TED2025，探索用其增强多语言LLM的最佳实践，实验证明该方法相比非对齐数据训练能显著提升模型性能。
摘要：
在大规模多语言数据上进行持续预训练和指令微调已被证明可有效将大语言模型（LLM）扩展至低资源语言。然而，此类数据的非对齐特性限制了其有效捕捉跨语言语义的能力。相比之下，多路平行数据（即相同内容在多种语言间对齐）能提供更强的跨语言一致性，在提升多语言性能方面具有更大潜力。本文提出了一个基于TED Talks的大规模高质量多路平行语料库TED2025。该语料库覆盖113种语言，最多支持50种语言的平行对齐，确保了广泛的多语言覆盖范围。利用该数据集，我们研究了利用多路平行数据增强LLM的最佳实践，包括持续预训练策略、指令微调以及关键影响因素分析。在六个多语言基准上的实验表明，使用多路平行数据训练的模型始终优于使用非对齐多语言数据训练的模型。
英文摘要：
Continued pretraining and instruction tuning on large-scale multilingual data have proven to be effective in scaling large language models (LLMs) to low-resource languages. However, the unaligned nature of such data limits its ability to effectively capture cross-lingual semantics. In contrast, multi-way parallel data, where identical content is aligned across multiple languages, provides stronger cross-lingual consistency and offers greater potential for improving multilingual performance. In this paper, we introduce a large-scale, high-quality multi-way parallel corpus, TED2025, based on TED Talks. The corpus spans 113 languages, with up to 50 languages aligned in parallel, ensuring extensive multilingual coverage. Using this dataset, we investigate best practices for leveraging multi-way parallel data to enhance LLMs, including strategies for continued pretraining, instruction tuning, and the analysis of key influencing factors. Experiments on six multilingual benchmarks show that models trained on multiway parallel data consistently outperform those trained on unaligned multilingual data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
