面向大语言模型多语言迁移的语码转换课程学习
Code-Switching Curriculum Learning for Multilingual Transfer in LLMs
AI总结：
本文提出语码转换课程学习CSCL，通过token级、句子级语码转换到单语语料的渐进训练提升LLM跨语言迁移，并在多语言、多模型实验中验证其增益与公平性。
摘要：
大语言模型（LLM）如今在多种任务上表现出接近人类水平的性能，但由于预训练数据不均衡，在少数高资源语言之外，其性能会急剧下降。受人类第二语言习得过程，尤其是语码转换（即在对话中交替使用不同语言的做法）启发，我们提出语码转换课程学习（CSCL），以增强大语言模型的跨语言迁移。CSCL模仿人类语言学习的阶段，通过包含以下三部分的课程渐进式训练模型：1）token级语码转换，2）句子级语码转换，以及3）单语语料。我们以Qwen 2作为基础模型，证明了CSCL在提升向韩语的语言迁移方面的有效性；与单语持续预训练方法相比，CSCL取得了显著的性能提升。消融研究表明，token级和句子级语码转换都能显著增强跨语言迁移，而课程学习会放大这些效果。我们还将研究发现扩展到多种语言，包括日语（高资源）和印尼语（低资源），并使用另外两个模型（Gemma 2和Phi 3.5）进行验证。我们进一步表明，CSCL缓解了语言资源与安全对齐之间的虚假相关性，为大语言模型中更公平的语言迁移提供了一个稳健、高效的框架。我们观察到，在难以获得用于语言迁移的高质量单语语料的低资源场景中，CSCL是有效的。
英文摘要：
Large language models (LLMs) now exhibit near human-level performance in various tasks, but their performance drops drastically after a handful of high-resource languages due to the imbalance in pre-training data. Inspired by the human process of second language acquisition, particularly code-switching$\unicode{x2014}$the practice of language alternation in a conversation$\unicode{x2014}$we propose code-switching curriculum learning (CSCL) to enhance cross-lingual transfer for LLMs. CSCL mimics the stages of human language learning by progressively training models with a curriculum consisting of 1) token-level code-switching, 2) sentence-level code-switching, and 3) monolingual corpora. Using Qwen 2 as our underlying model, we demonstrate the efficacy of the CSCL in improving language transfer to Korean, achieving significant performance gains compared to monolingual continual pre-training methods. Ablation studies reveal that both token- and sentence-level code-switching significantly enhance cross-lingual transfer and that curriculum learning amplifies these effects. We also extend our findings into various languages, including Japanese (high-resource) and Indonesian (low-resource), and using two additional models (Gemma 2 and Phi 3.5). We further show that CSCL mitigates spurious correlations between language resources and safety alignment, presenting a robust, efficient framework for more equitable language transfer in LLMs. We observe that CSCL is effective for low-resource settings where high-quality, monolingual corpora for language transfer are hardly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
