后训练者的多语言训练数据指南：揭示跨语言迁移动态
A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics
AI总结：
本研究通过控制多语言数据混合，在后训练中考察跨语言迁移动态，发现其受多种设置组合影响，并识别出有效迁移的条件。
摘要：
为了使大型语言模型在全球范围内发挥作用，它们会在多语言数据上进行微调以遵循指令。尽管此类后训练普遍存在，但对实现跨语言迁移的动态机制仍缺乏清晰理解。本研究考察了现实后训练设置中的跨语言迁移（CLT）动态。我们研究了两个规模高达35B参数的模型家族，这些模型在三种复杂度不同的生成任务（摘要生成、指令遵循和数学推理）上，于单任务和多任务指令调优设置中，使用精心控制的多语言数据混合进行训练。总体而言，我们发现跨语言迁移和多语言性能的动态无法用孤立变量解释，而是随后训练设置的组合而变化。最后，我们识别了在实践中实现有效跨语言迁移的条件。
英文摘要：
In order for large language models to be useful across the globe, they are fine-tuned to follow instructions on multilingual data. Despite the ubiquity of such post-training, a clear understanding of the dynamics that enable cross-lingual transfer remains elusive. This study examines cross-lingual transfer (CLT) dynamics in realistic post-training settings. We study two model families of up to 35B parameters in size trained on carefully controlled mixtures of multilingual data on three generative tasks with varying levels of complexity (summarization, instruction following, and mathematical reasoning) in both single-task and multi-task instruction tuning settings. Overall, we find that the dynamics of cross-lingual transfer and multilingual performance cannot be explained by isolated variables, varying depending on the combination of post-training settings. Finally, we identify the conditions that lead to effective cross-lingual transfer in practice.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
