跨语言在线策略自蒸馏用于多语言推理
Crosslingual On-Policy Self-Distillation for Multilingual Reasoning
AI总结：
本文提出COPSD方法，通过将高资源语言的推理行为迁移到低资源语言，提升多语言推理能力，实验显示其在17种低资源非洲语言上表现优异，优于GRPO。
摘要：
大型语言模型（LLMs）在数学推理上取得了显著进展，但这种能力在不同语言间并不均衡。尤其是低资源语言的推理性能较低。为此，我们提出了跨语言在线策略自蒸馏（COPSD），通过将模型自身的高资源推理行为转移到低资源语言。COPSD使用相同的模型作为学生和教师：学生仅看到低资源问题，而教师接收特权的跨语言上下文，包括问题的翻译和英文参考解决方案。训练过程最小化学生自身回放的全分布token级差异，提供密集监督，同时避免仅结果导向强化学习（RL）的稀疏性和不稳定性。在17种低资源非洲语言上的实验表明，COPSD在不同模型规模上一致提升了低资源数学推理能力，并显著优于组相对策略优化（GRPO）。进一步分析显示，COPSD提高了答案格式的符合性，增强了测试时的扩展性，并泛化到更难的多语言推理基准，尤其是对低资源语言有显著提升。我们提供了代码和数据：https://github.com/cisnlp/COPSD。
英文摘要：
Large language models (LLMs) have achieved remarkable progress in mathematical reasoning, but this ability is not equally accessible across languages. Especially low-resource languages exhibit much lower reasoning performance. To address this, we propose Crosslingual On-Policy Self-Distillation (COPSD), which transfers a model's own high-resource reasoning behavior to low-resource languages. COPSD uses the same model as student and teacher: the student sees only the low-resource problem, while the teacher receives privileged crosslingual context, including the problem translation and reference solution in English. Training minimizes full-distribution token-level divergence on the student's own rollouts, providing dense supervision while avoiding the sparsity and instability of outcome-only reinforcement learning (RL). Experiments on 17 low-resource African languages show that COPSD consistently improves low-resource mathematical reasoning across model sizes and substantially outperforms Group Relative Policy Optimization (GRPO). Further analyses show that COPSD improves answer-format adherence, strengthens test-time scaling, and generalizes to harder multilingual reasoning benchmarks, with especially large gains for lower-resource languages. We make our code and data available at: https://github.com/cisnlp/COPSD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
