发表机构
Nanjing University（南京大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对多语言推理迁移中现有在线策略自蒸馏未优先考虑关键推理信号的问题，提出 RP-OPSD 方法，利用教师视图分布偏移引导特权蒸馏，在 17 种语言的数学推理基准上优于基线，提升了多语言推理能力。
AI 中文摘要
多语言推理迁移对于将大型语言模型（LLM）的推理能力扩展到高资源语言之外至关重要。在线策略自蒸馏（OPSD）及其变体已成为一种有前景的范式，可为学生生成的 rollout 提供密集的 token 级监督，然而它们的目标并未明确优先考虑对跨语言迁移最关键的推理信号。我们指出，目标语言推理包括表层文本和推理枢轴的生成，推理枢轴是推进或重定向推理过程并塑造后续推理的决策。这促使我们将特权蒸馏集中在这类枢轴周围。因此，我们提出 RP-OPSD，即基于推理枢轴引导的在线策略自蒸馏，利用带有和不带有英语参考解的匹配教师视图之间的分布偏移作为操作代理，以引导特权蒸馏和参考锚定。在涵盖 17 种语言和多个难度级别的数学推理基准上进行的实验表明，我们的方法优于强大的多语言推理基线和 OPSD 变体。进一步分析显示，RP-OPSD 将特权蒸馏集中在推理控制和问题条件状态更新 token 上，同时降低了主要支持表层实现的 token 的权重。我们的代码可在此 https URL 获取。
英文摘要
Multilingual reasoning transfer is crucial for extending reasoning capabilities of large language models (LLMs) beyond high-resource languages. On-policy self-distillation (OPSD) and its variants have emerged as a promising paradigm, providing dense token-level supervision on student-generated rollouts, yet their objectives do not explicitly prioritize reasoning signals most critical to cross-lingual transfer. We characterize that target-language reasoning comprises the generation of both surface text and reasoning pivots, which are decisions that advance or redirect the reasoning process and shape subsequent inference. This motivates concentrating privileged distillation around such pivots. We therefore propose RP-OPSD, Reasoning-Pivot-guided On-Policy Self-Distillation, using the distributional shift between matched teacher views with and without an English reference solution as an operational proxy to guide privileged distillation and reference anchoring. Experiments on mathematical reasoning benchmarks covering 17 languages and multiple difficulty levels show that our method outperforms strong multilingual reasoning baselines and OPSD variants. Further analysis reveals that RP-OPSD concentrates privileged distillation on reasoning-control and problem-condistioned state-update tokens, while downweighting it for tokens that mainly support surface realization. Our code is available at https://github.com/NJUNLP/RP-OPSD.
Comments16 pages. Under review
