AdaMame: 一种自适应多语言推理的训练方案
AdaMame: A Training Recipe for Adaptive Multilingual Reasoning
AI总结：
针对多语言推理中的语言崩溃问题，提出两阶段训练方案AdaMame，通过SFT建立多语言能力，再以自适应GRPO优化推理语言对齐，在准确率、语言保真度和令牌效率上达到帕累托最优。
摘要：
尽管大型推理模型（LRMs）在英语中表现出色，但它们往往无法以查询语言进行推理，这种现象称为语言崩溃。现有的基于强化学习的修复方法通常在准确性目标上添加一个二元语言保真度奖励，但仍然会在准确性、中间轨迹代码切换和过度令牌使用方面产生权衡。在这项工作中，我们提出了AdaMame，一种用于多语言数学推理的两阶段训练方案，通过自适应地将推理语言与查询语言对齐来解决这些限制，同时不损害准确性。第一阶段的SFT在五种语言的自然推理轨迹上进行微调，以建立多语言推理能力。在随后的RL阶段，我们引入了AdaMame-GRPO，这是组相对策略优化（GRPO）的一种改编，其中查询条件的对齐因子在训练过程中逐渐增长，引导模型首先探索多样的推理语言，然后利用查询语言进行推理。在两个基准、两个LRM和12种语言上的评估表明，AdaMame-GRPO在所有基线上实现了推理准确性、语言保真度和令牌效率的帕累托最优性能，在领域外、低资源语言上取得了最强的提升。
英文摘要：
While Large Reasoning Models (LRMs) show strong performance in English, they often fail to reason in the language of the query, a phenomenon known as language collapse. Existing RL-based fixes typically add a binary language fidelity reward to the accuracy objective, yet still incur trade-off in accuracy, mid-trace code-switching, and excessive token usage. In this work, we propose AdaMame, a two-stage training recipe for multilingual mathematical reasoning that addresses these limitations by adaptively aligning the reasoning language to the query language without compromising accuracy. The first SFT stage fine-tunes on non-MT reasoning traces across five languages to establish multilingual reasoning capability. In the subsequent RL stage, we introduce AdaMame-GRPO, an adaptation of Group Relative Policy Optimization (GRPO) in which a query-conditioned alignment factor grows progressively during training, guiding the model to first explore diverse reasoning languages before exploiting reasoning in the query language. Evaluated across two benchmarks, two LRMs, and 12 languages, AdaMame-GRPO achieves Pareto-optimal performance across reasoning accuracy, language fidelity, and token efficiency over all baselines, with the strongest gains on out-of-domain, lower-resource languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
