原生思考：通过一致性增强强化学习解锁多语言推理
Think Natively: Unlocking Multilingual Reasoning with Consistency-Enhanced Reinforcement Learning
AI总结：
M-Thinker通过一致性增强强化学习提升多语言推理能力，实现高语言一致性和多语言基准优异表现。
摘要：
大型推理模型（LRMs）通过采用"思考-然后回答"范式，在复杂推理任务上取得了显著成效，从而提升了准确性和可解释性。然而，当前LRMs在处理非英语语言时存在两个关键限制：（1）它们往往难以维持输入-输出语言一致性；（2）它们在错误推理路径上表现较差，与英语相比答案准确率较低。这些限制严重削弱了推理过程的可解释性，并降低了非英语使用者的用户体验，阻碍了LRMs的全球部署。为了解决这些限制，我们提出了M-Thinker，其通过GRPO算法进行训练，该算法包含语言一致性（LC）奖励和一种新的跨语言思考对齐（CTA）奖励。具体而言，LC奖励对输入、思考和答案之间的语言一致性施加严格约束。此外，CTA奖励将模型的非英语推理路径与其英语推理路径进行比较，以将自身的推理能力从英语转移到非英语语言。通过迭代强化学习过程，我们的M-Thinker-1.5B/4B/7B模型不仅实现了接近100%的语言一致性和在两个多语言基准（MMATH和PolyMath）上的优异表现，还表现出在非领域语言上的出色泛化能力。
英文摘要：
Large Reasoning Models (LRMs) have achieved remarkable performance on complex reasoning tasks by adopting the ``think-then-answer'' paradigm, which enhances both accuracy and interpretability. However, current LRMs exhibit two critical limitations when processing non-English languages: (1) They often struggle to maintain input-output language consistency; (2) They generally perform poorly with wrong reasoning paths and lower answer accuracy compared to English. These limitations significantly compromise the interpretability of reasoning processes and degrade the user experience for non-English speakers, hindering the global deployment of LRMs. To address these limitations, we propose M-Thinker, which is trained by the GRPO algorithm that involves a Language Consistency (LC) reward and a novel Cross-lingual Thinking Alignment (CTA) reward. Specifically, the LC reward defines a strict constraint on the language consistency between the input, thought, and answer. Besides, the CTA reward compares the model's non-English reasoning paths with its English reasoning path to transfer its own reasoning capability from English to non-English languages. Through an iterative RL procedure, our M-Thinker-1.5B/4B/7B models not only achieve nearly 100% language consistency and superior performance on two multilingual benchmarks (MMATH and PolyMath), but also exhibit excellent generalization on out-of-domain languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
