arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.11549cs.LGcs.AI
收藏

原生推理模型:训练语言模型在不可验证数据上进行推理

Native Reasoning Models: Training Language Models to Reason on Unverifiable Data

Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学)

Yuanfu Wang, Zhixuan Liu, Xiangtian Li, Chaochao Lu, Chao Yang

AI总结:

本文提出NRT框架,通过让模型自行生成推理轨迹,无需专家示范,提升推理能力,实验证明其在复杂推理任务中表现优异,具有鲁棒性和可扩展性。

摘要:

现有的大规模推理模型训练范式——结合监督微调(SFT)与可验证奖励的强化学习(RLVR)——受到其依赖高质量人工标注推理数据和外部验证者限制。这种依赖导致显著的数据收集成本,风险嵌入人类认知偏见,并将强化学习阶段限制在可客观评估的领域,如数学和编码,使大量不可验证的任务超出其范围。为克服这些限制,我们引入NRT(原生推理训练),一种新的框架,通过让模型仅使用标准问题-答案对生成自己的推理轨迹,培养复杂推理。NRT将训练问题重新定义为潜在变量,采用统一的训练目标,将推理建模为优化问题,内在奖励增加模型生成真实答案的可能性的路径。这种统一视角使我们能够分析先前方法的内在失败模式,如策略崩溃,并系统设计更稳健的奖励聚合函数,创建一个自我强化的反馈循环,使模型学会以解决自身不确定性的方法进行推理。在Llama和Mistral模型家族上的实验证明,NRT在无验证者方法中达到最先进的性能,显著优于标准SFT基线和先前无验证者RL方法。我们的方法在复杂推理领域表现出特别强劲的性能提升,并表现出对策略崩溃的高鲁棒性,提供了一条通用、可扩展的路径,以构建更强大和广泛应用的推理系统。

英文摘要:

The prevailing paradigm for training large reasoning models--combining Supervised Fine-Tuning (SFT) with Reinforcement Learning with Verifiable Rewards (RLVR)--is fundamentally constrained by its reliance on high-quality, human-annotated reasoning data and external verifiers. This dependency incurs significant data-collection costs, risks embedding human cognitive biases, and confines the reinforcement learning stage to objectively assessable domains like mathematics and coding, leaving a wide range of unverifiable tasks beyond its scope. To overcome these limitations, we introduce NRT (Native Reasoning Training), a novel framework that cultivates complex reasoning by having the model generate its own reasoning traces using only standard question-answer pairs, thereby obviating the need for expert-written demonstrations. NRT reframes the training problem by treating the reasoning process as a latent variable. It employs a unified training objective that models reasoning as an optimization problem, intrinsically rewarding paths that increase the model's likelihood of producing the ground-truth answer. This unified perspective allows us to analyze intrinsic failure modes of prior methods, such as policy collapse, and systematically design more robust reward aggregation functions, creating a self-reinforcing feedback loop where the model learns to think in ways that resolve its own uncertainty. Empirical evaluation on Llama and Mistral model families demonstrates that NRT achieves state-of-the-art performance among verifier-free methods, significantly outperforming standard SFT baselines and prior verifier-free RL methods. Our approach yields particularly strong performance gains in complex reasoning domains and exhibits high robustness to policy collapse, offering a general, scalable path toward building more powerful and broadly applicable reasoning systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑