ReasonFlux-PRM：面向大语言模型长思维链推理的轨迹感知过程奖励模型
ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs
AI总结：
提出轨迹感知过程奖励模型ReasonFlux-PRM，融合步骤级与轨迹级监督，在离线/在线设置下用于数据筛选、强化学习及测试时扩展，在AIME等基准上显著提升性能。
摘要：
过程奖励模型（PRMs）近来已成为监督大型语言模型（LLMs）中间推理步骤的强大框架。以往的PRMs主要基于模型最终输出响应进行训练，难以稳健地评估中间思考轨迹，尤其是在前沿推理模型（如Deepseek-R1）生成的轨迹-响应输出这一新兴场景中。在本工作中，我们提出了ReasonFlux-PRM，一种新颖的轨迹感知PRM，专门设计用于评估轨迹-响应类型的推理痕迹。ReasonFlux-PRM同时整合了步骤级和轨迹级监督，实现了与结构化思维链数据对齐的细粒度奖励分配。我们使ReasonFlux-PRM支持离线和在线两种设置下的奖励监督，包括：（i）为下游较小模型的监督微调选择高质量模型蒸馏数据；（ii）在强化学习期间为策略优化提供密集的过程级奖励；（iii）实现奖励引导的最佳N（Best-of-N）测试时扩展。在AIME、MATH500和GPQA-Diamond等具有挑战性的下游基准上的实证结果表明，ReasonFlux-PRM-7B选择的数据质量优于强PRMs（如Qwen2.5-Math-PRM-72B）和人工精选基线。此外，我们推导出的ReasonFlux-PRM-7B带来了持续的性能提升，在监督微调中平均提升12.1%，在强化学习中平均提升4.5%，在测试时扩展中平均提升6.3%。我们还发布了高效的ReasonFlux-PRM-1.5B，用于资源受限的应用和边缘部署。项目链接：https://github.com/Gen-Verse/ReasonFlux
英文摘要：
Process Reward Models (PRMs) have recently emerged as a powerful framework for supervising intermediate reasoning steps in large language models (LLMs). Previous PRMs are primarily trained on model final output responses and struggle to evaluate intermediate thinking trajectories robustly, especially in the emerging setting of trajectory-response outputs generated by frontier reasoning models like Deepseek-R1. In this work, we introduce ReasonFlux-PRM, a novel trajectory-aware PRM explicitly designed to evaluate the trajectory-response type of reasoning traces. ReasonFlux-PRM incorporates both step-level and trajectory-level supervision, enabling fine-grained reward assignment aligned with structured chain-of-thought data. We adapt ReasonFlux-PRM to support reward supervision under both offline and online settings, including (i) selecting high-quality model distillation data for downstream supervised fine-tuning of smaller models, (ii) providing dense process-level rewards for policy optimization during reinforcement learning, and (iii) enabling reward-guided Best-of-N test-time scaling. Empirical results on challenging downstream benchmarks such as AIME, MATH500, and GPQA-Diamond demonstrate that ReasonFlux-PRM-7B selects higher quality data than strong PRMs (e.g., Qwen2.5-Math-PRM-72B) and human-curated baselines. Furthermore, our derived ReasonFlux-PRM-7B yields consistent performance improvements, achieving average gains of 12.1% in supervised fine-tuning, 4.5% in reinforcement learning, and 6.3% in test-time scaling. We also release our efficient ReasonFlux-PRM-1.5B for resource-constrained applications and edge deployment. Project: https://github.com/Gen-Verse/ReasonFlux
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
