arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.02479cs.AI
收藏

PRISM:通过过程奖励模型引导推理推动深度思考的前沿

PRISM: Pushing the Frontier of Deep Think via Process Reward Model-Guided Inference

Virginia Tech(弗吉尼亚理工大学)

Rituraj Sharma, Weiyuan Chen, Noah Provenzano, Tu Vu

AI总结:

PRISM通过过程奖励模型引导推理,提升深度思考性能,在数学和科学任务中表现优异。

摘要:

深度思考(DEEPTHINK)方法通过生成、细化和聚合候选解决方案的群体来提高推理能力,从而在复杂的数学和科学任务上表现出色。然而,现有框架在推理过程中往往缺乏可靠的正确性信号,这导致了群体增强的瓶颈,更深入的思考会放大错误,压制正确的少数解决方案,并产生额外计算的弱回报。在本文中,我们介绍了深度思考系统的方法分解,并提出了PRISM,一种由过程奖励模型(PRM)引导的推理算法,利用步骤级验证来指导群体细化和解决方案聚合。在细化过程中,PRISM将候选解决方案视为在PRM定义的能量景观中的粒子,并通过分数引导的重采样和随机细化来重塑群体,这将概率质量集中在高质量推理上,同时保持多样性。在数学和科学基准测试中,PRISM与现有深度思考方法竞争或表现更优,在AIME25、HMMT25和GPQA Diamond上分别达到90.0%、75.4%和71.4%的准确率,同时在gpt-oss-120b上匹配或超越其表现。此外,我们的分析显示,PRISM在细化过程中产生一致的净方向修正,即使初始群体中包含少数正确候选,也能保持可靠性,并且经常位于计算-准确性帕累托前沿。

英文摘要:

DEEPTHINK methods improve reasoning by generating, refining, and aggregating populations of candidate solutions, which enables strong performance on complex mathematical and scientific tasks. However, existing frameworks can still suffer from unreliable correctness signals during inference, which creates a population- enhancement bottleneck where deeper deliberation amplifies errors, suppresses correct minority solutions, and yields weak returns to additional compute. In this paper, we introduce a functional decomposition of DEEPTHINK systems and propose PRISM, a Process Reward Model (PRM)-guided inference algorithm that uses step-level verification to guide resampling, proposal acceptance, and solution aggregation. During refinement, PRISM uses PRM scores to guide population resampling and stochastic local refinement, which concentrates computation on higher-quality reasoning while preserving diversity. When paired with PRM- score-vote, a PRM-guided aggregation method, PRISM is competitive with or outperforms existing DEEPTHINK methods, reaching 90.0%, 75.4%, and 71.4% with gpt-oss-20b on AIME25, HMMT25, and GPQA Diamond, respectively, while exceeding gpt-oss-120b. Additionally, our analysis shows that PRISM produces consistent net-directional correction during refinement and maintains competitive accuracy when the initial population contains few correct candidates.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑