超越结果验证：用于结构化推理的可验证过程奖励模型
Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning
AI总结：
本文提出可验证过程奖励模型，用于提升结构化推理的连贯性和准确性，实验显示其在医学证据评估中优于现有方法。
摘要：
最近关于具有可验证奖励的强化学习（RLVR）的工作表明，使用结果层面的验证信号，如代码的单元测试或数学的精确匹配检查，可以显著提高大型语言模型（LLMs）的性能。同时，过程监督长期以来被探索为一种塑造LLMs中间推理行为的方法，但现有方法依赖于神经判断师来评分思维链步骤，使它们容易受到不透明性、偏见和奖励黑客的威胁。为了解决这一差距，我们引入了可验证过程奖励模型（VPRMs），这是一种强化学习框架，在其中中间推理步骤由确定性的、基于规则的验证器进行检查。我们将VPRMs应用于医学证据综合的风险偏倚评估，这是一个领域，其中指南定义的准则和基于规则的决策路径使程序化验证推理轨迹成为可能。在多个数据集中，我们发现VPRMs生成的推理紧密遵循领域规则，并且在步骤级决策与最终标签之间实现了显著更高的连贯性。结果表明，VPRMs在F1指标上比最先进的模型高出20%，比可验证结果奖励高出6.5%，在证据基础和逻辑连贯性方面也有显著提升。
英文摘要：
Recent work on reinforcement learning with verifiable rewards (RLVR) has shown that large language models (LLMs) can be substantially improved using outcome-level verification signals, such as unit tests for code or exact-match checks for mathematics. In parallel, process supervision has long been explored as a way to shape the intermediate reasoning behaviour of LLMs, but existing approaches rely on neural judges to score chain-of-thought steps, leaving them vulnerable to opacity, bias, and reward hacking. To address this gap, we introduce Verifiable Process Reward Models (VPRMs), a reinforcement-learning framework in which intermediate reasoning steps are checked by deterministic, rule-based verifiers. We apply VPRMs to risk-of-bias assessment for medical evidence synthesis, a domain where guideline-defined criteria and rule-based decision paths enable programmatic verification of reasoning traces. Across multiple datasets, we find that VPRMs generate reasoning that adheres closely to domain rules and achieve substantially higher coherence between step-level decisions and final labels. Results show that VPRMs achieve up to 20% higher F1 than state-of-the-art models and 6.5% higher than verifiable outcome rewards, with substantial gains in evidence grounding and logical coherence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
