DPRM: 多跳问答任务中的双隐式过程奖励模型
DPRM: A Dual Implicit Process Reward Model in Multi-Hop Question Answering
AI总结：
DPRM通过双隐式过程奖励模型提升多跳问答任务的推理准确性与一致性。
摘要：
在多跳问答（MHQA）任务中，思维链（CoT）通过引导大型语言模型（LLMs）进行多步推理来提高生成质量，而知识图谱（KGs）通过语义匹配减少幻觉。结果奖励模型（ORMs）在生成最终答案后提供反馈，但无法评估多步推理的过程。传统过程奖励模型（PRMs）评估推理过程，但需要昂贵的人工标注或生成回滚。隐式PRM仅通过结果信号进行训练，并通过奖励参数化推导步骤奖励，而无需显式标注，更适合MHQA任务中的多步推理。然而，现有隐式PRM仅适用于纯文本场景。当应用于MHQA任务时，它无法处理KG中的图结构约束，并捕捉到CoT和KG路径之间的潜在不一致。为解决这些限制，我们提出了DPRM（双隐式过程奖励模型）。它为MHQA任务中的CoT和KG推理训练两个隐式PRM。这两个PRM，即KG-PRM和CoT-PRM，通过奖励参数化从结果信号推导出步骤级奖励，而无需额外显式标注。其中，KG-PRM使用偏好对学习KG的结构约束。DPRM进一步引入了CoT和KG推理步骤之间的一致性约束，使两个PRM相互验证并协同优化推理路径。我们还提供了过程奖励推导的理论证明。实验结果表明，我们的方法在多个数据集上优于13个基线方法，最大在Hit@1上提高了16.6%。
英文摘要：
In multi-hop question answering (MHQA) tasks, Chain of Thought (CoT) improves the quality of generation by guiding large language models (LLMs) through multi-step reasoning, and Knowledge Graphs (KGs) reduce hallucinations via semantic matching. Outcome Reward Models (ORMs) provide feedback after generating the final answers but fail to evaluate the process for multi-step reasoning. Traditional Process Reward Models (PRMs) evaluate the reasoning process but require costly human annotations or rollout generation. While implicit PRM is trained only with outcome signals and derives step rewards through reward parameterization without explicit annotations, it is more suitable for multi-step reasoning in MHQA tasks. However, existing implicit PRM has only been explored for plain text scenarios. When adapting to MHQA tasks, it cannot handle the graph structure constraints in KGs and capture the potential inconsistency between CoT and KG paths. To address these limitations, we propose the DPRM (Dual Implicit Process Reward Model). It trains two implicit PRMs for CoT and KG reasoning in MHQA tasks. Both PRMs, namely KG-PRM and CoT-PRM, derive step-level rewards from outcome signals via reward parameterization without additional explicit annotations. Among them, KG-PRM uses preference pairs to learn structural constraints from KGs. DPRM further introduces a consistency constraint between CoT and KG reasoning steps, making the two PRMs mutually verify and collaboratively optimize the reasoning paths. We also provide a theoretical demonstration of the derivation of process rewards. Experimental results show that our method outperforms 13 baselines on multiple datasets with up to 16.6% improvement on Hit@1.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
