DC-W2S：双共识弱到强训练用于生物推理中的可靠过程奖励建模
DC-W2S: Dual-Consensus Weak-to-Strong Training for Reliable Process Reward Modeling in Biological Reasoning
AI总结：
DC-W2S通过双共识机制和课程学习方法，在无需专家标注的情况下训练出可靠的生物推理过程奖励模型。
摘要：
在科学推理任务中，推理过程的准确性与最终结果同样关键。尽管过程奖励模型（PRMs）解决了结果奖励模型（ORMs）固有的粗粒度监督问题，但其部署受到获取专家验证的步骤标签成本过高的限制。本文解决了一种利用大量但嘈杂的“弱”监督训练可靠PRMs的挑战。我们认为现有的弱到强泛化（W2SG）理论缺乏从嘈杂数据中选择高质量训练信号的指导方针。为此，我们引入了双共识弱到强（DC-W2S）框架。通过在弱监督者之间交叉自共识（SC）度量与嵌入空间中的邻域共识（NC）度量，我们将监督信号分层到不同的可靠性领域。然后，我们采用实例级平衡采样和标签级可靠性感知掩码的课程，以指导训练过程。我们证明了DC-W2S能够为复杂推理训练出稳健的PRMs，而无需大量专家标注，证明了战略性数据整理比在大规模嘈杂数据集上盲目训练更有效。
英文摘要：
In scientific reasoning tasks, the veracity of the reasoning process is as critical as the final outcome. While Process Reward Models (PRMs) offer a solution to the coarse-grained supervision problems inherent in Outcome Reward Models (ORMs), their deployment is hindered by the prohibitive cost of obtaining expert-verified step-wise labels. This paper addresses the challenge of training reliable PRMs using abundant but noisy "weak" supervision. We argue that existing Weak-to-Strong Generalization (W2SG) theories lack prescriptive guidelines for selecting high-quality training signals from noisy data. To bridge this gap, we introduce the Dual-Consensus Weak-to-Strong (DC-W2S) framework. By intersecting Self-Consensus (SC) metrics among weak supervisors with Neighborhood-Consensus (NC) metrics in the embedding space, we stratify supervision signals into distinct reliability regimes. We then employ a curriculum of instance-level balanced sampling and label-level reliability-aware masking to guide the training process. We demonstrate that DC-W2S enables the training of robust PRMs for complex reasoning without exhaustive expert annotation, proving that strategic data curation is more effective than indiscriminate training on large-scale noisy datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
