arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.23596cs.CL
收藏

再思两次:分支与再思考推理奖励模型

Think Twice: Branch-and-Rethink Reasoning Reward Model

NVIDIA(NVIDIA公司);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Yizhu Jiao, Jiaqi Zeng, Julien Veron Vialard, Oleksii Kuchaiev, Jiawei Han, Olivier Delalleau

AI总结:

BR-RM通过两轮次的分支与再思考机制,改进奖励模型的判断准确性与敏感性,实现更精确的推理能力。

摘要:

大型语言模型(LLMs)越来越依赖于将中间步骤外部化的思考模型,并分配额外的测试时间计算,其中think-twice策略表明,经过深思熟虑的第二次遍历可以产生更强的推理能力。相比之下,大多数奖励模型(RMs)仍然将许多质量维度压缩成一个单一的标量,这种设计导致了判断扩散:注意力在评估标准上分散,产生稀释的焦点和浅层分析。我们引入了分支与再思考(BR-RM),一种两轮次的奖励模型,将think-twice原则应用于奖励建模。第一轮执行自适应分支,选择一小部分实例关键维度(如事实性和安全性)并勾勒出简洁、证据导向的假设。第二轮执行分支条件再思考,一种有针对性的重读,测试这些假设并仅审视最重要的内容。我们使用GRPO风格的强化学习在结构化的两轮次轨迹上进行训练,使用简单的二元结果奖励和严格的格式检查,使该方法与标准的RLHF流水线兼容。通过将一次性评分转换为聚焦的二次审视推理,BR-RM减少了判断扩散,提高了对细微但重要的错误的敏感性,同时保持了实用性和可扩展性。实验结果表明,我们的模型在三个具有挑战性的奖励建模基准上实现了最先进的性能,覆盖了多样化的领域。

英文摘要:

Large language models (LLMs) increasingly rely on thinking models that externalize intermediate steps and allocate extra test-time compute, with think-twice strategies showing that a deliberate second pass can elicit stronger reasoning. In contrast, most reward models (RMs) still compress many quality dimensions into a single scalar in one shot, a design that induces judgment diffusion: attention spreads across evaluation criteria, yielding diluted focus and shallow analysis. We introduce branch-and-rethink (BR-RM), a two-turn RM that transfers the think-twice principle to reward modeling. Turn 1 performs adaptive branching, selecting a small set of instance-critical dimensions (such as factuality and safety) and sketching concise, evidence-seeking hypotheses. Turn 2 executes branch-conditioned rethinking, a targeted reread that tests those hypotheses and scrutinizes only what matters most. We train with GRPO-style reinforcement learning over structured two-turn traces using a simple binary outcome reward with strict format checks, making the approach compatible with standard RLHF pipelines. By converting all-at-once scoring into focused, second-look reasoning, BR-RM reduces judgment diffusion and improves sensitivity to subtle yet consequential errors while remaining practical and scalable. Experimental results demonstrate that our model achieves state-of-the-art performance on three challenging reward modeling benchmarks across diverse domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑