arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.17815cs.CL
收藏

通过蒙特卡洛网络信息增益实现链式推理过程监督

RAWR: Reward Assignment Without Rollouts in Verifiable Domains

Department of Computer Science, ETH Zurich(苏黎世联邦理工学院计算机科学系);IBM Research(IBM研究院)

Corentin Royer, Anna Hedström, Debarun Bhattacharjya, Gaetano Rossiello, Andrea Giovannini, Mennatallah El-Assady

AI总结:

本文提出利用信息论自动生成推理步骤标签,提升大语言模型推理过程的监督效率与可靠性,适用于多种推理基准测试。

摘要:

多步推理提升了大语言模型(LLMs)的能力,但会增加中间步骤错误传播的风险。过程奖励模型(PRMs)通过单独评分每个步骤来缓解这一问题,实现细粒度监督和改进的可靠性。现有PRMs训练方法依赖于昂贵的人工标注或计算密集型自动标注。我们提出了一种新的方法,利用信息论自动生成步骤级标签。我们的方法估计每个推理步骤对正确答案可能性的影响,提供步骤质量的信号。重要的是,它将计算复杂度降低到O(N),优于之前的O(N log N)方法。我们证明这些标签在最佳K评估设置下能有效选择链式推理,适用于数学、Python编程、SQL和科学问答等多种推理基准。这项工作实现了LLM推理的可扩展和高效监督,尤其适用于错误传播关键的任务。

英文摘要:

Understanding and evaluating multi-step reasoning in LLMs at the level of individual steps remains a key challenge. Process reward models (PRMs) provide a solution by scoring each step, enabling fine-grained supervision and improved reliability. However, training them requires costly human annotation or computationally intensive rollout-based labeling. To solve this, we introduce MCNIG, a scalable method for automatically labeling the quality of individual reasoning steps in any verifiable domain. Its step score, net information gain (NetIG), improves upon single-reference information gain (IG) by comparing the most-supported correct answer against the most-supported incorrect one, yielding a robust signal even for long and structured outputs like code and SQL, where IG fails. We show that the signal produced by MCNIG correlates with human judgments of step quality, and we apply MCNIG labels to train PRMs that achieve the best average best-of-K accuracy across eight benchmarks spanning mathematics, code generation, text-to-SQL, and scientific QA. Crucially, MCNIG generates no rollouts, cutting labeling complexity to O(N) and making it up to X times cheaper than rollout-based methods at comparable label quality, which makes large-scale process supervision practical.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑