通过口头批评过程监督提高大语言模型的推理能力
Process Supervision via Verbal Critique Improves Reasoning in Large Language Models
AI总结：
本文提出通过外部口头监督提升大语言模型推理能力的方法，通过结构化自然语言批评指导生成-批评-完善循环，提升推理性能和准确性。
摘要：
在推理时间扩展大语言模型推理能力时，研究重点集中在三个方向：链式深度、样本广度和学习的步骤评分器（PRMs）。我们引入第四个方向，即外部口头监督的细粒度，通过Verbal Process Supervision（VPS）框架实现。VPS是一种无训练框架，利用更强的监督者提供的结构化自然语言批评，指导生成-批评-完善循环，直到达到轮次预算R。在GPQA Diamond、AIME 2025和LiveCodeBench V6（涵盖封闭和开放模型）上，VPS取得了三个关键结果。首先，在GPQA Diamond上，GPT-5.4（High）| GPT-5.4（Low）在R=4时达到94.9%，超过无梯度更新的94.1%的当前最佳水平。其次，在AIME 2025上，VPS使弱演员救援显著增强，将分数从11.7-26.7%提升到63.3-90.0%（最高提升+63.3分）。第三，在同等计算资源下，VPS比Reflexion高出+8.5至+12.1分，比Self-Consistency@5高出+5.0个百分点（GPQA）和+8.3个百分点（LiveCodeBench），将批评细粒度确定为关键驱动因素。性能与监督者-演员能力差距（Pearson r=0.90）成正比，当错误无法用语言表达（如代码合成）时会退化，这推动了混合口头-可执行方法的发展。这些结果确立了批评细粒度作为推理时间扩展的新轴向。
英文摘要：
Inference-time scaling for LLM reasoning has focused on three axes: chain depth, sample breadth, and learned step-scorers (PRMs). We introduce a fourth axis, granularity of external verbal supervision, via Verbal Process Supervision (VPS), a training-free framework that uses structured natural-language critique from a stronger supervisor to guide an iterative generate-critique-refine loop up to a round budget R. Across GPQA Diamond, AIME 2025, and LiveCodeBench V6 (covering both closed and open models), VPS yields three key results. First, on GPQA Diamond, GPT-5.4 (High) | GPT-5.4 (Low) reaches 94.9% at R=4, surpassing the 94.1% state of the art without gradient updates. Second, on AIME 2025, VPS enables strong weak-actor rescue, boosting scores from 11.7-26.7% to 63.3-90.0% (up to +63.3 points). Third, at matched compute, VPS outperforms Reflexion by +8.5 to +12.1 points and Self-Consistency@5 by +5.0 pp (GPQA) and +8.3 pp (LiveCodeBench), isolating critique granularity as the key driver. Performance scales with the supervisor-actor capability gap (Pearson r=0.90) and degrades when errors are not linguistically expressible (e.g., code synthesis), motivating hybrid verbal-executable methods. These results establish critique granularity as a new axis of inference-time scaling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
