与批评共舞：通过逐步自然语言自我批评增强大语言模型推理能力
Dancing with Critiques: Enhancing LLM Reasoning with Stepwise Natural Language Self-Critique
AI总结：
针对大语言模型多步推理中传统标量奖励缺乏定性信息的问题，提出逐步自然语言自我批评方法PANEL，通过自生成自然语言批评指导步骤级搜索，无需任务特定验证器，在AIME、GPQA等基准上性能优于传统方法。
摘要：
增强大语言模型（LLM）的推理能力，尤其是针对需要多步逻辑推导的复杂任务，仍是一项重大挑战。传统推理时缩放方法利用过程奖励模型的标量奖励信号来评估候选推理步骤，但这类标量奖励缺乏理解和论证每一步骤所必需的细致定性信息。本文提出一种新颖的推理时缩放方法——逐步自然语言自我批评（PANEL），该方法利用自生成的自然语言批评作为反馈，指导步骤级搜索过程。通过为每个候选推理步骤生成丰富、人类可读的批评，PANEL保留了关键的定性信息，助力推理过程中做出更明智的决策。该方法无需任务特定的验证器及相关训练开销，可广泛应用于各类任务。在AIME、GPQA等具有挑战性的推理基准上的实验结果表明，PANEL显著提升了推理性能，优于传统基于标量奖励的方法。我们的代码可在https://github.com/puddingyeah/PANEL获取，以支持并推动该前沿领域的未来研究。
英文摘要：
Enhancing the reasoning capabilities of large language models (LLMs), particularly for complex tasks requiring multi-step logical deductions, remains a significant challenge. Traditional inference time scaling methods utilize scalar reward signals from process reward models to evaluate candidate reasoning steps, but these scalar rewards lack the nuanced qualitative information essential for understanding and justifying each step. In this paper, we propose a novel inference-time scaling approach -- stepwise natural language self-critique (PANEL), which employs self-generated natural language critiques as feedback to guide the step-level search process. By generating rich, human-readable critiques for each candidate reasoning step, PANEL retains essential qualitative information, facilitating better-informed decision-making during inference. This approach bypasses the need for task-specific verifiers and the associated training overhead, making it broadly applicable across diverse tasks. Experimental results on challenging reasoning benchmarks, including AIME and GPQA, demonstrate that PANEL significantly enhances reasoning performance, outperforming traditional scalar reward-based methods. Our code is available at https://github.com/puddingyeah/PANEL to support and encourage future research in this promising field.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
