通过强化学习教学语言模型进行批评
Teaching Language Models to Critique via Reinforcement Learning
AI总结：
本文提出CTRL框架，通过强化学习训练批评模型以提升代码生成的准确性和性能，实现迭代改进和误差缓解。
摘要：
教学大型语言模型（LLMs）进行批评和改进其输出对于构建能够迭代改进的系统至关重要，但其根本上受到提供准确判断和可行建议能力的限制。在本工作中，我们研究了用于代码生成的LLM批评者，并提出了CTRL，一种通过强化学习进行批评训练的框架，该框架训练批评模型生成反馈，以最大化固定生成模型的纠正性能，而无需人类监督。我们的结果表明，通过CTRL训练的批评者显著提高了通过率，并在基础和更强的生成模型上缓解了累积误差。此外，我们展示了这些批评模型能够作为准确的生成奖励模型，并通过迭代的批评-修订实现测试时的扩展，从而在具有挑战性的代码生成基准上实现了高达106.1%的相对改进。
英文摘要：
Teaching large language models (LLMs) to critique and refine their outputs is crucial for building systems that can iteratively improve, yet it is fundamentally limited by the ability to provide accurate judgments and actionable suggestions. In this work, we study LLM critics for code generation and propose $\texttt{CTRL}$, a framework for $\texttt{C}$ritic $\texttt{T}$raining via $\texttt{R}$einforcement $\texttt{L}$earning, which trains a critic model to generate feedback that maximizes correction performance for a fixed generator model without human supervision. Our results demonstrate that critics trained with $\texttt{CTRL}$ significantly enhance pass rates and mitigate compounding errors across both base and stronger generator models. Furthermore, we show that these critic models act as accurate generative reward models and enable test-time scaling through iterative critique-revision, achieving up to 106.1% relative improvements across challenging code generation benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
