arXiv 2509.228242026-03-12cs.CL

Critique-Coder: 通过批判强化学习增强Coder模型

Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning

Chi Ruan, Dongfu Jiang, Yubo Wang, Wenhu Chen

AI总结：

Critique-Coder通过批判强化学习提升Coder模型性能，优于传统强化学习方法。

摘要：

强化学习（RL）已成为一种流行的学习范式，尤其是在与推理模型结合使用时。虽然有效，但其主要关注生成响应，缺乏机制显式促进批判或反思。近年来，一些研究如批判微调（CFT）和批判引导蒸馏（CGD）已显示出显式教授LLM如何批判的好处。受这些研究启发，我们提出了批判强化学习（CRL），其中模型的任务是为给定的（问题，解决方案）对生成批判。奖励仅由生成的批判的最终判断标签$ c \in \{\texttt{True}, \texttt{False}\} $是否与地面真实判断$ c^* $一致决定。基于此点，我们引入了Critique-Coder，其在RL和CRL的混合数据上进行训练，通过将标准RL数据的20%替换为CRL数据。我们对多个模型（Critique-Coder）进行微调，并在不同基准上评估它们，以展示其优于仅RL模型的优势。我们证明Critique-Coder在所有评估的基准上均优于仅RL基线。值得注意的是，我们的Critique-Coder-8B在LiveCodeBench（v5）上可达到超过60%，优于其他推理模型如DeepCoder-14B和GPT-o1。除了代码生成外，Critique-Coder还展示了增强的一般推理能力，这在BBEH数据集上的逻辑推理任务中得到了证明。这表明在代码数据集上应用CRL增强了通用推理和批判能力，这些能力可以跨广泛任务转移。因此，我们认为CRL在LLM推理中是标准RL的有力补充。

英文摘要：

Reinforcement Learning (RL) has emerged as a popular training paradigm, particularly when paired with reasoning models. While effective, it primarily focuses on generating responses and lacks mechanisms to explicitly foster critique or reflection. Several recent studies, like Critique-Fine-Tuning (CFT) and Critique-Guided-Distillation (CGD) have shown the benefits of explicitly teaching LLMs how to critique. Motivated by them, we propose Critique Reinforcement Learning (CRL), where the model is tasked with generating a critique for a given (question, solution) pair. The reward is determined solely by whether the final judgment label $c \in \{\texttt{True}, \texttt{False}\}$ of the generated critique aligns with the ground-truth judgment $c^*$. Building on this point, we introduce Critique-Coder, which is trained on a hybrid of RL and CRL by substituting 20% of the standard RL data with CRL data. We fine-tune multiple models (Critique-Coder) and evaluate them on different benchmarks to show their advantages over RL-only models. We show that Critique-Coder consistently outperforms RL-only baselines on all the evaluated benchmarks. Notably, our Critique-Coder-8B can reach over 60% on LiveCodeBench (v5), outperforming other reasoning models like DeepCoder-14B and GPT-o1. Beyond code generation, Critique-Coder also demonstrates enhanced general reasoning abilities, as evidenced by its better performance on logic reasoning tasks from the BBEH dataset. This indicates that the application of CRL on coding datasets enhances general reasoning and critique abilities, which are transferable across a broad range of tasks. Hence, we believe that CRL works as a great complement to standard RL for LLM reasoning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文