arXiv 2510.243202025-10-28cs.CLcs.AI

Critique-RL：通过两阶段强化学习训练用于批评的语言模型

Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning

Fudan University（复旦大学）；ByteDance Seed（字节跳动种子）

Zhiheng Xi, Jixuan Huang, Xin Guo, Boyang Hong, Dingwen Yang, Xiaoran Fan, Shuo Li, Zehui Chen, Junjie Ye, Siyu Yuan, Zhengyin Du, Xuesong Yao, Yufei Xu, Jiecao… 展开作者

Zhiheng Xi, Jixuan Huang, Xin Guo, Boyang Hong, Dingwen Yang, Xiaoran Fan, Shuo Li, Zehui Chen, Junjie Ye, Siyu Yuan, Zhengyin Du, Xuesong Yao, Yufei Xu, Jiecao Chen, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang

备注：: Preprint, 25 pages, 9 figures. Code: https://github.com/WooooDyy/Critique-RL

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有批评语言模型训练依赖更强监督的问题，提出采用两阶段强化学习的Critique-RL方法，先强化批评者判别性再提升有用性，在多任务多模型上实现显著性能提升。

摘要：

训练批评语言模型来评估模型输出并提供反馈，是提升大语言模型（LLM）复杂推理任务表现的一种有前景的方法。然而，现有方法通常依赖更强的监督者来标注批评数据。为解决这一问题，我们提出Critique-RL，这是一种无需更强监督即可开发批评语言模型的在线强化学习（RL）方法。该方法采用双玩家范式：行动者生成回复，批评者提供反馈，行动者据此优化回复。我们首先发现，仅依赖行动者输出的间接奖励信号进行RL优化，往往会得到效果不佳的批评者：尽管其有用性（即提供建设性反馈的能力）有所提升，但判别性（即判断回复质量高低的能力）仍然较差，最终带来的性能提升十分有限。为克服这一问题，Critique-RL采用两阶段优化策略：第一阶段，通过基于规则的直接奖励信号强化批评者的判别性；第二阶段，引入基于行动者优化的间接奖励来提升批评者的有用性，同时通过适当的正则化保持其判别性。在多种任务和模型上开展的大量实验表明，Critique-RL带来了显著的性能提升。例如，对于Qwen2.5-7B模型，该方法在域内任务上实现了9.02%的提升，在域外任务上实现了5.70%的提升，凸显了其应用潜力。

英文摘要：

Training critiquing language models to assess and provide feedback on model outputs is a promising way to improve LLMs for complex reasoning tasks. However, existing approaches typically rely on stronger supervisors for annotating critique data. To address this, we propose Critique-RL, an online RL approach for developing critiquing language models without stronger supervision. Our approach operates on a two-player paradigm: the actor generates a response, the critic provides feedback, and the actor refines the response accordingly. We first reveal that relying solely on indirect reward signals from the actor's outputs for RL optimization often leads to unsatisfactory critics: while their helpfulness (i.e., providing constructive feedback) improves, the discriminability (i.e., determining whether a response is high-quality or not) remains poor, resulting in marginal performance gains. To overcome this, Critique-RL adopts a two-stage optimization strategy. In stage I, it reinforces the discriminability of the critic with direct rule-based reward signals; in stage II, it introduces indirect rewards based on actor refinement to improve the critic's helpfulness, while maintaining its discriminability via appropriate regularization. Extensive experiments across various tasks and models show that Critique-RL delivers substantial performance improvements. For example, it achieves a 9.02% gain on in-domain tasks and a 5.70% gain on out-of-domain tasks for Qwen2.5-7B, highlighting its potential.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。