arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.01758cs.LGcs.AIcs.CL
收藏

RLAC:用于自由生成任务的对抗性评论家强化学习

RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks

Shanghai Jiao Tong University(上海交通大学);UC Berkeley(加州大学伯克利分校);Carnegie Mellon University(卡内基梅隆大学)

Mian Wu, Gavin Zhang, Sewon Min, Sergey Levine, Aviral Kumar

AI总结:

针对自由生成任务中评估准则过多导致验证成本高的问题,提出RLAC方法,通过大语言模型动态识别失败模式并联合优化生成器与评论家,显著提升了生成质量与验证效率。

摘要:

开放式生成任务要求输出满足多样化且通常是隐式的特定任务评估准则。大量相关准则导致验证成本过高且对响应的评估不完整,使得基于准则奖励的强化学习(RL)后训练难以扩展。通常,将这些准则组合成单一奖励的最佳方式也高度依赖于特定提示,这加剧了该问题。我们提出了对抗性评论家强化学习(RLAC),这是一种通过动态准则验证来解决这些挑战的后训练方法。我们的方法采用大语言模型(LLM)作为评论家,动态识别最可能出现的失败模式(例如事实错误或未处理的边缘情况),随后由外部验证器进行验证,以联合优化生成器和评论家。通过同时训练生成器和评论家,这种博弈提升了评论家的错误检测能力和生成器的输出质量,同时减少了所需的验证次数。我们的实验表明,RLAC提高了文本生成的事实准确性和代码生成的正确性,同时优于穷举验证和奖励模型方法。我们证明了动态评论家比固定评论家更有效,展示了RLAC在将RL后训练扩展到自由生成任务方面的潜力。

英文摘要:

Open-ended generation tasks require outputs to satisfy diverse and often implicit task-specific evaluation rubrics. The sheer number of relevant rubrics leads to prohibitively high verification costs and incomplete assessments of a response, making reinforcement learning (RL) post-training with rubric-based rewards difficult to scale. This problem is exacerbated by the fact that often the best way to combine these rubrics into one single reward is also highly prompt-specific. We propose Reinforcement Learning with Adversarial Critic (RLAC), a post-training approach that addresses these challenges via dynamic rubric verification. Our approach employs a large language model (LLM) as a critic that dynamically identifies only the most likely failure modes (e.g., a factual error or unhandled edge case), which are then verified by an external validator to optimize both generator and critic jointly. By training both the generator and the critic, this game enhances the critic's error detection and the generator's output quality while reducing required verifications. Our experiments demonstrate that RLAC improves factual accuracy in text generation and correctness in code generation, while also outperforming exhaustive verification and reward model methods. We show that dynamic critics are more effective than fixed critics, showcasing the potential of RLAC for scaling RL post-training to free-form generation tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑