arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.07922cs.LG
收藏

SERL:面向开放域的自我审查强化学习

SERL: Self-Examining Reinforcement Learning on Open-Domain

Weixuan Ou, Yanzhao Zheng, Shuoshuo Sun, Wei Zhang, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu, Pengwei Yan, Yifan Qiao

AI总结:

SERL通过自我审查机制提升开放域任务中的LLM性能,无需外部信号,实现演员与裁判的协同改进。

摘要:

强化学习(RL)已被证明能够提升大语言模型(LLMs)的能力。然而,将RL应用于开放域任务面临两个关键挑战:(1)这些任务的固有主观性使得无法满足强化学习可验证奖励(RLVR)所要求的可验证奖励;(2)强化学习从人类反馈(RLHF)依赖于外部奖励机制。为克服这些限制,我们提出了自我审查强化学习(SERL),一种新颖的自我改进框架,其中LLM同时担任演员和裁判。SERL引入了两种协同作用的奖励机制,无需任何外部信号。一方面,为了提升演员的能力,我们从一组生成响应的Copeland式成对比较判断中推导出奖励;另一方面,提出了一种自我一致性奖励,以鼓励一致的判断,从而提升裁判的可靠性。这一过程提升了裁判的能力,从而为演员提供更稳健的奖励。实验表明,我们的方法优于现有的自我改进训练方法。SERL使Qwen3-8B在AlpacaEval 2上的LC胜率从52.37%提升至59.90%。据我们所知,我们的方法在自我改进方法中实现了最先进的性能。此外,它在性能上与显著更大的模型如Qwen3-32B相当,证明了在开放域任务上具有优越的效果和鲁棒性。

英文摘要:

Reinforcement Learning (RL) has been shown to improve the capabilities of large language models (LLMs). However, applying RL to open-domain tasks faces two key challenges: (1) the inherent subjectivity of these tasks prevents the verifiable rewards as required by Reinforcement Learning with Verifiable Rewards (RLVR); (2) Reinforcement Learning from Human Feedback (RLHF) relies on external reward mechanisms. To overcome these limitations, we propose Self-Examining Reinforcement Learning (SERL), a novel self-improving framework where the LLM serves as both Actor and Judge. SERL introduces two synergistic reward mechanisms without any external signals. On the one hand, to improve the Actor's capability, we derive rewards from Copeland-style pairwise comparison judgments across a group of generated responses. On the other hand, a self-consistency reward that encourages coherent judgments is proposed to improve the Judge's reliability. This process refines the Judge's capability, which in turn provides a more robust reward for Actor. Experiments show that our method outperforms existing self-improvement training methods. SERL improves the LC win rate of Qwen3-8B on AlpacaEval 2 from 52.37% to 59.90%. To the best of our knowledge, our method achieves state-of-the-art performance among self-improving approaches. Furthermore, it achieves a performance comparable to significantly larger models like Qwen3-32B, demonstrating superior effectiveness and robustness on open-domain tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑