arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.15224cs.AIcs.MA
收藏

ICRL: 通过强化学习学习内化自我批评

ICRL: Learning to Internalize Self-Critique with Reinforcement Learning

Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Nanjing University(南京大学);Sun Yat-sen University(中山大学);National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学);SAP;Microsoft Research(微软研究院)

Jianbo Lin, Xiaomin Yu, Yi Xin, Yifu Guo, Zhuosong Jiang, Zhongqi Yue, Weishi Wang, Heqing Zou, Chengwei Qin, Hui Xiong

AI总结:

本文提出ICRL框架,通过联合训练求解器和批评者,使求解器在无外部批评时也能自我改进,提升代理和数学推理任务性能。

摘要:

基于大语言模型的智能体在做出错误决策后,批评往往能引导其行为趋于正确。然而,当去除批评时,模型可能在相同查询上再次失败,表明其未将批评指导内化为自身能力。同时,冻结的批评者无法随时间提升反馈质量,限制了迭代自我改进的潜力。为此,我们提出ICRL,一种新颖的框架,通过共享骨干联合训练求解器和批评者,将批评引发的成功转化为无监督求解能力。批评者基于求解器后续表现提升获得奖励,激励产生有行动性的反馈。为解决批评条件与无批评行为之间的分布偏移,ICRL引入分布校准再加权比,选择性地转移与求解器自身提示分布兼容的批评引导改进。此外,角色间组优势估计稳定了两个角色的联合优化。这些机制确保求解器在无外部批评时学习自我改进,而非依赖批评条件行为。我们在多样化的基准上评估ICRL,使用Qwen3-4B和Qwen3-8B作为骨干。结果表明,平均在代理任务上比GRPO提升6.4分,在数学推理任务上提升7.0分。值得注意的是,学习的8B批评者在使用显著更少token的情况下,性能可与32B批评者媲美。代码可在https://github.com/brick-pid/ICRL获取。

英文摘要:

Large language model-based agents make mistakes, yet critique can often guide the same model toward correct behavior. However, when critique is removed, the model may fail again on the same query, indicating that it has not internalized the critique's guidance into its underlying capability. Meanwhile, a frozen critic cannot improve its feedback quality over time, limiting the potential for iterative self-improvement. To address this, we propose learning to internalize self-critique with reinforcement learning(ICRL), a novel framework that jointly trains a solver and a critic from a shared backbone to convert critique-induced success into unassisted solver ability. The critic is rewarded based on the solver's subsequent performance gain, incentivizing actionable feedback. To address the distribution shift between critique-conditioned and critique-free behavior, ICRL introduces a distribution-calibration re-weighting ratio that selectively transfers critique-guided improvements compatible with the solver's own prompt distribution. Additionally, a role-wise group advantage estimation stabilizes joint optimization across the two roles. Together, these mechanisms ensure that the solver learns to improve itself without external critique, rather than becoming dependent on critique-conditioned behavior. We evaluate ICRL on diverse benchmarks spanning agentic and mathematical reasoning tasks, using Qwen3-4B and Qwen3-8B as backbones. Results show consistent improvements, with average gains of 6.4 points over GRPO on agentic tasks, and 7.0 points on mathematical reasoning. Notably, the learned 8B critic is comparable to 32B critics while using substantially fewer tokens. The code is available at https://github.com/brick-pid/ICRL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑