arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.00698cs.LGcs.AIcs.CL
收藏

为强化学习学习提示

Learning to Hint for Reinforcement Learning

University of California, San Diego(加州大学圣迭戈分校);Snowflake AI Research

Yu Xia, Canwen Xu, Zhewei Yao, Julian McAuley, Yuxiong He

AI总结:

本文提出HiLL框架,通过联合训练提示策略和推理策略,解决强化学习中因提示不足导致的优势崩溃问题,通过提示依赖性理论提升提示的迁移效果,实验显示HiLL在多个基准上优于GRPO和传统提示方法。

摘要:

组相对策略优化(GRPO)广泛用于具有可验证奖励的强化学习,但常遭遇优势崩溃:当一组中的所有回放获得相同奖励时,该组将产生零相对优势,从而无法学习。例如,如果一个问题对推理器来说太难,所有采样的回放都可能错误并获得零奖励。近期工作通过添加提示或辅助支架来解决这一问题,使推理器产生混合结果并恢复非零更新。然而,现有提示通常是固定的,而非适应当前推理器,且在提示输入下产生学习信号的提示不一定能改进测试时的无提示策略。为此,我们提出强化学习中的提示学习(HiLL),一种在RL中联合训练提示策略和推理策略的框架。对于每个困难问题,提示器根据当前推理器的错误回放生成提示,使提示生成适应推理器的演变错误。我们进一步引入提示依赖性,衡量正确提示轨迹对提示的依赖程度。我们推导出一个可转移性结果,表明较低的提示依赖性意味着从提示成功到无提示成功的更强转移性,并利用此结果定义一个转移加权奖励来训练提示器。因此,HiLL倾向于那些不仅恢复信息丰富的GRPO组,还能产生更可能改进原始无提示策略的提示。在多个基准上的实验表明,HiLL在多个基准上优于GRPO和先前基于提示的基线,证明了适应性和迁移感知提示学习在RL中的价值。代码可在https://github.com/Andree-9/HiLL获取。

英文摘要:

Group Relative Policy Optimization (GRPO) is widely used for reinforcement learning with verifiable rewards, but it often suffers from advantage collapse: when all rollouts in a group receive the same reward, the group yields zero relative advantage and thus no learning signal. For example, if a question is too hard for the reasoner, all sampled rollouts can be incorrect and receive zero reward. Recent work addresses this issue by adding hints or auxiliary scaffolds to such hard questions so that the reasoner produces mixed outcomes and recovers a non-zero update. However, existing hints are usually fixed rather than adapted to the current reasoner, and a hint that creates learning signal under the hinted input does not necessarily improve the no-hint policy used at test time. To this end, we propose Hint Learning for Reinforcement Learning (HiLL), a framework that jointly trains a hinter policy and a reasoner policy during RL. For each hard question, the hinter generates hints online conditioned on the current reasoner's incorrect rollout, allowing hint generation to adapt to the reasoner's evolving errors. We further introduce hint reliance, which measures how strongly correct hinted trajectories depend on the hint. We derive a transferability result showing that lower hint reliance implies stronger transfer from hinted success to no-hint success, and we use this result to define a transfer-weighted reward for training the hinter. Therefore, HiLL favors hints that not only recover informative GRPO groups, but also produce signals that are more likely to improve the original no-hint policy. Experiments across multiple benchmarks show that HiLL consistently outperforms GRPO and prior hint-based baselines, demonstrating the value of adaptive and transfer-aware hint learning for RL. The code is available at https://github.com/Andree-9/HiLL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑