通过可验证准确性和回避奖励的课程强化学习缓解多轮对话中的迷失问题
Mitigating Lost in Multi-turn Conversation via Curriculum RL with Verifiable Accuracy and Abstention Rewards
AI总结：
本文提出RLAAR框架，通过可验证准确性和回避奖励的课程强化学习，减少多轮对话中因提前回答导致的性能下降，提升模型可靠性。
摘要：
大型语言模型在单轮指令遵循中表现优异，但在多轮对话中易出现Lost-in-Conversation（LiC）问题。受可验证奖励强化学习（RLVR）进展的启发，本文提出课程强化学习与可验证准确性和回避奖励（RLAAR）框架，鼓励模型不仅生成正确答案，还判断多轮对话中问题的可解性。该方法采用基于能力的课程，逐步增加对话难度（以指令碎片为单位），稳定训练并提升可靠性。通过多轮在线策略滚动和混合奖励系统，RLAAR教会模型在解决问题与有信息的回避之间取得平衡，减少导致LiC的提前回答行为。在LiC基准测试中，RLAAR显著减轻LiC性能下降（62.6%到75.1%），并提高校准回避率（33.5%到73.4%）。这些结果为构建多轮可靠和可信的LLM提供了实用的配方。
英文摘要：
Large Language Models demonstrate strong capabilities in single-turn instruction following but suffer from Lost-in-Conversation (LiC), a degradation in performance as information is revealed progressively in multi-turn settings. Motivated by the current progress on Reinforcement Learning with Verifiable Rewards (RLVR), we propose Curriculum Reinforcement Learning with Verifiable Accuracy and Abstention Rewards (RLAAR), a framework that encourages models not only to generate correct answers, but also to judge the solvability of questions in the multi-turn conversation setting. Our approach employs a competence-gated curriculum that incrementally increases dialogue difficulty (in terms of instruction shards), stabilizing training while promoting reliability. Using multi-turn, on-policy rollouts and a mixed-reward system, RLAAR teaches models to balance problem-solving with informed abstention, reducing premature answering behaviors that cause LiC. Evaluated on LiC benchmarks, RLAAR significantly mitigates LiC performance decay (62.6% to 75.1%) and improves calibrated abstention rates (33.5% to 73.4%). Together, these results provide a practical recipe for building multi-turn reliable and trustworthy LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
