长期交互中的内在信用分配
Intrinsic Credit Assignment for Long Horizon Interaction
AI总结：
本文提出ΔBelief-RL，通过内在ΔBelief奖励实现长期交互中的信用分配，提升强化学习在不确定性环境下的性能。
摘要：
如何训练代理在长期范围内应对不确定性？在本文中，我们提出了ΔBelief-RL，该方法利用语言模型自身内在信念来奖励中间进展。我们的方法利用代理对目标解决方案概率分配的变化来进行信用分配。通过在合成交互数据上训练，ΔBelief-RL教会了信息寻求能力，这种能力在强化学习中始终优于纯结果导向奖励，改进效果可推广到从客户服务到个性化等非分布应用。值得注意的是，随着我们在训练时间范围之外扩展测试时间交互，性能持续提高，即使在Pass@k指标上，交互效率也增加。总体而言，我们的工作通过通过内在ΔBelief奖励使中间动作获得信用，引入了一种可扩展的训练策略，以在长期范围内应对不确定性。
英文摘要：
How can we train agents to navigate uncertainty over long horizons? In this work, we propose ΔBelief-RL, which leverages a language model's own intrinsic beliefs to reward intermediate progress. Our method utilizes the change in the probability an agent assigns to the target solution for credit assignment. By training on synthetic interaction data, ΔBelief-RL teaches information-seeking capabilities that consistently outperform purely outcome-based rewards for Reinforcement Learning, with improvements generalizing to out-of-distribution applications ranging from customer service to personalization. Notably, the performance continues to improve as we scale test-time interactions beyond the training horizon, with interaction-efficiency increasing even on Pass@k metrics. Overall, our work introduces a scalable training strategy for navigating uncertainty over a long-horizon, by enabling credit assignment to intermediate actions via intrinsic ΔBelief rewards.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
