利用大语言模型反馈的强化学习以对抗目标错误泛化
Reinforcement Learning from LLM Feedback to Counteract Goal Misgeneralization
AI总结:
本文提出利用大语言模型反馈识别强化学习中的目标错误泛化失败场景,并学习奖励模型以重新训练智能体,在迷宫导航任务中显著提升了目标泛化能力。
摘要:
我们提出了一种方法,通过利用大语言模型(LLM)在训练期间的反馈来解决强化学习(RL)中的目标错误泛化问题。目标错误泛化是强化学习中的一种鲁棒性失效类型,当智能体在分布外仍保留其能力,却追求一个替代目标而非预期目标时就会发生。我们的方法利用LLM分析强化学习智能体在训练期间的行为策略,并识别潜在的失败场景。随后,将强化学习智能体部署到这些场景中,并通过LLM的偏好和反馈学习一个奖励模型。这个由LLM信息指导的奖励模型被用于在原始数据集上进一步训练强化学习智能体。我们将该方法应用于迷宫导航任务,结果显示在目标泛化方面有显著提升,尤其是在真实目标与替代目标具有一定可区分性且行为偏差明显的情况下。该研究表明,尽管LLM缺乏任务熟练度,它仍能高效地监督强化学习智能体,为通过使用LLM增强强化学习中的目标导向学习提供可扩展的监督和有价值的见解。
英文摘要:
We introduce a method to address goal misgeneralization in reinforcement learning (RL), leveraging Large Language Model (LLM) feedback during training. Goal misgeneralization, a type of robustness failure in RL occurs when an agent retains its capabilities out-of-distribution yet pursues a proxy rather than the intended one. Our approach utilizes LLMs to analyze an RL agent's policies during training and identify potential failure scenarios. The RL agent is then deployed in these scenarios, and a reward model is learnt through the LLM preferences and feedback. This LLM-informed reward model is used to further train the RL agent on the original dataset. We apply our method to a maze navigation task, and show marked improvements in goal generalization, especially in cases where true and proxy goals are somewhat distinguishable and behavioral biases are pronounced. This study demonstrates how the LLM, despite its lack of task proficiency, can efficiently supervise RL agents, providing scalable oversight and valuable insights for enhancing goal-directed learning in RL through the use of LLMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。