arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.21052cs.LGcs.AI
收藏

借助导师帮助克服目标误泛化

Getting By Goal Misgeneralization With a Little Help From a Mentor

University of California, Berkeley(加州大学伯克利分校);McGill University(麦吉尔大学)

Tu Trinh, Mohamad H. Danesh, Nguyen X. Khanh, Benjamin Plaut

AI总结:

本文研究强化学习中的目标误泛化问题,提出让智能体在不熟悉情境中向监督者请求帮助,并在CoinRun环境中验证该方法能持续提升性能,同时揭示内部状态表征不足的局限。

摘要:

虽然强化学习(RL)智能体在训练期间通常表现良好,但在实际部署中面对分布偏移时可能会遇到困难。分布偏移的一个特别严重的风险是目标误泛化,即智能体学习到一个代理目标,该目标在训练期间与真实目标一致,但在部署期间却不一致。在本文中,我们探讨了允许智能体在不熟悉情境中向监督者请求帮助是否能缓解这一问题。我们专注于在CoinRun环境中使用PPO训练的智能体,该环境已知会表现出目标误泛化。我们评估了多种用于确定智能体何时应请求帮助的方法,并发现请求帮助能持续提升性能。然而,我们也发现基于智能体内部状态的方法未能主动请求帮助,而是等到错误已经发生后才行动。进一步调查表明,智能体的内部状态根本没有表示硬币,这突显了学习细致表征的重要性、忽略所有与奖励不直接相关信息的风险,以及开发针对智能体训练算法定制的请求帮助策略的必要性。

英文摘要:

While reinforcement learning (RL) agents often perform well during training, they can struggle with distribution shift in real-world deployments. One particularly severe risk of distribution shift is goal misgeneralization, where the agent learns a proxy goal that coincides with the true goal during training but not during deployment. In this paper, we explore whether allowing an agent to ask for help from a supervisor in unfamiliar situations can mitigate this issue. We focus on agents trained with PPO in the CoinRun environment, a setting known to exhibit goal misgeneralization. We evaluate multiple methods for determining when the agent should request help and find that asking for help consistently improves performance. However, we also find that methods based on the agent's internal state fail to proactively request help, instead waiting until mistakes have already occurred. Further investigation suggests that the agent's internal state does not represent the coin at all, highlighting the importance of learning nuanced representations, the risks of ignoring everything not immediately relevant to reward, and the necessity of developing ask-for-help strategies tailored to the agent's training algorithm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑