arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.07411cs.LG
收藏

基于势函数的内在动机奖励塑形

Potential-Based Reward Shaping For Intrinsic Motivation

North Carolina State University(北卡罗来纳州立大学)

Grant C. Forbes, Nitish Gupta, Leonardo Villalobos-Arias, Colin M. Potts, Arnav Jhala, David L. Roberts

AI总结:

本文针对内在动机奖励塑形可能改变最优策略集合的问题,提出了PBRS的扩展方法及Potential-Based Intrinsic Motivation(PBIM),将IM奖励转换为基于势函数的形式,在不改变最优策略集合的前提下防止次优收敛并加速训练。

摘要:

近年来,内在动机(IM)奖励塑形方法在复杂和稀疏奖励环境中大量涌现。这些方法往往会无意中改变环境中的最优策略集合,导致次优行为。先前关于缓解奖励塑形风险的工作,特别是通过基于势函数的奖励塑形(PBRS),并不适用于许多IM方法,因为这些方法本身往往是复杂的、可训练的函数,因此依赖于比PBRS最初为之开发的传统奖励函数更广泛的变量集。我们提出了PBRS的一种扩展,并证明该扩展在比以往证明的更广泛的函数集下能够保持最优策略集合不变。我们还提出了Potential-Based Intrinsic Motivation(PBIM,基于势函数的内在动机),这是一种将IM奖励转换为基于势函数形式的方法,可在不改变最优策略集合的情况下使用。在MiniGrid DoorKey和Cliff Walking环境中进行的测试表明,PBIM成功防止了智能体收敛到次优策略,并且能够加快训练速度。

英文摘要:

Recently there has been a proliferation of intrinsic motivation (IM) reward-shaping methods to learn in complex and sparse-reward environments. These methods can often inadvertently change the set of optimal policies in an environment, leading to suboptimal behavior. Previous work on mitigating the risks of reward shaping, particularly through potential-based reward shaping (PBRS), has not been applicable to many IM methods, as they are often complex, trainable functions themselves, and therefore dependent on a wider set of variables than the traditional reward functions that PBRS was developed for. We present an extension to PBRS that we prove preserves the set of optimal policies under a more general set of functions than has been previously proven. We also present {\em Potential-Based Intrinsic Motivation} (PBIM), a method for converting IM rewards into a potential-based form that is useable without altering the set of optimal policies. Testing in the MiniGrid DoorKey and Cliff Walking environments, we demonstrate that PBIM successfully prevents the agent from converging to a suboptimal policy and can speed up training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑