arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.13900cs.LGcs.AIstat.ML
收藏

量化奖励函数中的差异

Quantifying Differences in Reward Functions

DeepMind

Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, Jan Leike

摘要:

对于许多任务来说,奖励函数无法进行内省,或者过于复杂而无法在程序上指定,因此必须从用户数据中学习。先前的工作已经通过评估针对学习奖励优化的策略来评估学习奖励函数。然而,该方法不能区分未能反映用户偏好的学习奖励函数和未能优化学习奖励的策略优化过程。此外,这种方法只能告诉我们评估环境中的行为,但即使在稍微不同的部署环境中,奖励也可能激励非常不同的行为。为了解决这些问题,我们引入了等价策略不变比较(EPIC)距离来直接量化两个奖励函数之间的差异,而无需策略优化步骤。我们证明了EPIC是不变的等价类的回报函数,总是导致相同的最优策略。此外,我们发现EPIC可以有效地近似,是更强大的覆盖分布的选择比基线。最后,我们表明,EPIC距离界限的遗憾,即使在不同的过渡动态的最佳政策,我们实证证实,它预测政策培训的成功。我们的源代码可以在https://github.com/HumanCompatibleAI/evaluating-rewards上找到。

英文摘要:

For many tasks, the reward function is inaccessible to introspection or too complex to be specified procedurally, and must instead be learned from user data. Prior work has evaluated learned reward functions by evaluating policies optimized for the learned reward. However, this method cannot distinguish between the learned reward function failing to reflect user preferences and the policy optimization process failing to optimize the learned reward. Moreover, this method can only tell us about behavior in the evaluation environment, but the reward may incentivize very different behavior in even a slightly different deployment environment. To address these problems, we introduce the Equivalent-Policy Invariant Comparison (EPIC) distance to quantify the difference between two reward functions directly, without a policy optimization step. We prove EPIC is invariant on an equivalence class of reward functions that always induce the same optimal policy. Furthermore, we find EPIC can be efficiently approximated and is more robust than baselines to the choice of coverage distribution. Finally, we show that EPIC distance bounds the regret of optimal policies even under different transition dynamics, and we confirm empirically that it predicts policy training success. Our source code is available at https://github.com/HumanCompatibleAI/evaluating-rewards.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑