arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2112.08438cs.LGcs.AIcs.PL
收藏

程序化奖励设计实例

Programmatic Reward Design by Example

Boston University(波士顿大学)

Weichao Zhou, Wenchao Li

摘要:

奖励设计是强化学习(RL)中的一个基本问题。错误指定或设计不当的奖励可能导致低样本效率和不期望的行为。在本文中,我们提出了程序化奖励设计的思想,即在RL环境中使用程序来指定奖励函数。程序允许人类工程师以结构化和可解释的方式表达子目标和复杂的任务场景。然而,程序化奖励设计的挑战在于,虽然人类可以提供高层次的结构,但适当地设置低层次的细节,例如为特定的子任务提供适当的奖励,仍然很困难。本文的一个主要贡献是一个概率框架,可以推断出最好的候选人的编程奖励功能,从专家演示。受最近的生成对抗方法的启发,我们的框架搜索最有可能的编程奖励函数,在该函数下,最佳生成的轨迹无法与演示的轨迹区分开来。实验结果表明,使用该框架学习的程序化奖励函数可以显著优于使用现有奖励学习算法学习的程序化奖励函数,并使RL代理能够在高度复杂的任务上实现最先进的性能。

英文摘要:

Reward design is a fundamental problem in reinforcement learning (RL). A misspecified or poorly designed reward can result in low sample efficiency and undesired behaviors. In this paper, we propose the idea of programmatic reward design, i.e. using programs to specify the reward functions in RL environments. Programs allow human engineers to express sub-goals and complex task scenarios in a structured and interpretable way. The challenge of programmatic reward design, however, is that while humans can provide the high-level structures, properly setting the low-level details, such as the right amount of reward for a specific sub-task, remains difficult. A major contribution of this paper is a probabilistic framework that can infer the best candidate programmatic reward function from expert demonstrations. Inspired by recent generative-adversarial approaches, our framework searches for the most likely programmatic reward function under which the optimally generated trajectories cannot be differentiated from the demonstrated trajectories. Experimental results show that programmatic reward functionslearned using this framework can significantly outperform those learned using existing reward learning algo-rithms, and enable RL agents to achieve state-of-the-artperformance on highly complex tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑