arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2001.09227cs.ROcs.SYeess.SY
收藏

主动任务推理引导的深度逆强化学习

Active Task-Inference-Guided Deep Inverse Reinforcement Learning

Oden Institute for Computational Engineering and Sciences, University of Texas at Austin(德克萨斯大学奥斯汀分校奥登计算工程与科学研究所);Google LLC(谷歌有限责任公司)

Farzan Memarian, Zhe Xu, Bo Wu, Min Wen, Ufuk Topcu

摘要:

我们考虑时间扩展任务的奖励学习问题。对于奖励学习,逆强化学习(IRL)是一种广泛使用的范式。给定一个马尔可夫决策过程(MDP)和一组任务的演示,IRL学习一个奖励函数,该函数为MDP的每个状态分配一个实值奖励。然而,对于时间上扩展的任务,潜在的奖励函数可能无法表达为MDP的各个状态的函数。相反,可能需要考虑访问状态的历史来确定当前状态下的奖励。为了解决这个问题,我们提出了一个迭代算法来学习时间扩展任务的奖励函数。在每次迭代时,算法在两个模块之间交替,任务推断模块推断底层任务结构,奖励学习模块使用推断的任务结构来学习奖励函数。任务推理模块产生一系列查询,其中每个查询是一系列子目标。演示器通过尝试在环境中执行查询并观察环境的反馈,为每个查询提供二进制响应。在回答了查询之后,任务推理模块返回一个自动机,该自动机对任务结构的当前假设进行编码。奖励学习模块用自动机的状态来扩充MDP的状态空间。然后,该模块继续使用新的深度最大熵IRL算法在增强的状态空间上学习奖励函数。这个迭代过程一直持续,直到它学习到一个性能令人满意的奖励函数。实验表明,该算法在时间扩展任务上的性能明显优于几个IRL基线。

英文摘要:

We consider the problem of reward learning for temporally extended tasks. For reward learning, inverse reinforcement learning (IRL) is a widely used paradigm. Given a Markov decision process (MDP) and a set of demonstrations for a task, IRL learns a reward function that assigns a real-valued reward to each state of the MDP. However, for temporally extended tasks, the underlying reward function may not be expressible as a function of individual states of the MDP. Instead, the history of visited states may need to be considered to determine the reward at the current state. To address this issue, we propose an iterative algorithm to learn a reward function for temporally extended tasks. At each iteration, the algorithm alternates between two modules, a task inference module that infers the underlying task structure and a reward learning module that uses the inferred task structure to learn a reward function. The task inference module produces a series of queries, where each query is a sequence of subgoals. The demonstrator provides a binary response to each query by attempting to execute it in the environment and observing the environment's feedback. After the queries are answered, the task inference module returns an automaton encoding its current hypothesis of the task structure. The reward learning module augments the state space of the MDP with the states of the automaton. The module then proceeds to learn a reward function over the augmented state space using a novel deep maximum entropy IRL algorithm. This iterative process continues until it learns a reward function with satisfactory performance. The experiments show that the proposed algorithm significantly outperforms several IRL baselines on temporally extended tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑