arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1902.09996cs.AIcs.LGstat.ML
收藏

终止批评

The Termination Critic

DeepMind

Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Remi Munos, Doina Precup

摘要:

在这项工作中,我们考虑的问题自主发现行为抽象,或选项,强化学习代理。我们提出了一个算法,侧重于终止条件,而不是-因为是常见的-政策。终止条件通常是为了优化控制目标而训练的:如果另一个选项具有更好的价值,则应该终止。我们提供了一个不同的,信息理论的角度来看,并建议终止应该集中在选项的编码的压缩性-可以说是使用抽象的一个关键原因。为了在算法上实现这一点,我们利用经典的期权框架,并学习期权转换模型作为终止条件的“评论家”。使用这个模型,我们得到优化所需标准的梯度。我们表明,由此产生的选项是不平凡的,直观上有意义的,有用的学习和规划。

英文摘要:

In this work, we consider the problem of autonomously discovering behavioral abstractions, or options, for reinforcement learning agents. We propose an algorithm that focuses on the termination condition, as opposed to -- as is common -- the policy. The termination condition is usually trained to optimize a control objective: an option ought to terminate if another has better value. We offer a different, information-theoretic perspective, and propose that terminations should focus instead on the compressibility of the option's encoding -- arguably a key reason for using abstractions. To achieve this algorithmically, we leverage the classical options framework, and learn the option transition model as a "critic" for the termination condition. Using this model, we derive gradients that optimize the desired criteria. We show that the resulting options are non-trivial, intuitively meaningful, and useful for learning and planning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑