arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2003.07069cs.CV
收藏

强化学习中可解释特征的自监督发现

Self-Supervised Discovering of Interpretable Features for Reinforcement Learning

Tsinghua University(清华大学);Beijing Institute of Electronic System Engineering(北京电子系统工程研究所)

Wenjie Shi, Gao Huang, Shiji Song, Zhuoyuan Wang, Tingyu Lin, Cheng Wu

摘要:

深度强化学习(RL)最近在一系列复杂的控制任务上取得了许多突破。然而,代理人的决策过程一般不透明。可解释性的缺乏阻碍了RL在安全关键场景中的适用性。虽然有几种方法试图解释基于视觉的强化学习,但大多数都没有详细解释代理的行为。在本文中,我们提出了一个自我监督的可解释的框架,它可以发现可解释的功能,即使是非专家也可以轻松理解RL代理。具体来说,一个自我监督的可解释网络(SSINet)是用来产生细粒度的注意力掩模突出任务相关的信息,这构成了代理的决策的大部分证据。我们在几款Atari 2600游戏以及Duckietown(一个具有挑战性的自动驾驶汽车模拟器环境)上验证和评估了我们的方法。结果表明,我们的方法呈现的经验证据,代理如何作出决定,为什么代理表现好或坏,特别是当转移到新的场景。总的来说,我们的方法为基于视觉的RL的内部决策过程提供了有价值的见解。此外,我们的方法不使用任何外部标记数据,因此证明了通过自监督方式学习高质量掩模的可能性,这可能为无标记视觉学习(如自监督分割和检测)提供新的范例。

英文摘要:

Deep reinforcement learning (RL) has recently led to many breakthroughs on a range of complex control tasks. However, the agent's decision-making process is generally not transparent. The lack of interpretability hinders the applicability of RL in safety-critical scenarios. While several methods have attempted to interpret vision-based RL, most come without detailed explanation for the agent's behavior. In this paper, we propose a self-supervised interpretable framework, which can discover interpretable features to enable easy understanding of RL agents even for non-experts. Specifically, a self-supervised interpretable network (SSINet) is employed to produce fine-grained attention masks for highlighting task-relevant information, which constitutes most evidence for the agent's decisions. We verify and evaluate our method on several Atari 2600 games as well as Duckietown, which is a challenging self-driving car simulator environment. The results show that our method renders empirical evidences about how the agent makes decisions and why the agent performs well or badly, especially when transferred to novel scenes. Overall, our method provides valuable insight into the internal decision-making process of vision-based RL. In addition, our method does not use any external labelled data, and thus demonstrates the possibility to learn high-quality mask through a self-supervised manner, which may shed light on new paradigms for label-free vision learning such as self-supervised segmentation and detection.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑