arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.08686cs.LGcs.AI
收藏

完全通用的在线模仿学习

Fully General Online Imitation Learning

University of Oxford(牛津大学);Future of Humanity Institute(人类未来研究所);DeepMind(深度思维);Australian National University(澳大利亚国立大学)

Michael K. Cohen, Marcus Hutter, Neel Nanda

摘要:

在模仿学习中,模仿者和示范者是根据过去与环境的交互选择行动的策略。如果我们运行一个模仿者,我们可能希望事件以类似的方式展开,如果示威者一直在行动。一般来说,学习过程中的一个错误可能会导致完全不同的事件。在重新启动的环境的特殊设置中,现有的工作提供了如何模仿的正式指导,以便事件类似地展开,但在该设置之外,没有正式的指导存在。我们解决了一个完全通用的设置,其中(随机)的环境和演示从来没有重置,即使是为了培训的目的,我们允许我们的模仿者在线学习的演示。我们新的保守贝叶斯模仿学习器低估了每个可用动作的概率,并以剩余的概率查询更多的数据。我们的主要结果:如果一个事件在演示者一直在行动的情况下是不可能发生的,那么当运行(最初完全无知的)模仿者时,该事件的可能性可以被限制在上面。与此同时,对演示者的询问频率迅速减少。如果任何这样的事件符合“危险”的条件,我们的模仿者将有相对“安全”的显着区别。

英文摘要:

In imitation learning, imitators and demonstrators are policies for picking actions given past interactions with the environment. If we run an imitator, we probably want events to unfold similarly to the way they would have if the demonstrator had been acting the whole time. In general, one mistake during learning can lead to completely different events. In the special setting of environments that restart, existing work provides formal guidance in how to imitate so that events unfold similarly, but outside that setting, no formal guidance exists. We address a fully general setting, in which the (stochastic) environment and demonstrator never reset, not even for training purposes, and we allow our imitator to learn online from the demonstrator. Our new conservative Bayesian imitation learner underestimates the probabilities of each available action, and queries for more data with the remaining probability. Our main result: if an event would have been unlikely had the demonstrator acted the whole time, that event's likelihood can be bounded above when running the (initially totally ignorant) imitator instead. Meanwhile, queries to the demonstrator rapidly diminish in frequency. If any such event qualifies as "dangerous", our imitator would have the notable distinction of being relatively "safe".

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑