arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1907.03046cs.LGcs.AI
收藏

从演示中学习行为技巧

Learning a Behavioral Repertoire from Demonstrations

IT University of Copenhagen(哥本哈根信息技术大学);Universidad Nacional de Colombia(哥伦比亚国立大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);Université de Lorraine(洛林大学)

Niels Justesen, Miguel Gonzalez Duque, Daniel Cabarcas Jaramillo, Jean-Baptiste Mouret, Sebastian Risi

摘要:

模仿学习(IL)是一种从演示数据集学习策略的机器学习方法。在应用强化学习(RL)之前,IL可以用于启动学习,但它本身也可以是有用的,例如学习模仿视频游戏中的人类玩家。然而,当前IL方法的一个主要限制是,它们仅基于可能包含许多不同类型行为的演示的数据集来学习单个“平均”策略。在本文中,我们提出了一种新的方法,称为行为库模仿学习(BRIL),而不是从一组演示通过增强状态动作对行为描述学习的行为库。这种方法的结果是一个单一的神经网络政策的行为描述,可以精确地调制。我们应用这种方法来训练一个策略,在7,777个人类回放中执行星际争霸II中的构建顺序规划。主成分分析(PCA)是应用于构建一个低维的行为空间,从高维的军队单位组成的每个示范。结果表明,学习的政策可以有效地操纵,以表达不同的行为。此外,通过应用UCB 1算法,我们能够在游戏之间调整策略的行为,以达到超越传统IL基线方法的性能。

英文摘要:

Imitation Learning (IL) is a machine learning approach to learn a policy from a dataset of demonstrations. IL can be useful to kick-start learning before applying reinforcement learning (RL) but it can also be useful on its own, e.g. to learn to imitate human players in video games. However, a major limitation of current IL approaches is that they learn only a single "average" policy based on a dataset that possibly contains demonstrations of numerous different types of behaviors. In this paper, we propose a new approach called Behavioral Repertoire Imitation Learning (BRIL) that instead learns a repertoire of behaviors from a set of demonstrations by augmenting the state-action pairs with behavioral descriptions. The outcome of this approach is a single neural network policy conditioned on a behavior description that can be precisely modulated. We apply this approach to train a policy on 7,777 human replays to perform build-order planning in StarCraft II. Principal Component Analysis (PCA) is applied to construct a low-dimensional behavioral space from the high-dimensional army unit composition of each demonstration. The results demonstrate that the learned policy can be effectively manipulated to express distinct behaviors. Additionally, by applying the UCB1 algorithm, we are able to adapt the behavior of the policy - in-between games - to reach a performance beyond that of the traditional IL baseline approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑