arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.16207cs.NEcs.AI
收藏

Quality-Diversity Transformer:使用Decision Transformers生成行为条件化轨迹

The Quality-Diversity Transformer: Generating Behavior-Conditioned Trajectories with Decision Transformers

InstaDeep;CNRS(法国国家科学研究中心);Sorbonne Université(索邦大学)

Valentin Macé, Raphaël Boige, Felix Chalumeau, Thomas Pierrot, Guillaume Richard, Nicolas Perrin-Gilbert

AI总结:

针对不确定环境中策略缺乏鲁棒性和储备库离散性导致行为不连续的问题,提出结合MAP-Elites Low-Spread(ME-LS)和Quality-Diversity Transformer(QDT)的方法,实现高精度按需生成行为条件化轨迹。

摘要:

在神经进化背景下,Quality-Diversity算法被证明能有效通过依赖行为空间的定义,生成包含多样且高效策略的储备库。创建此类储备库引发的一个自然目标是尝试按需实现行为,这可以通过运行储备库中相应的策略来完成。然而,在不确定环境中,会出现两个问题。首先,策略可能缺乏鲁棒性和可重复性,这意味着在略微不同条件下的多个回合通常会导致截然不同的行为。其次,由于储备库的离散性质,解决方案会不连续地变化。在此,我们提出了一种基于两个机制来实现行为条件化轨迹生成的新方法:首先,MAP-Elites Low-Spread (ME-LS) 将解决方案的选择限制在行为空间中最一致的那些方案上。其次,Quality-Diversity Transformer (QDT) 是一个基于连续行为描述符的条件化Transformer模型,它在由ME-LS储备库中的策略生成的数据集上进行训练,并学习自回归地生成能够实现目标行为的动作序列。结果表明,ME-LS产生了一致且鲁棒的策略,并且它与QDT的结合产生了一个能够按需高精度实现多样行为的单一策略。

英文摘要:

In the context of neuroevolution, Quality-Diversity algorithms have proven effective in generating repertoires of diverse and efficient policies by relying on the definition of a behavior space. A natural goal induced by the creation of such a repertoire is trying to achieve behaviors on demand, which can be done by running the corresponding policy from the repertoire. However, in uncertain environments, two problems arise. First, policies can lack robustness and repeatability, meaning that multiple episodes under slightly different conditions often result in very different behaviors. Second, due to the discrete nature of the repertoire, solutions vary discontinuously. Here we present a new approach to achieve behavior-conditioned trajectory generation based on two mechanisms: First, MAP-Elites Low-Spread (ME-LS), which constrains the selection of solutions to those that are the most consistent in the behavior space. Second, the Quality-Diversity Transformer (QDT), a Transformer-based model conditioned on continuous behavior descriptors, which trains on a dataset generated by policies from a ME-LS repertoire and learns to autoregressively generate sequences of actions that achieve target behaviors. Results show that ME-LS produces consistent and robust policies, and that its combination with the QDT yields a single policy capable of achieving diverse behaviors on demand with high accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑