arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2111.05992cs.LGcs.AI
收藏

多智能体强化学习中吸收态的使用与误用

On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning

Unity Technologies(Unity科技)

Andrew Cohen, Ervin Teng, Vincent-Pierre Berges, Ruo-Ping Dong, Hunter Henry, Marwan Mattar, Alexander Zook, Sujoy Ganguly

摘要:

合作多智能体强化学习(MARL)中智能体的创建和销毁是一个严重不足的研究领域。目前的MARL算法通常假设一组中的代理数量在整个实验过程中保持固定。然而,在许多实际问题中,智能体可能会在其队友之前终止。这种提前终止的问题提出了一个挑战:被终止的代理人必须从集团的成功或失败中吸取教训,这是在其自身存在之外发生的。我们将剩余队友获得的奖励的价值传播给被终止的代理称为死后信用分配问题。目前的MARL方法通过将这些代理置于吸收状态直到整个代理组达到终止条件来处理这个问题。虽然吸收状态使现有的算法和API处理终止代理无需修改,实际的培训效率和资源使用的问题存在。在这项工作中,我们首先证明了样本的复杂性增加的数量的吸收状态的玩具监督学习任务的一个完全连接的网络,而注意力是更强大的可变大小的输入。然后,我们提出了一种新的架构,现有的国家的最先进的MARL算法,它使用的注意力,而不是一个完全连接的层吸收状态。最后,我们证明了这种新的架构显着优于标准架构的任务,其中代理人创建或销毁的情节,以及标准的多代理协调任务。

英文摘要:

The creation and destruction of agents in cooperative multi-agent reinforcement learning (MARL) is a critically under-explored area of research. Current MARL algorithms often assume that the number of agents within a group remains fixed throughout an experiment. However, in many practical problems, an agent may terminate before their teammates. This early termination issue presents a challenge: the terminated agent must learn from the group's success or failure which occurs beyond its own existence. We refer to propagating value from rewards earned by remaining teammates to terminated agents as the Posthumous Credit Assignment problem. Current MARL methods handle this problem by placing these agents in an absorbing state until the entire group of agents reaches a termination condition. Although absorbing states enable existing algorithms and APIs to handle terminated agents without modification, practical training efficiency and resource use problems exist. In this work, we first demonstrate that sample complexity increases with the quantity of absorbing states in a toy supervised learning task for a fully connected network, while attention is more robust to variable size input. Then, we present a novel architecture for an existing state-of-the-art MARL algorithm which uses attention instead of a fully connected layer with absorbing states. Finally, we demonstrate that this novel architecture significantly outperforms the standard architecture on tasks in which agents are created or destroyed within episodes as well as standard multi-agent coordination tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑