arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.05334cs.LGcs.AIcs.MA
收藏

TransfQMix:利用多智能体强化学习问题的图结构的Transformers

TransfQMix: Transformers for Leveraging the Graph Structure of Multi-Agent Reinforcement Learning Problems

Matteo Gallici, Mario Martin, Ivan Masmitja

摘要:

协调是多智能体强化学习(MARL)中最困难的方面之一。一个原因是,代理通常选择他们的行动彼此独立。为了看到从独立的政策的组合出现的协调策略,最近的研究集中在使用一个集中的功能(CF),学习每个代理的贡献,团队奖励。然而,环境呈现给代理和CF的结构通常被忽视。我们已经观察到,用于描述协调问题的特征可以表示为潜在图结构的顶点特征。在这里,我们提出了TransfQMix,一种新的方法,使用Transformers来利用这种潜在的结构,并学习更好的协调策略。我们的Transformer代理对可观察实体的状态执行图形推理。我们的Transformer Q-mixer从一个更大的图中学习单调混合函数,该图包括代理的内部和外部状态。TransfQMix被设计为完全可转移的,这意味着相同的参数可以用于控制和训练更大或更小的代理团队。这使得部署有前途的方法,以节省培训时间,并在MARL中获得一般政策,如迁移学习,zero-shot迁移,和课程学习。我们报告TransfQMix在Spread和StarCraft II环境中的表现。在这两种情况下,它都优于最先进的Q-Learning模型,并且在解决其他方法无法解决的问题方面表现出了有效性。

英文摘要:

Coordination is one of the most difficult aspects of multi-agent reinforcement learning (MARL). One reason is that agents normally choose their actions independently of one another. In order to see coordination strategies emerging from the combination of independent policies, the recent research has focused on the use of a centralized function (CF) that learns each agent's contribution to the team reward. However, the structure in which the environment is presented to the agents and to the CF is typically overlooked. We have observed that the features used to describe the coordination problem can be represented as vertex features of a latent graph structure. Here, we present TransfQMix, a new approach that uses transformers to leverage this latent structure and learn better coordination policies. Our transformer agents perform a graph reasoning over the state of the observable entities. Our transformer Q-mixer learns a monotonic mixing-function from a larger graph that includes the internal and external states of the agents. TransfQMix is designed to be entirely transferable, meaning that same parameters can be used to control and train larger or smaller teams of agents. This enables to deploy promising approaches to save training time and derive general policies in MARL, such as transfer learning, zero-shot transfer, and curriculum learning. We report TransfQMix's performances in the Spread and StarCraft II environments. In both settings, it outperforms state-of-the-art Q-Learning models, and it demonstrates effectiveness in solving problems that other methods can not solve.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑