arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.01649cs.MA
收藏

基于注意力的递归方法用于随机部分可观测性下的多智能体强化学习

Attention-Based Recurrence for Multi-Agent Reinforcement Learning under Stochastic Partial Observability

Thomy Phan, Fabian Ritz, Philipp Altmann, Maximilian Zorn, Jonas Nüßlein, Michael Kölle, Thomas Gabor, Claudia Linnhoff-Popien

AI总结:

针对随机部分可观测性下多智能体强化学习,提出基于注意力的递归嵌入(AERIAL)方法,并构建更通用的MessySMAC基准,实验验证其优于基于状态的CTDE。

摘要:

随机部分可观测性对多智能体强化学习中的分散协调构成了重大挑战,但由于当前研究主要关注基于状态的中心化训练与去中心化执行(CTDE),以及如星际争霸多智能体挑战(SMAC)等缺乏足够随机性的基准,这一挑战在很大程度上被忽视了。在本文中,我们提出了基于注意力的多智能体学习递归嵌入(AERIAL)方法,以在随机部分可观测性下近似价值函数。AERIAL用多智能体递归的学习表示替代真实状态,相比基于状态的CTDE,能更准确地考虑去中心化智能体决策的信息。随后,我们引入了MessySMAC,这是SMAC的一个修改版本,具有随机观测和更高的初始状态方差,以提供一个更通用且可配置的随机部分可观测性基准。我们在Dec-Tiger以及多种SMAC和MessySMAC地图上评估了AERIAL,并与基于状态的CTDE进行了结果比较。此外,我们评估了AERIAL和基于状态的CTDE在MessySMAC中各种随机性配置下的鲁棒性。

英文摘要:

Stochastic partial observability poses a major challenge for decentralized coordination in multi-agent reinforcement learning but is largely neglected in state-of-the-art research due to a strong focus on state-based centralized training for decentralized execution (CTDE) and benchmarks that lack sufficient stochasticity like StarCraft Multi-Agent Challenge (SMAC). In this paper, we propose Attention-based Embeddings of Recurrence In multi-Agent Learning (AERIAL) to approximate value functions under stochastic partial observability. AERIAL replaces the true state with a learned representation of multi-agent recurrence, considering more accurate information about decentralized agent decisions than state-based CTDE. We then introduce MessySMAC, a modified version of SMAC with stochastic observations and higher variance in initial states, to provide a more general and configurable benchmark regarding stochastic partial observability. We evaluate AERIAL in Dec-Tiger as well as in a variety of SMAC and MessySMAC maps, and compare the results with state-based CTDE. Furthermore, we evaluate the robustness of AERIAL and state-based CTDE against various stochasticity configurations in MessySMAC.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑