arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.06274cs.LG
收藏

多智能体强化学习中的集中式训练与混合执行

Centralized Training with Hybrid Execution in Multi-Agent Reinforcement Learning

INESC-ID(INESC-ID(系统与计算机工程研究所));Instituto Superior Técnico, University of Lisbon(里斯本大学高等技术学院)

Pedro P. Santos, Diogo S. Carvalho, Miguel Vasco, Alberto Sardinha, Pedro A. Santos, Ana Paiva, Francisco S. Melo

AI总结:

本文提出混合执行范式与 hybrid-POMDP 形式化,并通过集中训练的自回归预测方法 MARO 估计缺失观测,使智能体在任意通信水平下稳健协作。

摘要:

我们在多智能体强化学习(MARL)中引入混合执行,这是一种新范式:智能体在执行时利用彼此之间的信息共享,力求在任意通信水平下成功完成协作任务。在混合执行下,通信水平可以从智能体之间不允许通信的设置(完全去中心化),到具有完全通信的设置(完全集中化),但智能体事先并不知道自己在执行时将遇到哪种通信水平。为形式化该设置,我们定义了一类新的多智能体部分可观测马尔可夫决策过程(POMDP),并将其命名为 hybrid-POMDP,它显式建模智能体之间的通信过程。我们提出 MARO,该方法采用以集中方式训练的自回归预测模型,在执行时估计缺失的智能体观测。我们在标准场景以及为强调 MARL 中部分可观测性的负面影响而定制的既往基准扩展上评估 MARO。实验结果表明,我们的方法始终优于相关基线,使智能体能够在通信故障时采取行动,同时成功利用共享信息。

英文摘要:

We introduce hybrid execution in multi-agent reinforcement learning (MARL), a new paradigm in which agents aim to successfully complete cooperative tasks with arbitrary communication levels at execution time by taking advantage of information-sharing among the agents. Under hybrid execution, the communication level can range from a setting in which no communication is allowed between agents (fully decentralized), to a setting featuring full communication (fully centralized), but the agents do not know beforehand which communication level they will encounter at execution time. To formalize our setting, we define a new class of multi-agent partially observable Markov decision processes (POMDPs) that we name hybrid-POMDPs, which explicitly model a communication process between the agents. We contribute MARO, an approach that makes use of an auto-regressive predictive model, trained in a centralized manner, to estimate missing agents' observations at execution time. We evaluate MARO on standard scenarios and extensions of previous benchmarks tailored to emphasize the negative impact of partial observability in MARL. Experimental results show that our method consistently outperforms relevant baselines, allowing agents to act with faulty communication while successfully exploiting shared information.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑