基于全局状态预测的去中心化多智能体强化学习
Decentralized Multi-Agent Reinforcement Learning with Global State Prediction
AI总结：
针对多智能体强化学习的非平稳性难题，提出无需全局信息的全局状态预测网络GSP，在集体运输场景下验证了其性能提升与鲁棒性增强效果。
摘要：
深度强化学习（DRL）已在单机器人控制领域取得显著成效，但将其应用于机器人群体仍面临诸多重大挑战，其中核心难题是非平稳性：当两个及以上机器人同时更新个体或共享策略时，会形成相互依赖的训练过程，无法保证收敛。规避非平稳性通常需要借助其他智能体状态和/或动作的全局信息来训练机器人。\n与现有思路不同，本文探索如何消除对全局信息的依赖。由于无法获取其他智能体的全局知识，我们将问题建模为部分可观测马尔可夫决策过程（Partially Observable Markov Decision Process）。以集体运输为测试场景，我们研究了两种多智能体训练方法：第一种方法中机器人之间不交换任何消息，训练其通过推拉运输物体的方式进行隐式通信；第二种方法中我们提出全局状态预测网络（Global State Prediction, GSP），该网络经过训练可形成对整个机器人群体的信念并预测其未来状态。\n我们在含障碍物的环境中针对四种经典深度强化学习算法开展了全面研究，以在规定时间内成功将物体运输至目标点作为性能衡量标准。通过消融实验我们证明，与使用全局知识的方法相比，加入GSP能够提升性能并增强鲁棒性。
英文摘要：
Deep reinforcement learning (DRL) has seen remarkable success in the control of single robots. However, applying DRL to robot swarms presents significant challenges. A critical challenge is non-stationarity, which occurs when two or more robots update individual or shared policies concurrently, thereby engaging in an interdependent training process with no guarantees of convergence. Circumventing non-stationarity typically involves training the robots with global information about other agents' states and/or actions. In contrast, in this paper we explore how to remove the need for global information. We pose our problem as a Partially Observable Markov Decision Process, due to the absence of global knowledge on other agents. Using collective transport as a testbed scenario, we study two approaches to multi-agent training. In the first, the robots exchange no messages, and are trained to rely on implicit communication through push-and-pull on the object to transport. In the second approach, we introduce Global State Prediction (GSP), a network trained to forma a belief over the swarm as a whole and predict its future states. We provide a comprehensive study over four well-known deep reinforcement learning algorithms in environments with obstacles, measuring performance as the successful transport of the object to the goal within a desired time-frame. Through an ablation study, we show that including GSP boosts performance and increases robustness when compared with methods that use global knowledge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
