SACHA:基于启发式注意力的软演员-评论家方法用于部分可观测多智能体路径规划
SACHA: Soft Actor-Critic with Heuristic-Based Attention for Partially Observable Multi-Agent Path Finding
AI总结:
本文提出SACHA方法,通过基于启发式的注意力机制改进多智能体演员-评论家框架,以智能体为中心的评论家实现更公正的信用分配和更好泛化,并在多种MAPF实例上验证了其在成功率和解质量上的提升。
摘要:
多智能体路径规划(MAPF)是许多大规模机器人系统的关键组成部分,其中智能体必须规划出到达给定目标位置的无碰撞路径。近年来,多智能体强化学习被引入以解决MAPF的部分可观测变体,其通过基于每个智能体的部分观测,以集中式方式学习去中心化的单智能体策略。然而,由于该设置的非平稳性,现有的基于学习的方法在实现复杂的多智能体协作方面效果不佳,尤其是在拥挤环境中。为应对这一挑战,我们提出了一种名为基于启发式注意力的软演员-评论家(SACHA)的多智能体演员-评论家方法,该方法为演员和评论家采用了新颖的基于启发式的注意力机制,以鼓励智能体之间的协作。SACHA为每个智能体学习一个神经网络,使其选择性地关注其视野范围内多个智能体的最短路径启发式引导,从而实现更具可扩展性的协作学习。SACHA还通过引入以每个智能体为中心的新颖评论家来近似Q值,从而扩展了现有的多智能体演员-评论家框架。与使用完全可观测评论家的现有方法相比,我们以智能体为中心的多智能体演员-评论家方法实现了更公正的信用分配,并使学习到的策略对具有不同智能体数量和不同环境类型的MAPF实例具有更好的泛化能力。我们还实现了SACHA(C),它在智能体的策略网络中嵌入了一个通信模块,以实现智能体之间的信息交换。我们在多种MAPF实例上评估了SACHA和SACHA(C),并证明了它们在成功率和解决方案质量方面相较于几种最先进的基于学习的MAPF方法有显著提升。
英文摘要:
Multi-Agent Path Finding (MAPF) is a crucial component for many large-scale robotic systems, where agents must plan their collision-free paths to their given goal positions. Recently, multi-agent reinforcement learning has been introduced to solve the partially observable variant of MAPF by learning a decentralized single-agent policy in a centralized fashion based on each agent's partial observation. However, existing learning-based methods are ineffective in achieving complex multi-agent cooperation, especially in congested environments, due to the non-stationarity of this setting. To tackle this challenge, we propose a multi-agent actor-critic method called Soft Actor-Critic with Heuristic-Based Attention (SACHA), which employs novel heuristic-based attention mechanisms for both the actors and critics to encourage cooperation among agents. SACHA learns a neural network for each agent to selectively pay attention to the shortest path heuristic guidance from multiple agents within its field of view, thereby allowing for more scalable learning of cooperation. SACHA also extends the existing multi-agent actor-critic framework by introducing a novel critic centered on each agent to approximate $Q$-values. Compared to existing methods that use a fully observable critic, our agent-centered multi-agent actor-critic method results in more impartial credit assignment and better generalizability of the learned policy to MAPF instances with varying numbers of agents and types of environments. We also implement SACHA(C), which embeds a communication module in the agent's policy network to enable information exchange among agents. We evaluate both SACHA and SACHA(C) on a variety of MAPF instances and demonstrate decent improvements over several state-of-the-art learning-based MAPF methods with respect to success rate and solution quality.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。