PTDE:面向多智能体强化学习的个性化训练与蒸馏执行
PTDE: Personalized Training with Distilled Execution for Multi-Agent Reinforcement Learning
AI总结:
本文提出多智能体强化学习新范式PTDE,通过为每个智能体定制并蒸馏全局信息至局部信息,在去中心化执行时保持性能,显著提升SMAC、GRF和LTR等基准表现。
摘要:
集中式训练与去中心化执行(CTDE)已成为多智能体强化学习中广泛采用的范式,强调利用全局信息来学习增强的联合 $Q$-函数或集中式评论家。相比之下,我们的研究深入探讨了如何利用全局信息来直接增强个体 $Q$-函数或个体智能体。值得注意的是,我们发现将相同的全局信息普遍应用于所有智能体不足以实现最优性能。因此,我们提倡为每个智能体定制全局信息,创建智能体个性化的全局信息以提升整体性能。此外,我们引入了一种名为个性化训练与蒸馏执行(PTDE)的新范式,其中智能体个性化的全局信息被蒸馏到智能体的局部信息中。这些蒸馏出的信息随后在去中心化执行期间使用,从而导致最小的性能下降。PTDE 可以与最先进的算法无缝集成,在包括 SMAC 基准、Google Research Football (GRF) 基准和 Learning to Rank (LTR) 任务在内的多种基准上实现显著的性能提升。
英文摘要:
Centralized Training with Decentralized Execution (CTDE) has emerged as a widely adopted paradigm in multi-agent reinforcement learning, emphasizing the utilization of global information for learning an enhanced joint $Q$-function or centralized critic. In contrast, our investigation delves into harnessing global information to directly enhance individual $Q$-functions or individual actors. Notably, we discover that applying identical global information universally across all agents proves insufficient for optimal performance. Consequently, we advocate for the customization of global information tailored to each agent, creating agent-personalized global information to bolster overall performance. Furthermore, we introduce a novel paradigm named Personalized Training with Distilled Execution (PTDE), wherein agent-personalized global information is distilled into the agent's local information. This distilled information is then utilized during decentralized execution, resulting in minimal performance degradation. PTDE can be seamlessly integrated with state-of-the-art algorithms, leading to notable performance enhancements across diverse benchmarks, including the SMAC benchmark, Google Research Football (GRF) benchmark, and Learning to Rank (LTR) task.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。