arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2506.07548cs.AIcs.RO
收藏

用于多智能体强化学习的反事实群体相对策略优势课程学习

Curriculum Learning With Counterfactual Group Relative Policy Advantage For Multi-Agent Reinforcement Learning

School of Information and Communications Engineering, Xi'an Jiaotong University(西安交通大学信息与通信工程学院);Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程系);School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);Department of Electrical and Electronic Engineering, The University of Hong Kong(香港大学电子与电气工程系)

Weiqiang Jin, Hongyang Du, Guizhong Liu, Dong In Kim

AI总结:

针对MARL在动态对抗环境中的适应与信用分配难题,提出结合自适应难度课程和CGRPA反事实优势的框架,提升非平稳条件下的训练稳定性与最终性能。

摘要:

多智能体强化学习(MARL)已在合作对抗任务中取得优异性能。然而,现有大多数方法通常让智能体与固定对手策略进行训练,并依赖这种元静态难度条件,这限制了它们对变化环境的适应能力,并往往导致次优策略。受课程学习(CL)在监督任务中成功的启发,我们提出了一种用于MARL的动态CL框架,该框架采用自适应难度调节机制。该机制根据智能体的实时训练表现持续调节对手强度,使智能体能够逐步从较简单场景学习到更具挑战性的场景。然而,由于非平稳环境和稀疏全局奖励,CL的动态特性会引入不稳定性。为应对这一挑战,我们开发了反事实群体相对策略优势(CGRPA),它通过提供反映每个智能体在不断演变的任务需求下影响的内在信用信号,与课程紧密耦合。CGRPA构建反事实优势函数,在群体行为中分离个体贡献,从而在整个课程中促进更可靠的策略更新。CGRPA通过构建反事实动作优势函数评估每个智能体的贡献,提供内在奖励,以增强信用分配并稳定非平稳条件下的学习。大量实验表明,我们的方法同时提升了训练稳定性和最终性能,在与最先进方法的比较中取得了具有竞争力的结果。代码可在https://github.com/NICE-HKU/CL2MARL-SMAC获取。

英文摘要:

Multi-agent reinforcement learning (MARL) has achieved strong performance in cooperative adversarial tasks. However, most existing methods typically train agents against fixed opponent strategies and rely on such meta-static difficulty conditions, which limits their adaptability to changing environments and often leads to suboptimal policies. Inspired by the success of curriculum learning (CL) in supervised tasks, we propose a dynamic CL framework for MARL that employs an self-adaptive difficulty adjustment mechanism. This mechanism continuously modulates opponent strength based on real-time agent training performance, allowing agents to progressively learn from easier to more challenging scenarios. However, the dynamic nature of CL introduces instability due to nonstationary environments and sparse global rewards. To address this challenge, we develop a Counterfactual Group Relative Policy Advantage (CGRPA), which is tightly coupled with the curriculum by providing intrinsic credit signals that reflect each agent's impact under evolving task demands. CGRPA constructs a counterfactual advantage function that isolates individual contributions within group behavior, facilitating more reliable policy updates throughout the curriculum. CGRPA evaluates each agent's contribution through constructing counterfactual action advantage function, providing intrinsic rewards that enhance credit assignment and stabilize learning under non-stationary conditions. Extensive experiments demonstrate that our method improves both training stability and final performance, achieving competitive results against state-of-the-art methods. The code is available at https://github.com/NICE-HKU/CL2MARL-SMAC.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑