合作多智能体强化学习中潜在最优联合动作识别
Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning
AI总结：
针对值函数分解中单调性约束限制表达能力的问题，提出潜在最优联合动作加权方法，通过迭代加权训练保证最优策略恢复，在多个任务上超越现有方法。
摘要：
值函数分解在合作多智能体强化学习（MARL）中被广泛使用。现有方法通常对联合动作值与个体动作值之间施加单调性约束以实现分散执行。然而，此类约束限制了值函数分解的表达能力，缩小了可表示的联合动作值范围，并阻碍了最优策略的学习。为解决这一问题，我们提出了潜在最优联合动作加权（POW）方法，该方法在现有近似加权策略可能失效的情况下确保最优策略恢复。POW通过一个理论上有依据的迭代加权训练过程，迭代地识别潜在最优联合动作并为其分配更高的训练权重。我们证明该机制保证了真实最优策略的恢复，克服了先前启发式加权策略的局限性。POW是架构无关的，可以无缝集成到现有的值函数分解算法中。在矩阵博弈、难度增强的捕食者-猎物任务、SMAC、SMACv2以及高速公路环境交叉口场景上的大量实验表明，POW显著提升了稳定性，并持续超越了最先进的基于值的MARL方法。
英文摘要：
Value function factorization is widely used in cooperative multi-agent reinforcement learning (MARL). Existing approaches often impose monotonicity constraints between the joint action value and individual action values to enable decentralized execution. However, such constraints limit the expressiveness of value factorization, restricting the range of joint action values that can be represented and hindering the learning of optimal policies. To address this, we propose Potentially Optimal Joint Actions Weighting (POW), a method that ensures optimal policy recovery where existing approximate weighting strategies may fail. POW iteratively identifies potentially optimal joint actions and assigns them higher training weights through a theoretically grounded iterative weighted training process. We prove that this mechanism guarantees recovery of the true optimal policy, overcoming the limitations of prior heuristic weighting strategies. POW is architecture-agnostic and can be seamlessly integrated into existing value factorization algorithms. Extensive experiments on matrix games, difficulty-enhanced predator-prey tasks, SMAC, SMACv2, and a highway-env intersection scenario show that POW substantially improves stability and consistently surpasses state-of-the-art value-based MARL methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
