保留次优动作以跟随移动的最优解在多智能体强化学习中
Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning
AI总结:
本文提出S2Q算法,通过学习多个子价值函数来保留替代的高价值动作,以解决多智能体强化学习中适应值函数变化时的最优解移动问题,实验表明其在多智能体强化学习基准上表现优异。
摘要:
价值分解是合作多智能体强化学习(MARL)的核心方法。然而,现有方法仍然依赖于单一最优动作,在训练过程中底层价值函数变化时难以适应,往往收敛到次优策略。为了解决这一限制,我们提出了Successive Sub-value Q-learning(S2Q),该方法学习多个子价值函数以保留替代的高价值动作。将这些子价值函数纳入基于Softmax的行为策略中,S2Q鼓励持续探索并使$Q^{ ext{tot}}$能够快速调整到变化的最优解。在具有挑战性的MARL基准上的实验表明,S2Q在各种MARL算法中始终表现更优,证明了其改进的适应性和整体性能。我们的代码可在https://github.com/hyeon1996/S2Q上获得。
英文摘要:
Value decomposition is a core approach for cooperative multi-agent reinforcement learning (MARL). However, existing methods still rely on a single optimal action and struggle to adapt when the underlying value function shifts during training, often converging to suboptimal policies. To address this limitation, we propose Successive Sub-value Q-learning (S2Q), which learns multiple sub-value functions to retain alternative high-value actions. Incorporating these sub-value functions into a Softmax-based behavior policy, S2Q encourages persistent exploration and enables $Q^{\text{tot}}$ to adjust quickly to the changing optima. Experiments on challenging MARL benchmarks confirm that S2Q consistently outperforms various MARL algorithms, demonstrating improved adaptability and overall performance. Our code is available at https://github.com/hyeon1996/S2Q.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。