arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.09792cs.LGcs.MA
收藏

超越单调性:重新审视多智能体Q学习中的分解原理

Beyond Monotonicity: Revisiting Factorization Principles in Multi-Agent Q-Learning

Department of Computer Science, University of Exeter, U.K.(埃克塞特大学计算机科学系);School of Automation, Central South University, China(中南大学自动化学院)

Tianmeng Hu, Yongzheng Cui, Rui Tang, Biao Luo, Ke Li

AI总结:

本文通过动力系统分析非单调值分解,证明非单调分解能可靠恢复IGM最优解并优于单调基线。

摘要:

本文通过动力系统分析非单调值分解,证明非单调分解能可靠恢复IGM最优解并优于单调基线。

英文摘要:

Value decomposition is a central approach in multi-agent reinforcement learning (MARL), enabling centralized training with decentralized execution by factorizing the global value function into local values. To ensure individual-global-max (IGM) consistency, existing methods either enforce monotonicity constraints, which limit expressive power, or adopt softer surrogates at the cost of algorithmic complexity. In this work, we present a dynamical systems analysis of non-monotonic value decomposition, modeling learning dynamics as continuous-time gradient flow. We prove that, under approximately greedy exploration, all zero-loss equilibria violating IGM consistency are unstable saddle points, while only IGM-consistent solutions are stable attractors of the learning dynamics. Extensive experiments on both synthetic matrix games and challenging MARL benchmarks demonstrate that unconstrained, non-monotonic factorization reliably recovers IGM-optimal solutions and consistently outperforms monotonic baselines. Additionally, we investigate the influence of temporal-difference targets and exploration strategies, providing actionable insights for the design of future value-based MARL algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑