QSIM：通过动作相似性加权Q学习缓解多智能体强化学习中的过估计
QSIM: Mitigating Overestimation in Multi-Agent Reinforcement Learning via Action Similarity Weighted Q-Learning
AI总结：
QSIM通过动作相似性加权Q学习缓解多智能体强化学习中的过估计问题，提升学习稳定性与性能。
摘要：
价值分解（VD）方法在合作多智能体强化学习（MARL）中取得了显著成功。然而，它们依赖于最大操作符进行时差（TD）目标计算，导致系统性的Q值过估计。这个问题在MARL中尤为严重，因为联合动作空间的指数爆炸通常导致学习不稳定和次优策略。为了解决这个问题，我们提出了QSIM，一种基于动作相似性加权的Q学习框架，该框架利用动作相似性重构TD目标。与直接使用贪婪联合动作不同，QSIM在结构化的近贪婪联合动作空间上形成相似性加权的期望。这种公式允许目标整合来自多样化但行为相关的动作的Q值，同时赋予更相似于贪婪选择的动作更大的影响。通过用结构相关替代方案平滑目标，QSIM有效缓解了过估计并提高了学习稳定性。广泛的实验表明，QSIM可以无缝集成到各种VD方法中，一致地比原始算法产生更优的性能和稳定性。此外，实证分析证实，QSIM显著减轻了MARL中的系统性价值过估计。代码可在https://github.com/MaoMaoLYJ/pymarl-qsim上获得。
英文摘要：
Value decomposition (VD) methods have achieved remarkable success in cooperative multi-agent reinforcement learning (MARL). However, their reliance on the max operator for temporal-difference (TD) target calculation leads to systematic Q-value overestimation. This issue is particularly severe in MARL due to the combinatorial explosion of the joint action space, which often results in unstable learning and suboptimal policies. To address this problem, we propose QSIM, a similarity weighted Q-learning framework that reconstructs the TD target using action similarity. Instead of using the greedy joint action directly, QSIM forms a similarity weighted expectation over a structured near-greedy joint action space. This formulation allows the target to integrate Q-values from diverse yet behaviorally related actions while assigning greater influence to those that are more similar to the greedy choice. By smoothing the target with structurally relevant alternatives, QSIM effectively mitigates overestimation and improves learning stability. Extensive experiments demonstrate that QSIM can be seamlessly integrated with various VD methods, consistently yielding superior performance and stability compared to the original algorithms. Furthermore, empirical analysis confirms that QSIM significantly mitigates the systematic value overestimation in MARL. Code is available at https://github.com/MaoMaoLYJ/pymarl-qsim.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
