基于样本内顺序策略优化的离线多智能体强化学习
Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization
AI总结：
针对离线多智能体强化学习中的分布外联合动作和次优收敛问题，提出顺序更新策略的 InSPO 算法，避免 OOD 动作并增强协调，理论保证单调改进与收敛于 QRE。
摘要：
Offline Multi-Agent Reinforcement Learning (MARL) 旨在从预先收集的数据集中学习最优的多智能体策略。与单智能体场景相比，多智能体设定涉及庞大的联合状态-动作空间以及多个智能体的耦合行为，这给离线策略优化带来了额外的复杂性。在本研究中，我们重新审视了现有的离线 MARL 方法，并表明在某些场景下它们可能存在问题，会导致不协调的行为和分布外 (OOD) 联合动作。为了解决这些问题，我们提出了一种新的离线 MARL 算法，命名为 In-Sample Sequential Policy Optimization (InSPO)。InSPO 以样本内 (in-sample) 的方式顺序更新每个智能体的策略，这不仅避免了选择 OOD 联合动作，还仔细考虑了队友已更新的策略以增强协调。此外，通过充分探索行为策略中的低概率动作，InSPO 能够很好地解决过早收敛到次优解的问题。在理论上，我们证明了 InSPO 能保证策略的单调改进，并收敛于量化响应均衡 (QRE)。实验结果表明，与当前最先进的离线 MARL 方法相比，我们的方法具有有效性。
英文摘要：
Offline Multi-Agent Reinforcement Learning (MARL) is an emerging field that aims to learn optimal multi-agent policies from pre-collected datasets. Compared to single-agent case, multi-agent setting involves a large joint state-action space and coupled behaviors of multiple agents, which bring extra complexity to offline policy optimization. In this work, we revisit the existing offline MARL methods and show that in certain scenarios they can be problematic, leading to uncoordinated behaviors and out-of-distribution (OOD) joint actions. To address these issues, we propose a new offline MARL algorithm, named In-Sample Sequential Policy Optimization (InSPO). InSPO sequentially updates each agent's policy in an in-sample manner, which not only avoids selecting OOD joint actions but also carefully considers teammates' updated policies to enhance coordination. Additionally, by thoroughly exploring low-probability actions in the behavior policy, InSPO can well address the issue of premature convergence to sub-optimal solutions. Theoretically, we prove InSPO guarantees monotonic policy improvement and converges to quantal response equilibrium (QRE). Experimental results demonstrate the effectiveness of our method compared to current state-of-the-art offline MARL methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
