ComaDICE：具有稳态分布偏移正则化的离线协作多智能体强化学习
ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization
AI总结：
本文提出ComaDICE算法，通过在稳态分布空间引入正则化并结合多智能体价值分解策略，有效解决离线协作多智能体强化学习中的分布偏移问题，在MuJoCo和StarCraft II基准上取得领先性能。
摘要：
离线强化学习（RL）因能够从预先收集的数据集中学习有效策略而无需进一步与环境交互，已获得广泛关注。尽管在单智能体环境中已展现出有前景的结果，但离线多智能体强化学习（MARL）由于联合状态-动作空间庞大以及多智能体行为的复杂性而面临额外挑战。离线RL中的一个关键问题是分布偏移，即被优化的目标策略偏离生成数据的行为策略时就会出现该问题。在MARL中，由于智能体局部策略之间的相互依赖以及庞大的联合状态-动作空间，这一问题更加严重。先前的方法主要通过将正则化引入Q函数空间或策略空间来应对这一挑战。在本工作中，我们在稳态分布空间中引入正则化器，以更好地处理分布偏移。我们的算法ComaDICE通过为全局学习策略引入稳态分布正则化，并辅以精心设计的多智能体价值分解策略以促进多智能体训练，为离线协作MARL提供了一个有原则的框架。通过在多智能体MuJoCo和StarCraft II基准上的大量实验，我们证明ComaDICE在几乎所有任务上都取得了优于最先进离线MARL方法的性能。
英文摘要：
Offline reinforcement learning (RL) has garnered significant attention for its ability to learn effective policies from pre-collected datasets without the need for further environmental interactions. While promising results have been demonstrated in single-agent settings, offline multi-agent reinforcement learning (MARL) presents additional challenges due to the large joint state-action space and the complexity of multi-agent behaviors. A key issue in offline RL is the distributional shift, which arises when the target policy being optimized deviates from the behavior policy that generated the data. This problem is exacerbated in MARL due to the interdependence between agents' local policies and the expansive joint state-action space. Prior approaches have primarily addressed this challenge by incorporating regularization in the space of either Q-functions or policies. In this work, we introduce a regularizer in the space of stationary distributions to better handle distributional shift. Our algorithm, ComaDICE, offers a principled framework for offline cooperative MARL by incorporating stationary distribution regularization for the global learning policy, complemented by a carefully structured multi-agent value decomposition strategy to facilitate multi-agent training. Through extensive experiments on the multi-agent MuJoCo and StarCraft II benchmarks, we demonstrate that ComaDICE achieves superior performance compared to state-of-the-art offline MARL methods across nearly all tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
