基于耦合值分解的离线多智能体强化学习
Offline Multi-Agent Reinforcement Learning with Coupled Value Factorization
AI总结：
针对现有离线多智能体强化学习方法未充分利用可分解结构、复杂任务表现不佳的问题，提出带耦合值分解的OMAC算法，通过值函数分解、样本内学习避免分布偏移，在星际争霸II微操任务上性能超越现有SOTA方法。
摘要：
离线强化学习（RL）无需与环境交互即可从离线数据集中学习策略，近年来受到广泛关注。与单智能体场景下的丰富研究相比，离线多智能体强化学习仍是一个探索相对不足的领域。现有大多数方法直接将离线强化学习的组件应用于多智能体场景，未充分利用问题的可分解结构，导致在复杂任务中表现不尽如人意。我们提出了OMAC——一种新型的带耦合值分解的离线多智能体强化学习算法。OMAC采用耦合值分解方案，将全局值函数分解为局部分量和共享分量，同时保持状态值函数与Q值函数之间的信用分配一致性。此外，OMAC对分解后的局部状态值函数执行样本内学习，这一操作在局部层面隐式执行了max-Q运算，同时避免了因评估分布外动作而导致的分布偏移。通过在离线多智能体《星际争霸II》微操任务上的综合评估，我们证明了OMAC的性能优于当前最先进的离线多智能体强化学习方法。
英文摘要：
Offline reinforcement learning (RL) that learns policies from offline datasets without environment interaction has received considerable attention in recent years. Compared with the rich literature in the single-agent case, offline multi-agent RL is still a relatively underexplored area. Most existing methods directly apply offline RL ingredients in the multi-agent setting without fully leveraging the decomposable problem structure, leading to less satisfactory performance in complex tasks. We present OMAC, a new offline multi-agent RL algorithm with coupled value factorization. OMAC adopts a coupled value factorization scheme that decomposes the global value function into local and shared components, and also maintains the credit assignment consistency between the state-value and Q-value functions. Moreover, OMAC performs in-sample learning on the decomposed local state-value functions, which implicitly conducts max-Q operation at the local level while avoiding distributional shift caused by evaluating out-of-distribution actions. Based on the comprehensive evaluations of the offline multi-agent StarCraft II micro-management tasks, we demonstrate the superior performance of OMAC over the state-of-the-art offline multi-agent RL methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
