解开谜题：用于离线多智能体强化学习的局部到全局世界模型
Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learning
AI总结：
提出局部到全局世界模型，通过局部预测提升全局状态动态推断，结合不确定性采样机制提升离线多智能体强化学习的泛化能力。
摘要：
离线多智能体强化学习（MARL）旨在利用预收集的数据集解决多智能体系统中的协作决策问题。现有离线MARL方法主要限制在数据集分布内进行训练，导致过于保守的策略难以在数据支持范围外泛化。虽然基于模型的方法通过用学习的世界模型生成的合成数据扩展原始数据集提供了一个有前途的解决方案，但多智能体系统的高维性、非平稳性和复杂性使得在离线MARL中准确估计转移和奖励函数具有挑战性。鉴于直接建模联合动态的困难，我们提出了一种局部到全局（LOGO）世界模型，这是一种新的框架，利用局部预测（更容易估计）来推断全局状态动态，从而提高预测精度的同时隐式捕捉智能体间的依赖关系。使用训练好的世界模型，我们生成合成数据以扩展原始数据集，扩大有效状态-动作空间。为了确保可靠的策略学习，我们进一步引入了不确定性感知的采样机制，该机制通过预测不确定性自适应加权合成数据，减少对策略的近似误差传播。与传统基于集合的方法不同，我们的方法只需额外一个编码器进行不确定性估计，显著降低了计算开销同时保持准确性。在8种场景中对8种基线进行广泛实验，证明我们的方法在标准离线MARL基准上超越了最先进的基线，建立了可泛化的离线多智能体学习的新基于模型的基线。
英文摘要：
Offline multi-agent reinforcement learning (MARL) aims to solve cooperative decision-making problems in multi-agent systems using pre-collected datasets. Existing offline MARL methods primarily constrain training within the dataset distribution, resulting in overly conservative policies that struggle to generalize beyond the support of the data. While model-based approaches offer a promising solution by expanding the original dataset with synthetic data generated from a learned world model, the high dimensionality, non-stationarity, and complexity of multi-agent systems make it challenging to accurately estimate the transitions and reward functions in offline MARL. Given the difficulty of directly modeling joint dynamics, we propose a local-to-global (LOGO) world model, a novel framework that leverages local predictions-which are easier to estimate-to infer global state dynamics, thus improving prediction accuracy while implicitly capturing agent-wise dependencies. Using the trained world model, we generate synthetic data to augment the original dataset, expanding the effective state-action space. To ensure reliable policy learning, we further introduce an uncertainty-aware sampling mechanism that adaptively weights synthetic data by prediction uncertainty, reducing approximation error propagation to policies. In contrast to conventional ensemble-based methods, our approach requires only an additional encoder for uncertainty estimation, significantly reducing computational overhead while maintaining accuracy. Extensive experiments across 8 scenarios against 8 baselines demonstrate that our method surpasses state-of-the-art baselines on standard offline MARL benchmarks, establishing a new model-based baseline for generalizable offline multi-agent learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
