均场扩散器：将离线多智能体强化学习扩展到数千个智能体
Mean-Field Diffuser: Scaling Offline MARL to Thousands of Agents
AI总结：
提出MF-Diffuser框架，通过将轨迹规划提升到轨迹分布的Wasserstein空间，并利用混沌传播和分层粗到细策略，将离线多智能体强化学习扩展到数千个智能体，理论证明均场近似误差为$O(H^2/\sqrt{N})$且离线分布偏移不随群体规模增长，实验在三个均场RL基准上取得最佳回报。
摘要：
基于扩散的规划在单智能体离线强化学习中取得了显著成果，但由于联合轨迹空间的维度灾难，扩展到多智能体系统仍然难以处理。我们引入了MF-Diffuser，这是一个将轨迹规划提升到轨迹分布的Wasserstein空间的框架，其中混沌传播确保一个小的代表性智能体子集能够捕捉整个群体的动态。我们的方法包括一个值加权的混沌熵目标，该目标协调生成保真度与回报最大化，以及一个分层粗到细策略，在去噪过程中逐步增加智能体群体。我们建立了端到端的次优性界，包含四个可解释项，揭示了均场近似误差以$O(H^2/\sqrt{N})$缩放，而离线分布偏移被证明不会随群体规模$N$增长，并证明生成的策略是一个近似的均场纳什均衡，具有明确的收敛保证。在三个均场RL基准（包括阶段博弈、序列动态和对抗性团队竞争）上的实验表明，MF-Diffuser在大多数设置中实现了最佳回报，在次优离线数据和极端规模（$N \geq 10^3$）下增益最大。
英文摘要：
Diffusion-based planning has achieved strong results in single-agent offline reinforcement learning, yet scaling to many-agent systems remains intractable due to the curse of dimensionality in the joint trajectory space. We introduce MF-Diffuser, a framework that lifts trajectory planning to the Wasserstein space of trajectory distributions, where the propagation of chaos ensures a small representative subset of agents captures the full population dynamics. Our approach features a value-weighted chaotic entropy objective that reconciles generative fidelity with return maximization, and a hierarchical coarse-to-fine strategy that progressively grows the agent population during denoising. We establish end-to-end suboptimality bounds with four interpretable terms, revealing that mean-field approximation error scales as $O(H^2/\sqrt{N})$ while offline distribution shift provably does not grow with population size $N$, and prove the generated policy is an approximate mean-field Nash equilibrium with explicit convergence guarantees. Experiments on three mean-field RL benchmarks -- spanning stage games, sequential dynamics, and adversarial team competition -- show MF-Diffuser achieves the best return in the majority of settings, with the largest gains on suboptimal offline data and at extreme scales ($N \geq 10^3$).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
