面向平均场博弈的可扩展离线强化学习
Scalable Offline Reinforcement Learning for Mean Field Games
AI总结：
针对平均场博弈中在线交互与环境动态不可得的问题，提出纯离线 Off-MMD 算法，结合镜像下降、重要性采样和离线 RL 技术估计均衡策略，并验证其性能与稳健性。
摘要：
用于平均场博弈的强化学习算法为在大量交互智能体中优化策略提供了一种可扩展框架。现有方法通常依赖在线交互或对系统动态的访问，这限制了它们在真实场景中的实用性，因为在此类场景中，这些交互不可行或难以建模。本文提出 Offline Munchausen Mirror Descent（Off-MMD），这是一种新颖的平均场强化学习算法，仅使用离线数据来近似平均场博弈中的均衡策略。通过利用迭代镜像下降和重要性采样技术，Off-MMD 从静态数据集中估计平均场分布，而无需依赖仿真或环境动态。此外，本文结合离线强化学习中的技术来解决 Q 值过估计等常见问题，即使在数据覆盖有限的情况下也能确保稳健的策略学习。该算法可扩展到复杂环境，并在人群探索或导航等基准任务上表现出强劲性能，突显了其适用于无法进行在线实验的真实多智能体系统。本文通过实验证明了 Off-MMD 对低质量数据集的稳健性，并通过实验研究了其对超参数选择的敏感性。
英文摘要：
Reinforcement learning algorithms for mean-field games offer a scalable framework for optimizing policies in large populations of interacting agents. Existing methods often depend on online interactions or access to system dynamics, limiting their practicality in real-world scenarios where such interactions are infeasible or difficult to model. In this paper, we present Offline Munchausen Mirror Descent (Off-MMD), a novel mean-field RL algorithm that approximates equilibrium policies in mean-field games using purely offline data. By leveraging iterative mirror descent and importance sampling techniques, Off-MMD estimates the mean-field distribution from static datasets without relying on simulation or environment dynamics. Additionally, we incorporate techniques from offline reinforcement learning to address common issues like Q-value overestimation, ensuring robust policy learning even with limited data coverage. Our algorithm scales to complex environments and demonstrates strong performance on benchmark tasks like crowd exploration or navigation, highlighting its applicability to real-world multi-agent systems where online experimentation is infeasible. We empirically demonstrate the robustness of Off-MMD to low-quality datasets and conduct experiments to investigate its sensitivity to hyperparameter choices.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
