面向通用效用的可扩展多智能体强化学习
Scalable Multi-Agent Reinforcement Learning with General Utilities
AI总结：
针对通用效用下多智能体强化学习的完全可观测性限制问题，提出带影子奖励的分布式策略梯度算法，利用空间相关性衰减实现局部策略优化，首次在无完全观测条件下达成ε平稳收敛。
摘要：
我们研究了具有通用效用的可扩展多智能体强化学习（MARL），通用效用定义为团队长期状态-动作占用度量的非线性函数。研究目标是在团队中每个智能体不具备完全可观测性的情况下，找到一种局部化策略，使团队局部效用函数的平均值最大化。通过利用网络结构的空间相关性衰减特性，我们提出了一种带影子奖励和局部化策略的可扩展分布式策略梯度算法，该算法包含三个步骤：（1）影子奖励估计，（2）截断影子Q函数估计，（3）截断策略梯度估计与策略更新。我们的算法以高概率收敛到ε平稳点，样本复杂度为$\tilde{\r\nm{O}}(ε^{-2})$，仅存在随通信半径指数衰减的近似误差。这是文献中首个在通用效用多智能体强化学习场景下无需完全可观测性的研究成果。
英文摘要：
We study the scalable multi-agent reinforcement learning (MARL) with general utilities, defined as nonlinear functions of the team's long-term state-action occupancy measure. The objective is to find a localized policy that maximizes the average of the team's local utility functions without the full observability of each agent in the team. By exploiting the spatial correlation decay property of the network structure, we propose a scalable distributed policy gradient algorithm with shadow reward and localized policy that consists of three steps: (1) shadow reward estimation, (2) truncated shadow Q-function estimation, and (3) truncated policy gradient estimation and policy update. Our algorithm converges, with high probability, to $ε$-stationarity with $\widetilde{\mathcal{O}}(ε^{-2})$ samples up to some approximation error that decreases exponentially in the communication radius. This is the first result in the literature on multi-agent RL with general utilities that does not require the full observability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
