可扩展的多智能体离线强化学习与信息的作用
Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information
AI总结：
本文提出SCAM-FQI算法，通过信息共享网络实现可扩展的多智能体离线强化学习，理论证明其能高概率收敛到近最优策略，并在实验中验证了可扩展性与性能的平衡。
摘要：
离线强化学习（RL）专注于仅从一批先前收集的数据中学习策略，从而无需进行昂贵或危险的主动探索即可有效利用此类数据集。尽管离线多智能体强化学习（MARL）的最新进展已显示出前景，但现有方法大多要么依赖所有智能体联合收集的大型数据集，要么依赖各智能体独立收集的智能体特定数据集。前者确保了较强的性能，但引发了可扩展性问题；后者则强调可扩展性，却以牺牲性能保证为代价。在本工作中，我们提出了一种新颖的可扩展流程，用于数据集收集和离线学习。智能体首先按照预先指定的信息共享网络一致地收集多样化数据集，随后学习一致且局部化的策略，而无需完全可观测性，也不必退回到完全去中心化。我们从理论上证明，这种结构化方法使得经典的Fitted Q-Iteration（FQI）算法的多智能体扩展能够以高概率全局收敛到近最优策略。该收敛受到误差项的影响，这些误差项取决于共享信息的信息量。此外，我们展示了该方法如何利用共享信息与未共享信息之间的互信息来界定FQI监督学习阶段固有误差的上界。我们的算法，SCAlable Multi-agent FQI（SCAM-FQI），随后在一个分布式决策问题上进行了评估。实证结果与我们的理论发现一致，支持了SCAM-FQI在实现可扩展性与策略性能之间平衡方面的有效性。
英文摘要：
Offline Reinforcement Learning (RL) focuses on learning policies solely from a batch of previously collected data. offering the potential to leverage such datasets effectively without the need for costly or risky active exploration. While recent advances in Offline Multi-Agent RL (MARL) have shown promise, most existing methods either rely on large datasets jointly collected by all agents or agent-specific datasets collected independently. The former approach ensures strong performance but raises scalability concerns, while the latter emphasizes scalability at the expense of performance guarantees. In this work, we propose a novel scalable routine for both dataset collection and offline learning. Agents first collect diverse datasets coherently with a pre-specified information-sharing network and subsequently learn coherent localized policies without requiring either full observability or falling back to complete decentralization. We theoretically demonstrate that this structured approach allows a multi-agent extension of the seminal Fitted Q-Iteration (FQI) algorithm to globally converge, in high probability, to near-optimal policies. The convergence is subject to error terms that depend on the informativeness of the shared information. Furthermore, we show how this approach allows to bound the inherent error of the supervised-learning phase of FQI with the mutual information between shared and unshared information. Our algorithm, SCAlable Multi-agent FQI (SCAM-FQI), is then evaluated on a distributed decision-making problem. The empirical results align with our theoretical findings, supporting the effectiveness of SCAM-FQI in achieving a balance between scalability and policy performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
