arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1810.07792cs.LGcs.DCstat.ML
收藏

线性收敛的多智能体完全分散价值函数学习

Multi-Agent Fully Decentralized Value Function Learning with Linear Convergence Rates

Ecole Polytechnique Federale de Lausanne (EPFL)(洛桑联邦理工学院)

Lucas Cassano, Kun Yuan, Ali H. Sayed

摘要:

这项工作开发了一个完全分散的多智能体算法的政策评估。所提出的方案可以应用于两个不同的场景。在第一种情况下,一组代理具有在同一环境的不同实例中遵循不同行为策略(不需要其中任何一个来探索完整的状态空间)收集的不同数据集,并且它们都协作以评估共同的目标策略。网络方法允许有效的探索状态空间,并允许所有代理收敛到最优解,即使在没有合作的情况下,代理也不能自己收敛。第二种情况是多智能体博弈,其中状态是全局的,奖励是局部的。在此方案中,代理协作估计目标组策略的价值函数。该算法结合了离策略学习、资格迹和线性函数逼近。所提出的算法是方差减少的类型,并实现线性收敛与$O(1)$内存的要求。分析了算法的线性收敛性,并通过仿真验证了算法的有效性。

英文摘要:

This work develops a fully decentralized multi-agent algorithm for policy evaluation. The proposed scheme can be applied to two distinct scenarios. In the first scenario, a collection of agents have distinct datasets gathered following different behavior policies (none of which is required to explore the full state space) in different instances of the same environment and they all collaborate to evaluate a common target policy. The network approach allows for efficient exploration of the state space and allows all agents to converge to the optimal solution even in situations where neither agent can converge on its own without cooperation. The second scenario is that of multi-agent games, in which the state is global and rewards are local. In this scenario, agents collaborate to estimate the value function of a target team policy. The proposed algorithm combines off-policy learning, eligibility traces and linear function approximation. The proposed algorithm is of the variance-reduced kind and achieves linear convergence with $O(1)$ memory requirements. The linear convergence of the algorithm is established analytically, and simulations are used to illustrate the effectiveness of the method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑