arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.01188cs.LGcs.MA
收藏

最佳Q学习

Best Possible Q-Learning

Peking University(北京大学)

Jiechuan Jiang, Zongqing Lu

摘要:

完全分散的学习,其中的全球信息,即,其他智能体的行为是不可访问的,这是协作多智能体强化学习中的一个基本挑战。然而,大多数分散算法的收敛性和最优性在理论上没有保证,因为所有代理同时更新策略时,转移概率是非平稳的。为了解决这一挑战,我们提出了最佳可能的运营商,一个新的分散运营商,并证明了代理的政策将收敛到最优的联合政策,如果每个代理独立更新其个人的状态动作值的运营商。进一步,为了使更新更有效和实用,我们简化了操作,并证明了收敛性和最优性仍然保持与简化的。通过实例化的简化操作,派生的完全分散的算法,最佳可能的Q学习(BQL),不受非平稳性。从经验上讲,我们表明,BQL实现了显着的改善基线在各种合作的多智能体任务。

英文摘要:

Fully decentralized learning, where the global information, i.e., the actions of other agents, is inaccessible, is a fundamental challenge in cooperative multi-agent reinforcement learning. However, the convergence and optimality of most decentralized algorithms are not theoretically guaranteed, since the transition probabilities are non-stationary as all agents are updating policies simultaneously. To tackle this challenge, we propose best possible operator, a novel decentralized operator, and prove that the policies of agents will converge to the optimal joint policy if each agent independently updates its individual state-action value by the operator. Further, to make the update more efficient and practical, we simplify the operator and prove that the convergence and optimality still hold with the simplified one. By instantiating the simplified operator, the derived fully decentralized algorithm, best possible Q-learning (BQL), does not suffer from non-stationarity. Empirically, we show that BQL achieves remarkable improvement over baselines in a variety of cooperative multi-agent tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑