arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2111.00345cs.AIcs.MA
收藏

多智能体顾问Q-Learning

Multi-Agent Advisor Q-Learning

University of Waterloo(滑铁卢大学);Vector Institute(向量研究所);University of Alberta(阿尔伯塔大学);Alberta Machine Intelligence Institute (Amii)(阿尔伯塔机器智能研究所(Amii))

Sriram Ganapathi Subramanian, Matthew E. Taylor, Kate Larson, Mark Crowley

摘要:

在过去的十年中,多智能体强化学习(MARL)取得了重大进展,但仍然存在许多挑战,例如高样本复杂性和收敛到稳定策略的速度慢,需要在广泛部署之前加以克服。然而,许多现实世界的环境中,在实践中,部署次优或启发式的方法来生成策略。一个有趣的问题是如何最好地使用这些方法作为顾问,以帮助改善多智能体领域的强化学习。在本文中,我们提供了一个原则性的框架,将行动建议,从在线次优顾问在多代理设置。我们描述了在非限制性一般和随机博弈环境中ADVISING多智能强化代理(ADMIRAL)的问题,并提出了两种新的基于Q学习的算法:ADMIRAL -决策(ADMIRAL-DM)和ADMIRAL -顾问评估(ADMIRAL-AE),这使我们能够通过适当地纳入顾问(ADMIRAL-DM)的建议来改善学习,并评估顾问(ADMIRAL-AE)的有效性。我们从理论上分析的算法,并提供不动点保证他们的学习一般和随机游戏。此外,大量的实验表明,这些算法:可以在各种环境中使用,具有性能相比其他相关的基线,可以扩展到大的状态-动作空间,并从顾问的意见不佳是强大的。

英文摘要:

In the last decade, there have been significant advances in multi-agent reinforcement learning (MARL) but there are still numerous challenges, such as high sample complexity and slow convergence to stable policies, that need to be overcome before wide-spread deployment is possible. However, many real-world environments already, in practice, deploy sub-optimal or heuristic approaches for generating policies. An interesting question that arises is how to best use such approaches as advisors to help improve reinforcement learning in multi-agent domains. In this paper, we provide a principled framework for incorporating action recommendations from online sub-optimal advisors in multi-agent settings. We describe the problem of ADvising Multiple Intelligent Reinforcement Agents (ADMIRAL) in nonrestrictive general-sum stochastic game environments and present two novel Q-learning based algorithms: ADMIRAL - Decision Making (ADMIRAL-DM) and ADMIRAL - Advisor Evaluation (ADMIRAL-AE), which allow us to improve learning by appropriately incorporating advice from an advisor (ADMIRAL-DM), and evaluate the effectiveness of an advisor (ADMIRAL-AE). We analyze the algorithms theoretically and provide fixed-point guarantees regarding their learning in general-sum stochastic games. Furthermore, extensive experiments illustrate that these algorithms: can be used in a variety of environments, have performances that compare favourably to other related baselines, can scale to large state-action spaces, and are robust to poor advice from advisors.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑