arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.11153cs.LGcs.AIcs.MA
收藏

多智能体强化学习中多个独立顾问的学习

Learning from Multiple Independent Advisors in Multi-agent Reinforcement Learning

Vector Institute(向量研究所);University of Waterloo(滑铁卢大学);University of Alberta(阿尔伯塔大学);Alberta Machine Intelligence Institute(阿尔伯塔机器智能研究所)

Sriram Ganapathi Subramanian, Matthew E. Taylor, Kate Larson, Mark Crowley

摘要:

多智能体强化学习通常会遇到样本效率低下的问题,其中学习合适的策略涉及使用许多数据样本。向外部示范者学习是缓解这一问题的一个可能的解决方案。然而,在这一领域中的大多数先前的方法假设存在单个演示器。利用多个知识来源(即,在环境的不同方面具有专门知识的顾问可以大大加快在复杂环境中的学习。研究了多智能体强化学习中同时向多个独立顾问学习的问题。该方法利用两级Q学习架构,并将该框架从单代理扩展到多代理设置。我们提供了原则性的算法,通过评估每个状态下的顾问,并随后使用顾问来指导动作选择,来合并一组顾问。我们还提供理论收敛性和样本复杂性保证。在实验中,我们在三个不同的测试床中验证了我们的方法,并表明我们的算法比基线提供了更好的性能,可以有效地整合不同顾问的综合专业知识,并学会忽略坏建议。

英文摘要:

Multi-agent reinforcement learning typically suffers from the problem of sample inefficiency, where learning suitable policies involves the use of many data samples. Learning from external demonstrators is a possible solution that mitigates this problem. However, most prior approaches in this area assume the presence of a single demonstrator. Leveraging multiple knowledge sources (i.e., advisors) with expertise in distinct aspects of the environment could substantially speed up learning in complex environments. This paper considers the problem of simultaneously learning from multiple independent advisors in multi-agent reinforcement learning. The approach leverages a two-level Q-learning architecture, and extends this framework from single-agent to multi-agent settings. We provide principled algorithms that incorporate a set of advisors by both evaluating the advisors at each state and subsequently using the advisors to guide action selection. We also provide theoretical convergence and sample complexity guarantees. Experimentally, we validate our approach in three different test-beds and show that our algorithms give better performances than baselines, can effectively integrate the combined expertise of different advisors, and learn to ignore bad advice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑