arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1702.08736cs.MAcs.AI
收藏

多智能体强化学习中的拥塞问题分析

Analysing Congestion Problems in Multi-agent Reinforcement Learning

Roxana Rădulescu, Peter Vrancx, Ann Nowé

摘要:

拥塞问题在当今的复杂网络中无处不在,并且在许多研究领域都是一个挑战。在多智能体强化学习(MARL)的背景下,差异奖励和资源抽象等方法在解决此类问题方面表现出了良好的效果。资源抽象被证明是以完全分散的方式解决大规模资源分配问题的理想候选者。然而,它的性能和适用性在很大程度上取决于一些,直到现在,没有记录的假设。在文献中考虑的两个主要的拥堵基准问题是:海滩问题域和交通车道域。在这两种设置中,当过度拥挤一个资源并保持其余资源处于最佳容量时,可以实现最高的系统效用。我们分析了抽象分组如何促进这种行为,以及在现实世界的领域中应用这种方法的可行性(即,需要满足哪些假设以及需要哪些知识)。我们引入了一个新的测试问题,道路网络域(RND),其中资源不再是独立的,而是网络的一部分(例如,道路网络),因此选择一条路径也会影响具有共同路段的其他路径的负载。我们展示了应用国家的最先进的MARL方法,这个新的拥塞模型,并分析其性能。RND使我们能够突出资源抽象的一个重要限制,并表明差异奖励方法能够更好地捕获和通知代理环境的动态。

英文摘要:

Congestion problems are omnipresent in today's complex networks and represent a challenge in many research domains. In the context of Multi-agent Reinforcement Learning (MARL), approaches like difference rewards and resource abstraction have shown promising results in tackling such problems. Resource abstraction was shown to be an ideal candidate for solving large-scale resource allocation problems in a fully decentralized manner. However, its performance and applicability strongly depends on some, until now, undocumented assumptions. Two of the main congestion benchmark problems considered in the literature are: the Beach Problem Domain and the Traffic Lane Domain. In both settings the highest system utility is achieved when overcrowding one resource and keeping the rest at optimum capacity. We analyse how abstract grouping can promote this behaviour and how feasible it is to apply this approach in a real-world domain (i.e., what assumptions need to be satisfied and what knowledge is necessary). We introduce a new test problem, the Road Network Domain (RND), where the resources are no longer independent, but rather part of a network (e.g., road network), thus choosing one path will also impact the load on other paths having common road segments. We demonstrate the application of state-of-the-art MARL methods for this new congestion model and analyse their performance. RND allows us to highlight an important limitation of resource abstraction and show that the difference rewards approach manages to better capture and inform the agents about the dynamics of the environment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑