arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1910.05366cs.LGstat.ML
收藏

通过通信最小化学习几乎可分解的价值函数

Learning Nearly Decomposable Value Functions Via Communication Minimization

Tsinghua University(清华大学);Turing AI Institute of Nanjing(南京图灵人工智能研究院)

Tonghan Wang, Jianhao Wang, Chongyi Zheng, Chongjie Zhang

摘要:

强化学习在多智能体环境中遇到了重大挑战,例如可扩展性和非平稳性。最近,价值函数因子分解学习出现了一个很有前途的方式来解决这些挑战的协作多智能体系统。然而,现有的方法一直专注于学习完全分散的价值函数,这对于需要通信的任务来说是没有效率的。为了解决这一问题,本文提出了一种新的框架,通过通信最小化学习几乎可分解的Q函数(NDQ),与代理自己的大部分时间,但偶尔发送消息给其他代理,以有效的协调。该框架通过引入两个信息论正则化器,将价值函数分解学习和交流学习混合起来。这些正则化器最大化代理的动作选择和通信消息之间的互信息,同时最小化代理之间的消息的熵。我们展示了如何优化这些正则化的方式,很容易与现有的价值函数因式分解方法,如QMIX集成。最后,我们证明,在星际单位微观管理基准,我们的框架显着优于基线方法,并允许我们切断超过$80\%$的通信,而不牺牲性能。我们实验的视频可以在https://sites.google.com/view/ndq上找到。

英文摘要:

Reinforcement learning encounters major challenges in multi-agent settings, such as scalability and non-stationarity. Recently, value function factorization learning emerges as a promising way to address these challenges in collaborative multi-agent systems. However, existing methods have been focusing on learning fully decentralized value functions, which are not efficient for tasks requiring communication. To address this limitation, this paper presents a novel framework for learning nearly decomposable Q-functions (NDQ) via communication minimization, with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination. This framework hybridizes value function factorization learning and communication learning by introducing two information-theoretic regularizers. These regularizers are maximizing mutual information between agents' action selection and communication messages while minimizing the entropy of messages between agents. We show how to optimize these regularizers in a way that is easily integrated with existing value function factorization methods such as QMIX. Finally, we demonstrate that, on the StarCraft unit micromanagement benchmark, our framework significantly outperforms baseline methods and allows us to cut off more than $80\%$ of communication without sacrificing the performance. The videos of our experiments are available at https://sites.google.com/view/ndq.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑