Reinforcement learning encounters major challenges in multi-agent settings, such as scalability and non-stationarity. Recently, value function factorization learning emerges as a promising way to address these challenges in collaborative multi-agent systems. However, existing methods have been focusing on learning fully decentralized value functions, which are not efficient for tasks requiring communication. To address this limitation, this paper presents a novel framework for learning nearly decomposable Q-functions (NDQ) via communication minimization, with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination. This framework hybridizes value function factorization learning and communication learning by introducing two information-theoretic regularizers. These regularizers are maximizing mutual information between agents' action selection and communication messages while minimizing the entropy of messages between agents. We show how to optimize these regularizers in a way that is easily integrated with existing value function factorization methods such as QMIX. Finally, we demonstrate that, on the StarCraft unit micromanagement benchmark, our framework significantly outperforms baseline methods and allows us to cut off more than $80\%$ of communication without sacrificing the performance. The videos of our experiments are available at https://sites.google.com/view/ndq.
Fast Decomposable Submodular Function Minimization using Constrained Total Variation
Imperial College London(帝国理工学院);INRIA(法国国家信息与自动化研究所);Ecole normale superieure(巴黎高等师范学院);PSL Research University(巴黎文理研究大学);Graz University of Technology(格拉茨工业大学)
Decomposable Submodular Function Minimization via Maximum Flow
MIT(麻省理工学院);University of Warsaw(华沙大学);CNRS(法国国家科学研究中心);IRIF(法国信息学研究所);Université de Paris(巴黎大学)
本文桥梁离散和连续优化方法可分解的子模函数最小化,在标准和参数设置。我们为这个问题提供了改进的运行时间,通过减少它到一个最大流的Oracle的调用数量。当分解中的每个函数作用在基集$V$的$O(1)$元素上并且多项式有界时,我们的运行时间达到了多项式对数因子,等于在具有$O(\vert V \vert)$顶点和多项式积分容量的稀疏图中求解最大流的时间。我们实现这一点,通过提供一个简单的迭代法,它可以优化到高精度的任何凸函数定义的次模基多面体,只要我们可以有效地减少它的基础多面体对应于一定的图,我们构造的切割功能。我们解决这个最小化问题的参数切割问题,我们通过一个新的有效的组合减少最大流的解决方案。这种减少是独立的利益,并意味着一些以前未知的界限参数最小$s,t$切割问题在多个设置。
Facebook AI Research(脸书人工智能研究院);Cognitive Machine Learning (ENS - EHESS - PSL - CNRS - INRIA)(认知机器学习实验室(巴黎高等师范学院-社会科学高等学院-巴黎文理研究大学-法国国家科学研究中心-法国国家信息与自动化研究所));Catalan Institute for Research and Advanced Studies(加泰罗尼亚研究与高级研究所)
Latency Minimization for UAV-Enabled Federated Learning: Trajectory Design and Resource Allocation
Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen)(深圳未来智能网络研究院);School of Science and Engineering (SSE)(科学与工程学院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Institute of Advanced Technology (SIAT)(深圳先进技术研究院);Chinese Academy of Sciences (CAS)(中国科学院);Southern University of Science and Technology(南方科技大学);Imperial College London(伦敦帝国学院);Central South University(中南大学);Shenzhen University of Advanced Technology(深圳理工大学)