arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.08482cs.AI
收藏

多智能体强化学习中最小化通信同时最大化性能

Minimizing Communication while Maximizing Performance in Multi-Agent Reinforcement Learning

Intel Labs(英特尔实验室)

Varun Kumar Vijay, Hassam Sheikh, Somdeb Majumdar, Mariano Phielipp

摘要:

智能体通信可以显着提高性能的多智能体任务,需要协调,以实现一个共同的目标。先前的工作已经表明,可以使用多智能体强化学习和消息传递网络架构来学习智能体间通信协议。然而,这些模型使用不受约束的广播通信模型,其中代理在每一步都与所有其他代理进行通信,即使任务不需要它。在现实世界的应用中,通信可能受到带宽,功率和网络容量等系统约束的限制,可能需要减少发送的消息数量。在这项工作中,我们探索了一种简单的方法,最大限度地减少通信,同时最大限度地提高多任务学习的性能:同时优化特定于任务的目标和通信惩罚。我们表明,目标可以使用增强和Gumbel-Softmax重新参数化进行优化。我们引入了两种技术来稳定训练:50%训练和消息转发。仅在50%的事件上进行通信惩罚的训练可以防止我们的模型关闭他们的传出消息。其次,重复先前接收的消息有助于模型保留信息,并进一步提高性能。通过这些技术,我们可以减少75%的通信,而不会损失性能。

英文摘要:

Inter-agent communication can significantly increase performance in multi-agent tasks that require co-ordination to achieve a shared goal. Prior work has shown that it is possible to learn inter-agent communication protocols using multi-agent reinforcement learning and message-passing network architectures. However, these models use an unconstrained broadcast communication model, in which an agent communicates with all other agents at every step, even when the task does not require it. In real-world applications, where communication may be limited by system constraints like bandwidth, power and network capacity, one might need to reduce the number of messages that are sent. In this work, we explore a simple method of minimizing communication while maximizing performance in multi-task learning: simultaneously optimizing a task-specific objective and a communication penalty. We show that the objectives can be optimized using Reinforce and the Gumbel-Softmax reparameterization. We introduce two techniques to stabilize training: 50% training and message forwarding. Training with the communication penalty on only 50% of the episodes prevents our models from turning off their outgoing messages. Second, repeating messages received previously helps models retain information, and further improves performance. With these techniques, we show that we can reduce communication by 75% with no loss of performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑