RGMComm：多智能体强化学习中通过离散通信最小化回报间隙
RGMComm: Return Gap Minimization via Discrete Communications in Multi-Agent Reinforcement Learning
AI总结：
针对多智能体强化学习中现有通信方法开销高、无定量回报保证的问题，提出RGMComm算法，通过将通信转化为在线聚类问题最小化回报间隙，实现低开销高可解释的接近最优性能。
摘要：
在部分可观测马尔可夫决策过程中，通信对于解决协作式多智能体强化学习任务至关重要。现有研究通常依赖黑盒方法将局部信息或特征编码为共享给其他智能体的消息，导致生成的连续消息通信开销高且可解释性差。此前的离散通信方法尝试生成作为智能体动作一部分训练的独热向量，并使用Gumbel softmax操作计算消息梯度，这些均为启发式设计，无法对期望回报提供任何定量保证。本文建立了具有完全可观测性的理想策略与采用离散通信的最优部分可观测策略之间回报间隙的上界。这一结果使我们能够将多智能体通信转化为一个针对每个智能体局部观测的新型在线聚类问题，其中消息作为聚类标签，回报间隙上界作为聚类损失。为最小化回报间隙，我们提出了回报间隙最小化通信（Return-Gap-Minimization Communication，RGMComm）算法，该算法采用了一种极为简洁的离散消息生成函数设计，并通过一种新型的正则化信息最大化损失函数与强化学习相结合，该损失函数将余弦距离作为聚类度量。评估结果表明，RGMComm显著优于当前最先进的多智能体通信基线方法，并且仅需使用位数极少、天然具备可解释性的消息即可实现接近最优的回报。
英文摘要：
Communication is crucial for solving cooperative Multi-Agent Reinforcement Learning tasks in partially observable Markov Decision Processes. Existing works often rely on black-box methods to encode local information/features into messages shared with other agents, leading to the generation of continuous messages with high communication overhead and poor interpretability. Prior attempts at discrete communication methods generate one-hot vectors trained as part of agents' actions and use the Gumbel softmax operation for calculating message gradients, which are all heuristic designs that do not provide any quantitative guarantees on the expected return. This paper establishes an upper bound on the return gap between an ideal policy with full observability and an optimal partially observable policy with discrete communication. This result enables us to recast multi-agent communication into a novel online clustering problem over the local observations at each agent, with messages as cluster labels and the upper bound on the return gap as clustering loss. To minimize the return gap, we propose the Return-Gap-Minimization Communication (RGMComm) algorithm, which is a surprisingly simple design of discrete message generation functions and is integrated with reinforcement learning through the utilization of a novel Regularized Information Maximization loss function, which incorporates cosine-distance as the clustering metric. Evaluations show that RGMComm significantly outperforms state-of-the-art multi-agent communication baselines and can achieve nearly optimal returns with few-bit messages that are naturally interpretable.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
