SCoUT: 多智能体强化学习中通过效用引导的时间分组实现可扩展通信
SCoUT: Scalable Communication via Utility-Guided Temporal Grouping in Multi-Agent Reinforcement Learning
AI总结:
SCoUT通过效用引导的时间分组在多智能体强化学习中实现可扩展通信,通过软智能体分组和组感知批评者减少复杂性,提升通信学习精度。
摘要:
在部分可观测的多智能体强化学习(MARL)中,通信可以提高协调性,但学习何时和与谁通信需要在众多可能的发送者-接收者对中进行选择,且任何单条消息对未来奖励的影响难以孤立。我们引入SCoUT(通过效用引导的时间分组实现可扩展通信),通过传统MARL中的时间和智能体抽象来解决这两个挑战。在训练过程中,SCoUT每$K$个环境步骤(宏步骤)通过Gumbel-Softmax重新采样软智能体组;这些组是潜在聚类,诱导出一个用于作为可微先验的亲和力。使用相同的分配,一个组感知的批评者为每个智能体组预测值并通过相同的软分配映射到单个智能体的基础线,从而减少批评者复杂性和方差。每个智能体都使用三头策略:环境动作、发送决策和接收者选择。为了获得精确的通信学习信号,我们通过分析地从接收者聚合消息中移除每个发送者的贡献来推导反事实通信优势。这种反事实计算使得发送和接收者选择决策都能获得精确的信用分配。在执行时,所有集中训练组件都被丢弃,仅运行单个智能体策略,保持去中心化执行。项目网站、视频和代码:https://scout-comm.github.io/
英文摘要:
Communication can improve coordination in partially observed multi-agent reinforcement learning (MARL), but learning \emph{when} and \emph{who} to communicate with requires choosing among many possible sender-recipient pairs, and the effect of any single message on future reward is hard to isolate. We introduce \textbf{SCoUT} (\textbf{S}calable \textbf{Co}mmunication via \textbf{U}tility-guided \textbf{T}emporal grouping), which addresses both these challenges via temporal and agent abstraction within traditional MARL. During training, SCoUT resamples \textit{soft} agent groups every \(K\) environment steps (macro-steps) via Gumbel-Softmax; these groups are latent clusters that induce an affinity used as a differentiable prior over recipients. Using the same assignments, a group-aware critic predicts values for each agent group and maps them to per-agent baselines through the same soft assignments, reducing critic complexity and variance. Each agent is trained with a three-headed policy: environment action, send decision, and recipient selection. To obtain precise communication learning signals, we derive counterfactual communication advantages by analytically removing each sender's contribution from the recipient's aggregated messages. This counterfactual computation enables precise credit assignment for both send and recipient-selection decisions. At execution time, all centralized training components are discarded and only the per-agent policy is run, preserving decentralized execution. Project website, videos and code: \hyperlink{https://scout-comm.github.io/}{https://scout-comm.github.io/}
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。