FedFormer：强化学习中基于注意力的上下文联邦
FedFormer: Contextual Federation with Attention in Reinforcement Learning
AI总结：
针对多智能体联邦强化学习的权重聚合问题，提出 FedFormer，用 Transformer Attention 按上下文加权聚合模型嵌入，在 Meta-World 上优于 FedAvg 和单智能体 SAC。
摘要：
多智能体联邦强化学习中的一个核心问题，是如何聚合来自多个智能体的见解。通常的做法是将每个参与智能体的模型权重取平均，形成一个共同模型（FedAvg）。与之不同，我们提出 FedFormer，这是一种新颖的联邦策略，利用 Transformer Attention 对来自不同学习器智能体模型的嵌入进行上下文聚合。通过这种方式，我们根据当前智能体的环境和已学习到的关系，对其他智能体的贡献进行注意力加权，从而提供更有效、更高效的联邦。我们在 Meta-World 环境上评估所提方法，发现该方法相比 FedAvg 和非联邦的 Soft Actor-Critic 单智能体方法取得显著改进。与 Soft Actor-Critic 相比的结果表明，FedFormer 获得更高的回合回报，同时仍遵守联邦学习的隐私约束。最后，我们还证明在某些任务中，随着所有方法的智能体池扩大，有效性会提升；与此形成对比的是，FedAvg 在扩展时未能取得明显改进。
英文摘要：
A core issue in multi-agent federated reinforcement learning is defining how to aggregate insights from multiple agents. This is commonly done by taking the average of each participating agent's model weights into one common model (FedAvg). We instead propose FedFormer, a novel federation strategy that utilizes Transformer Attention to contextually aggregate embeddings from models originating from different learner agents. In so doing, we attentively weigh the contributions of other agents with respect to the current agent's environment and learned relationships, thus providing a more effective and efficient federation. We evaluate our methods on the Meta-World environment and find that our approach yields significant improvements over FedAvg and non-federated Soft Actor-Critic single-agent methods. Our results compared to Soft Actor-Critic show that FedFormer achieves higher episodic return while still abiding by the privacy constraints of federated learning. Finally, we also demonstrate improvements in effectiveness with increased agent pools across all methods in certain tasks. This is contrasted by FedAvg, which fails to make noticeable improvements when scaled.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
