arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.06920cs.AIcs.MA
收藏

基于生成流网络的多智能体连续控制

Multi-Agent Continuous Control with Generative Flow Networks

Zhejiang University(浙江大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Xidian University(西安电子科技大学)

Shuang Luo, Yinchuan Li, Shunyu Liu, Xu Zhang, Yunfeng Shao, Chao Wu

AI总结:

针对生成流网络单流约束难以适配多智能体连续联合控制的问题,提出MACFN方法,通过集中式全局流匹配训练分散式单流策略,引入连续流分解网络推导各智能体流贡献,性能优于现有最优方法且探索能力更强。

摘要:

生成流网络(GFlowNets)旨在从轨迹终态与奖励成正比的分布中生成多样化轨迹,是探索性控制任务中强化学习的有力替代方案。然而,GFlowNets中的单流匹配约束限制了其在多智能体系统中的应用,尤其是连续联合控制问题。本文提出一种新颖的多智能体生成连续流网络(MACFN)方法,使多个智能体能够针对各种组合式连续对象开展协作探索。技术层面,MACFN以集中式全局流匹配的方式训练基于分散式单流的策略。在集中训练阶段,MACFN引入连续流分解网络,在仅存在全局奖励的情况下推导每个智能体的流贡献。随后智能体可仅根据分配到的局部流以分散方式执行动作,形成与奖励成正比的联合策略分布。为保证连续流分解的表达能力,我们从理论上推导了分解网络的一致性条件。实验结果表明,所提方法的效果优于现有最优同类方法,且具备更优的探索能力。我们的代码可在https://github.com/isluoshuang/MACFN获取。

英文摘要:

Generative Flow Networks (GFlowNets) aim to generate diverse trajectories from a distribution in which the final states of the trajectories are proportional to the reward, serving as a powerful alternative to reinforcement learning for exploratory control tasks. However, the individual-flow matching constraint in GFlowNets limits their applications for multi-agent systems, especially continuous joint-control problems. In this paper, we propose a novel Multi-Agent generative Continuous Flow Networks (MACFN) method to enable multiple agents to perform cooperative exploration for various compositional continuous objects. Technically, MACFN trains decentralized individual-flow-based policies in a centralized global-flow-based matching fashion. During centralized training, MACFN introduces a continuous flow decomposition network to deduce the flow contributions of each agent in the presence of only global rewards. Then agents can deliver actions solely based on their assigned local flow in a decentralized way, forming a joint policy distribution proportional to the rewards. To guarantee the expressiveness of continuous flow decomposition, we theoretically derive a consistency condition on the decomposition network. Experimental results demonstrate that the proposed method yields results superior to the state-of-the-art counterparts and better exploration capability. Our code is available at https://github.com/isluoshuang/MACFN.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑