arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.02733cs.LGcs.AIcs.MA
收藏

CURO:针对相对过泛化的课程学习

CURO: Curriculum Learning for Relative Overgeneralization

University of Michigan(密歇根大学);University College London(伦敦大学学院);University of Liverpool(利物浦大学)

Lin Shi, Qiyuan Liu, Bei Peng

AI总结:

针对合作型多智能体任务中的相对过泛化(RO)问题,提出课程学习方法CURO,通过微调奖励生成源任务并结合值函数与缓冲区迁移,有效提升了QMIX等算法的协调性能。

摘要:

相对过泛化(RO)是一种在合作型多智能体任务中可能出现的病态现象,当最优联合动作的效用低于次优联合动作时就会发生。RO可能导致智能体陷入局部最优,或者无法解决需要在给定时间步内进行大量智能体协调的合作任务。在本研究中,我们通过实证发现,在多智能体强化学习(MARL)中,基于价值和策略梯度的MARL算法都可能遭受RO影响,且无法学习到有效的协调策略。为了更好地克服RO,我们提出了一种名为针对相对过泛化的课程学习(CURO)的新方法。为了解决表现出强RO的目标任务,在CURO中,我们首先微调该目标任务的奖励函数以生成源任务来训练智能体。然后,为了有效地将一个任务中获取的知识迁移到下一个任务,我们使用了一种结合值函数迁移与缓冲区迁移的迁移学习方法,这使得在目标任务中的探索更加高效。CURO具有通用性,可应用于基于价值和策略梯度的MARL方法。我们证明,当应用于QMIX、HAPPO和HATRPO时,CURO能够成功克服严重的RO,实现性能提升,并在多种具有挑战性的合作型多智能体任务中优于基线方法。

英文摘要:

Relative overgeneralization (RO) is a pathology that can arise in cooperative multi-agent tasks when the optimal joint action's utility falls below that of a sub-optimal joint action. RO can cause the agents to get stuck into local optima or fail to solve cooperative tasks requiring significant coordination between agents within a given timestep. In this work, we empirically find that, in multi-agent reinforcement learning (MARL), both value-based and policy gradient MARL algorithms can suffer from RO and fail to learn effective coordination policies. To better overcome RO, we propose a novel approach called curriculum learning for relative overgeneralization (CURO). To solve a target task that exhibits strong RO, in CURO, we first fine-tune the reward function of the target task to generate source tasks to train the agent. Then, to effectively transfer the knowledge acquired in one task to the next, we use a transfer learning method that combines value function transfer with buffer transfer, which enables more efficient exploration in the target task. CURO is general and can be applied to both value-based and policy gradient MARL methods. We demonstrate that, when applied to QMIX, HAPPO, and HATRPO, CURO can successfully overcome severe RO, achieve improved performance, and outperform baseline methods in a variety of challenging cooperative multi-agent tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑