使用给定的子任务分解学习复杂的团队协作任务
Learning Complex Teamwork Tasks Using a Given Sub-task Decomposition
AI总结:
提出基于专家子任务分解的多智能体强化学习方法,通过子团队训练、合并与微调高效解决复杂任务,并改进现有actor-critic算法以克服朴素实现的缺陷。
摘要:
由于在大型联合策略空间中进行策略搜索以及相互适应的智能体导致的非平稳性等挑战,通过多智能体强化学习训练团队完成复杂任务可能十分困难。为了促进对复杂多智能体任务的高效学习,我们提出了一种方法,该方法使用专家提供的将任务分解为更简单的多智能体子任务的分解方案。在每个子任务中,训练整个团队的一个子集以获取特定于子任务的策略。然后将这些子团队合并并转移到目标任务中,在其中对其策略进行集体微调以解决更复杂的目标任务。我们凭经验证明,与从零开始训练相比,此类方法可以大大减少解决复杂目标任务所需的时间步数。然而,我们也发现并调查了基于子任务分解的方法的朴素实现所存在的两个问题,并提出了一种简单且可扩展的方法来解决这些问题,该方法增强了现有的 actor-critic 算法。我们展示了所提方法的实证益处,使子任务分解方法能够部署在多样化的多智能体任务中。
英文摘要:
Training a team to complete a complex task via multi-agent reinforcement learning can be difficult due to challenges such as policy search in a large joint policy space, and non-stationarity caused by mutually adapting agents. To facilitate efficient learning of complex multi-agent tasks, we propose an approach which uses an expert-provided decomposition of a task into simpler multi-agent sub-tasks. In each sub-task, a subset of the entire team is trained to acquire sub-task-specific policies. The sub-teams are then merged and transferred to the target task, where their policies are collectively fine-tuned to solve the more complex target task. We show empirically that such approaches can greatly reduce the number of timesteps required to solve a complex target task relative to training from-scratch. However, we also identify and investigate two problems with naive implementations of approaches based on sub-task decomposition, and propose a simple and scalable method to address these problems which augments existing actor-critic algorithms. We demonstrate the empirical benefits of our proposed method, enabling sub-task decomposition approaches to be deployed in diverse multi-agent tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。