arXiv 2305.109972023-05-18cs.LGcs.AIcs.DCcs.MA

通过调制掩码共享终身强化学习知识

Sharing Lifelong Reinforcement Learning Knowledge via Modulating Masks

Loughborough University（拉夫堡大学）；University of California, Riverside（加州大学河滨分校）；Vanderbilt University（范德堡大学）

Saptarshi Nath, Christos Peridis, Eseoghene Ben-Iwhiwhu, Xinran Liu, Shirin Dora, Cong Liu, Soheil Kolouri, Andrea Soltoggio

备注：: 25 pages, 14 figures, 9 tables, to be published in the Second Conference on Lifelong Learning Agents (CoLLAs 2023), code can be found at https://github.com/DMIU-ShELL/deeprl-shell

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出利用调制掩码的参数隔离机制，在完全分布式异步多智能体系统中通过按需通信协议共享特定任务知识，实现稳健的分布式终身强化学习，并在连接中断时保持快速学习。

摘要：

终身学习智能体旨在其生命周期内顺序学习多个任务。这包括在学习新任务时利用先前知识以及避免遗忘的能力。调制掩码作为参数隔离方法的一种特定类型，近期在监督学习和强化学习中均展现出潜力。尽管终身学习算法主要在单智能体方法内进行研究，但多个智能体如何相互共享终身学习知识仍是一个问题。我们表明，调制掩码使用的参数隔离机制特别适合在终身学习者的分布式和去中心化系统中进行智能体间的知识交换。关键思想是将特定任务知识隔离到特定掩码中，使智能体能够按需仅传输特定知识，从而实现稳健且有效的分布式终身学习。我们假设处于具有动态智能体数量和连接性的完全分布式和异步场景中。按需通信协议确保智能体在面对每个任务时向其同伴查询特定掩码，以进行传输并整合到其策略中。实验表明，按需掩码通信是实现分布式终身强化学习的有效方式，并且相对于DD-PPO、IMPALA和PPO+EWC等分布式RL基线提供了终身学习收益。该系统对连接中断特别稳健，并因知识交换而展现出快速学习的能力。

英文摘要：

Lifelong learning agents aim to learn multiple tasks sequentially over a lifetime. This involves the ability to exploit previous knowledge when learning new tasks and to avoid forgetting. Modulating masks, a specific type of parameter isolation approach, have recently shown promise in both supervised and reinforcement learning. While lifelong learning algorithms have been investigated mainly within a single-agent approach, a question remains on how multiple agents can share lifelong learning knowledge with each other. We show that the parameter isolation mechanism used by modulating masks is particularly suitable for exchanging knowledge among agents in a distributed and decentralized system of lifelong learners. The key idea is that the isolation of specific task knowledge to specific masks allows agents to transfer only specific knowledge on-demand, resulting in robust and effective distributed lifelong learning. We assume fully distributed and asynchronous scenarios with dynamic agent numbers and connectivity. An on-demand communication protocol ensures agents query their peers for specific masks to be transferred and integrated into their policies when facing each task. Experiments indicate that on-demand mask communication is an effective way to implement distributed lifelong reinforcement learning and provides a lifelong learning benefit with respect to distributed RL baselines such as DD-PPO, IMPALA, and PPO+EWC. The system is particularly robust to connection drops and demonstrates rapid learning due to knowledge exchange.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。