arXiv 2205.144952023-05-17cs.LG

任务无关的持续强化学习：获取洞见与克服挑战

Task-Agnostic Continual Reinforcement Learning: Gaining Insights and Overcoming Challenges

Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；Cleanlab；Seoul National University（首尔国立大学）；HEC Montréal（蒙特利尔高等商学院）；Canada CIFAR AI Chair（加拿大CIFAR人工智能讲席）；Amazon Web Services（亚马逊云科技）

Massimo Caccia, Jonas Mueller, Taesup Kim, Laurent Charlin, Rasool Fakoor

发表信息：: CoLLAs 2023

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文研究了任务无关持续学习与多任务学习的性能差异，提出基于回放的循环强化学习（3RL）方法，在合成任务和Meta-World基准上验证其优势，发现3RL在高维度等挑战性设置中优于基线甚至多任务方法，并揭示任务无关方法在资源受限环境中的潜力。

摘要：

持续学习（CL）使得模型和智能体能够从一系列任务中学习，同时解决标准深度学习方法的局限性，例如灾难性遗忘。在本研究中，我们调查了导致任务无关CL与多任务（MTL）智能体之间性能差异的因素。我们提出两个假设：（1）在数据有限、计算受限或高维度的设置中，任务无关方法可能具有优势；（2）更快的适应能力在持续学习设置中可能特别有益，有助于减轻灾难性遗忘的影响。为了调查这些假设，我们引入了一种基于回放的循环强化学习（3RL）方法，用于任务无关CL智能体。我们在一个合成任务和包含50个独特操作任务的Meta-World基准上评估了3RL。我们的结果表明，3RL优于基线方法，甚至在高维度的挑战性设置中能够超越其多任务等价物。我们还表明，循环任务无关智能体始终优于或匹配其基于transformer的对应物的性能。这些发现为任务无关CL相对于任务感知MTL方法的优势提供了洞见，并突出了任务无关方法在资源受限、高维度和多任务环境中的潜力。

英文摘要：

Continual learning (CL) enables the development of models and agents that learn from a sequence of tasks while addressing the limitations of standard deep learning approaches, such as catastrophic forgetting. In this work, we investigate the factors that contribute to the performance differences between task-agnostic CL and multi-task (MTL) agents. We pose two hypotheses: (1) task-agnostic methods might provide advantages in settings with limited data, computation, or high dimensionality, and (2) faster adaptation may be particularly beneficial in continual learning settings, helping to mitigate the effects of catastrophic forgetting. To investigate these hypotheses, we introduce a replay-based recurrent reinforcement learning (3RL) methodology for task-agnostic CL agents. We assess 3RL on a synthetic task and the Meta-World benchmark, which includes 50 unique manipulation tasks. Our results demonstrate that 3RL outperforms baseline methods and can even surpass its multi-task equivalent in challenging settings with high dimensionality. We also show that the recurrent task-agnostic agent consistently outperforms or matches the performance of its transformer-based counterpart. These findings provide insights into the advantages of task-agnostic CL over task-aware MTL approaches and highlight the potential of task-agnostic methods in resource-constrained, high-dimensional, and multi-task environments.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。