基于模型的强化学习与多任务离线预训练
Model-Based Reinforcement Learning with Multi-Task Offline Pretraining
AI总结：
提出一种基于模型的强化学习方法，通过世界模型衡量任务相关性并利用相似性权重自适应迁移物理动力学知识和源动作演示，显著提升新任务的在线训练效率。
摘要：
在离线数据集上预训练强化学习（RL）模型是提高其在在线任务中训练效率的一种有前景的方法，但由于不同任务之间在动力学和行为上的固有差异，这具有挑战性。我们提出了一种基于模型的RL方法，该方法学习将潜在有用的动力学和动作演示从离线数据迁移到新任务中。主要思想是使用世界模型不仅作为行为学习的模拟器，而且作为工具来衡量动力学表示迁移和策略迁移的任务相关性。我们构建了一个时变的、领域选择性的蒸馏损失，以生成一组离线到在线的相似性权重。这些权重有两个目的：（i）自适应地迁移任务无关的物理动力学知识以促进世界模型训练，以及（ii）学习重放相关的源动作以指导目标策略。我们在Meta-World和DeepMind Control Suite中展示了我们的方法与最先进方法相比的优势。
英文摘要：
Pretraining reinforcement learning (RL) models on offline datasets is a promising way to improve their training efficiency in online tasks, but challenging due to the inherent mismatch in dynamics and behaviors across various tasks. We present a model-based RL method that learns to transfer potentially useful dynamics and action demonstrations from offline data to a novel task. The main idea is to use the world models not only as simulators for behavior learning but also as tools to measure the task relevance for both dynamics representation transfer and policy transfer. We build a time-varying, domain-selective distillation loss to generate a set of offline-to-online similarity weights. These weights serve two purposes: (i) adaptively transferring the task-agnostic knowledge of physical dynamics to facilitate world model training, and (ii) learning to replay relevant source actions to guide the target policy. We demonstrate the advantages of our approach compared with the state-of-the-art methods in Meta-World and DeepMind Control Suite.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
