arXiv 2509.057352025-09-06cs.LGcs.AI

基于模型的强化学习中的离线学习与在线学习：数据收集策略的经验教训

Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategies

University of Tübingen（蒂宾根大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zurich（苏黎世联邦理工学院）

Jiaqi Chen, Ji Shi, Cansu Sancaktar, Jonas Frey, Georg Martius

备注：: Accepted at Reinforcement Learning Conference (RLC 2025); Code available at: https://github.com/swsychen/Offline_vs_Online_in_MBRL

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究比较了基于模型的强化学习中离线与在线数据收集策略，发现离线智能体因测试时分布外状态而性能下降，可通过额外在线交互或加入探索数据缓解，建议大型数据集收集时纳入探索数据。

摘要：

数据收集对于在基于模型的强化学习中学习鲁棒的世界模型至关重要。最普遍的策略是在在线训练期间通过与环境交互来主动收集轨迹，或者在离线数据集上进行训练。乍一看，学习与任务无关的环境动态这一性质使得世界模型成为有效离线训练的良好候选者。然而，在文献中，在线与离线数据对世界模型以及由此产生的任务性能的影响尚未得到深入研究。在这项工作中，我们研究了基于模型设置中的两种范式，在31个不同的环境中进行了实验。首先，我们展示了在线智能体优于其离线对应物。我们确定了离线智能体性能下降背后的一个关键挑战：在测试时遇到分布外状态。这个问题之所以出现，是因为没有在线智能体中的自我纠正机制，状态空间覆盖有限的离线数据集会导致智能体的想象与实际轨迹之间的不匹配，从而损害策略训练。我们证明，通过允许在固定或自适应调度中进行额外的在线交互，可以缓解这个问题，从而以有限的交互数据恢复在线训练的性能。我们还展示了纳入探索数据有助于缓解离线智能体的性能下降。基于我们的见解，我们建议在收集大型数据集时添加探索数据，因为当前的努力主要集中于专家数据。

英文摘要：

Data collection is crucial for learning robust world models in model-based reinforcement learning. The most prevalent strategies are to actively collect trajectories by interacting with the environment during online training or training on offline datasets. At first glance, the nature of learning task-agnostic environment dynamics makes world models a good candidate for effective offline training. However, the effects of online vs. offline data on world models and thus on the resulting task performance have not been thoroughly studied in the literature. In this work, we investigate both paradigms in model-based settings, conducting experiments on 31 different environments. First, we showcase that online agents outperform their offline counterparts. We identify a key challenge behind performance degradation of offline agents: encountering Out-Of-Distribution states at test time. This issue arises because, without the self-correction mechanism in online agents, offline datasets with limited state space coverage induce a mismatch between the agent's imagination and real rollouts, compromising policy training. We demonstrate that this issue can be mitigated by allowing for additional online interactions in a fixed or adaptive schedule, restoring the performance of online training with limited interaction data. We also showcase that incorporating exploration data helps mitigate the performance degradation of offline agents. Based on our insights, we recommend adding exploration data when collecting large datasets, as current efforts predominantly focus on expert data alone.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。