无监督到在线强化学习
Unsupervised-to-Online Reinforcement Learning
AI总结：
本文提出无监督到在线强化学习（U2O RL），用无监督离线RL替代特定领域的有监督离线RL，支持单一预训练模型多任务重用，并在九个环境中展现出匹配或超越传统方法的性能与稳定性。
摘要：
离线到在线强化学习（RL）是一种先用离线RL训练策略，再用在线RL进一步微调的框架，一直被认为是数据驱动决策的有效方法。尽管合理，但该框架存在缺陷：它需要为每个任务进行特定领域的离线RL预训练，且在实践中往往很脆弱。在本研究中，我们提出无监督到在线RL（U2O RL），用无监督离线RL替代特定领域的有监督离线RL，作为离线到在线RL的更佳替代方案。U2O RL不仅能为多个下游任务重用单一预训练模型，还能学习更好的表示，通常比有监督离线到在线RL带来更好的性能和稳定性。为了在实践中实例化U2O RL，我们提出了一种通用方案，将任务无关的无监督离线技能策略预训练与有监督在线微调相结合。通过在九个基于状态和基于像素的环境中进行实验，我们实证表明，U2O RL取得了与先前离线到在线RL方法相当甚至更优的强劲性能，同时能够为多个不同下游任务重用单一预训练模型。
英文摘要：
Offline-to-online reinforcement learning (RL), a framework that trains a policy with offline RL and then further fine-tunes it with online RL, has been considered a promising recipe for data-driven decision-making. While sensible, this framework has drawbacks: it requires domain-specific offline RL pre-training for each task, and is often brittle in practice. In this work, we propose unsupervised-to-online RL (U2O RL), which replaces domain-specific supervised offline RL with unsupervised offline RL, as a better alternative to offline-to-online RL. U2O RL not only enables reusing a single pre-trained model for multiple downstream tasks, but also learns better representations, which often result in even better performance and stability than supervised offline-to-online RL. To instantiate U2O RL in practice, we propose a general recipe for U2O RL to bridge task-agnostic unsupervised offline skill-based policy pre-training and supervised online fine-tuning. Throughout our experiments in nine state-based and pixel-based environments, we empirically demonstrate that U2O RL achieves strong performance that matches or even outperforms previous offline-to-online RL approaches, while being able to reuse a single pre-trained model for a number of different downstream tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
