仅使用离线预训练策略的高效在线强化学习微调
Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only
AI总结：
本文提出PORL方法，仅利用离线预训练策略进行在线强化学习微调，无需预训练Q函数，通过在线阶段快速初始化Q函数避免悲观估计，达到与先进算法相当的性能，并支持直接微调行为克隆策略。
摘要：
通过在线强化学习（RL）提升预训练策略的性能是一个关键且具有挑战性的课题。现有的在线RL微调方法需要继续使用离线预训练的Q函数进行训练，以保证稳定性和性能。然而，由于大多数离线RL方法具有保守性，这些离线预训练的Q函数通常会低估离线数据集之外的状态-动作对，这阻碍了从离线环境过渡到在线环境时的进一步探索。此外，这一要求限制了它们在仅提供预训练策略而缺少预训练Q函数场景中的适用性，例如在模仿学习（IL）预训练中。为解决这些挑战，我们提出了一种仅使用离线预训练策略进行高效在线RL微调的方法，消除了对预训练Q函数的依赖。我们引入了PORL（仅策略的强化学习微调），该方法在在线阶段从头快速初始化Q函数，以避免有害的悲观估计。我们的方法不仅在与先进的离线到在线RL算法以及利用数据或策略先验的在线RL方法的竞争中取得了具有竞争力的性能，而且为直接微调行为克隆（BC）策略开辟了一条新路径。
英文摘要：
Improving the performance of pre-trained policies through online reinforcement learning (RL) is a critical yet challenging topic. Existing online RL fine-tuning methods require continued training with offline pretrained Q-functions for stability and performance. However, these offline pretrained Q-functions commonly underestimate state-action pairs beyond the offline dataset due to the conservatism in most offline RL methods, which hinders further exploration when transitioning from the offline to the online setting. Additionally, this requirement limits their applicability in scenarios where only pre-trained policies are available but pre-trained Q-functions are absent, such as in imitation learning (IL) pre-training. To address these challenges, we propose a method for efficient online RL fine-tuning using solely the offline pre-trained policy, eliminating reliance on pre-trained Q-functions. We introduce PORL (Policy-Only Reinforcement Learning Fine-Tuning), which rapidly initializes the Q-function from scratch during the online phase to avoid detrimental pessimism. Our method not only achieves competitive performance with advanced offline-to-online RL algorithms and online RL approaches that leverage data or policies prior, but also pioneers a new path for directly fine-tuning behavior cloning (BC) policies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
