一种离线到在线强化学习的非整体式策略方法
A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learning
AI总结：
针对现有离线到在线强化学习方法过度探索导致在线策略学习不足的问题，提出一种非整体式探索的新方法，在不修改离线策略的前提下协调其利用优势与在线策略的探索优势，性能优于PEX方法。
摘要：
离线到在线强化学习（RL）同时利用预训练的离线策略和为下游任务训练的在线策略，旨在提升数据效率并加速性能提升。现有方法Policy Expansion（PEX，策略扩展）使用由两种策略组成的策略集，在不修改离线策略的情况下进行探索和学习。然而，该方法因过度关注两种策略的探索，无法确保在线策略得到充分学习。由于预训练的离线策略可凭借其先验经验辅助在线策略对下游任务进行利用，它应当被有效执行，并适配下游任务的特定需求。相比之下，行为策略尚不成熟的在线策略在训练阶段具备探索潜力。因此，本研究聚焦于在不修改离线策略的前提下，协调离线策略的优势（即利用）与在线策略的优势（即探索）。本研究提出了一种采用非整体式探索方法的创新离线到在线RL方法，该方法的性能表现优于PEX。
英文摘要：
Offline-to-online reinforcement learning (RL) leverages both pre-trained offline policies and online policies trained for downstream tasks, aiming to improve data efficiency and accelerate performance enhancement. An existing approach, Policy Expansion (PEX), utilizes a policy set composed of both policies without modifying the offline policy for exploration and learning. However, this approach fails to ensure sufficient learning of the online policy due to an excessive focus on exploration with both policies. Since the pre-trained offline policy can assist the online policy in exploiting a downstream task based on its prior experience, it should be executed effectively and tailored to the specific requirements of the downstream task. In contrast, the online policy, with its immature behavioral strategy, has the potential for exploration during the training phase. Therefore, our research focuses on harmonizing the advantages of the offline policy, termed exploitation, with those of the online policy, referred to as exploration, without modifying the offline policy. In this study, we propose an innovative offline-to-online RL method that employs a non-monolithic exploration approach. Our methodology demonstrates superior performance compared to PEX.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
