离线到在线强化学习中的分布外探索规划
Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning
AI总结：
针对离线到在线强化学习中原有约束限制策略性能、现有探索方法存在缺陷的问题，提出非短视的PTGOOD算法，瞄准高奖励分布外区域探索，在连续控制任务上显著提升微调回报并避免次优收敛。
摘要：
使用静态数据集进行离线预训练、随后进行在线微调的范式（离线到在线，简称OtO）与现实世界的强化学习部署流程高度契合。在该场景下，我们的目标是在有限的在线交互预算内找到性能最优的策略。此前OtO场景下的研究主要聚焦于修正离线强化学习算法的策略约束机制所引入的偏差。这类约束会让学习到的策略与采集数据集的行为策略保持接近，但我们的研究表明，若行为策略与最优策略差距较大，这类约束会不必要地限制策略性能。\n对此，我们放弃了约束机制，将OtO强化学习框架化为一个探索问题，目标是最大化在线数据采集的收益。我们首先研究了OtO场景下基于内在奖励和上置信界（UCB）的主流在线强化学习探索方法，发现内在奖励会通过修改奖励函数增加训练的不稳定性，而UCB方法存在短视性，且尚不清楚应使用哪个学习组件的集成来进行动作选择。\n随后我们提出了分布外探索规划算法（PTGOOD）以规避上述问题。PTGOOD采用非短视的规划流程，瞄准状态-动作空间中行为策略不太可能访问、但奖励相对较高的区域开展探索。通过借鉴条件熵瓶颈（Conditional Entropy Bottleneck）的相关概念，PTGOOD在不改变奖励的前提下，促使在线采集的数据提供与改进最终部署策略相关的新信息。我们在多项连续控制任务上的实验表明，PTGOOD显著提升了智能体在在线微调期间的回报，并且避免了许多基线方法在多个环境中出现的次优策略收敛问题。
英文摘要：
Offline pretraining with a static dataset followed by online fine-tuning (offline-to-online, or OtO) is a paradigm well matched to a real-world RL deployment process. In this scenario, we aim to find the best-performing policy within a limited budget of online interactions. Previous work in the OtO setting has focused on correcting for bias introduced by the policy-constraint mechanisms of offline RL algorithms. Such constraints keep the learned policy close to the behavior policy that collected the dataset, but we show this can unnecessarily limit policy performance if the behavior policy is far from optimal. Instead, we forgo constraints and frame OtO RL as an exploration problem that aims to maximize the benefit of online data-collection. We first study the major online RL exploration methods based on intrinsic rewards and UCB in the OtO setting, showing that intrinsic rewards add training instability through reward-function modification, and UCB methods are myopic and it is unclear which learned-component's ensemble to use for action selection. We then introduce an algorithm for planning to go out-of-distribution (PTGOOD) that avoids these issues. PTGOOD uses a non-myopic planning procedure that targets exploration in relatively high-reward regions of the state-action space unlikely to be visited by the behavior policy. By leveraging concepts from the Conditional Entropy Bottleneck, PTGOOD encourages data collected online to provide new information relevant to improving the final deployment policy without altering rewards. We show empirically in several continuous control tasks that PTGOOD significantly improves agent returns during online fine-tuning and avoids the suboptimal policy convergence that many of our baselines exhibit in several environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
