离线重训练用于在线强化学习：解耦策略学习以缓解探索偏差
Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
AI总结：
提出OOO框架，通过解耦探索与利用策略，利用离线重训练缓解在线RL中的探索偏差，显著提升微调与在线性能。
摘要：
在在线强化学习（RL）或微调过程中，策略乐观地探索新状态和行为是可取的，尤其是当先前的离线数据未提供足够的状态覆盖时。然而，探索奖励可能会使学习到的策略产生偏差，我们的实验发现，天真但标准地使用此类奖励可能无法恢复出性能良好的策略。与此同时，离线RL中的悲观训练已能够从静态数据集中恢复出性能良好的策略。我们能否利用离线RL从在线交互中恢复出更好的策略？我们做了一个简单的观察：可以在所有交互数据上从头训练一个策略，使用悲观目标，从而将用于数据收集和评估的策略解耦。具体来说，我们提出了离线重训练，即在我们的离线到在线到离线（OOO）强化学习框架中，在线微调结束时进行策略提取步骤。一个乐观（探索）策略用于与环境交互，而一个单独的悲观（利用）策略在所有观测数据上训练用于评估。这种解耦可以减少评估策略中来自在线交互（内在奖励、首要偏差）的任何偏差，并允许在线交互期间进行更多探索行为，这反过来可以为利用生成更好的数据。OOO与多种离线到在线RL和在线RL方法互补，在我们的微调实验中将其平均性能提高了14%至26%，在D4RL基准的多个环境中达到了最先进的性能，并在两个OpenAI gym环境中将在线RL性能提高了165%。此外，OOO可以从不完整的离线数据集中进行微调，而先前的方法可能无法恢复出性能良好的策略。实现：https://github.com/MaxSobolMark/OOO
英文摘要：
It is desirable for policies to optimistically explore new states and behaviors during online reinforcement learning (RL) or fine-tuning, especially when prior offline data does not provide enough state coverage. However, exploration bonuses can bias the learned policy, and our experiments find that naive, yet standard use of such bonuses can fail to recover a performant policy. Concurrently, pessimistic training in offline RL has enabled recovery of performant policies from static datasets. Can we leverage offline RL to recover better policies from online interaction? We make a simple observation that a policy can be trained from scratch on all interaction data with pessimistic objectives, thereby decoupling the policies used for data collection and for evaluation. Specifically, we propose offline retraining, a policy extraction step at the end of online fine-tuning in our Offline-to-Online-to-Offline (OOO) framework for reinforcement learning (RL). An optimistic (exploration) policy is used to interact with the environment, and a separate pessimistic (exploitation) policy is trained on all the observed data for evaluation. Such decoupling can reduce any bias from online interaction (intrinsic rewards, primacy bias) in the evaluation policy, and can allow more exploratory behaviors during online interaction which in turn can generate better data for exploitation. OOO is complementary to several offline-to-online RL and online RL methods, and improves their average performance by 14% to 26% in our fine-tuning experiments, achieves state-of-the-art performance on several environments in the D4RL benchmarks, and improves online RL performance by 165% on two OpenAI gym environments. Further, OOO can enable fine-tuning from incomplete offline datasets where prior methods can fail to recover a performant policy. Implementation: https://github.com/MaxSobolMark/OOO
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
