Offline-Boosted Actor-Critic：在深度离策略强化学习中自适应融合最优历史行为
Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
AI总结：
针对离策略RL未充分利用replay buffer信息的问题，提出无模型在线RL框架OBAC，通过价值比较识别更优离线策略作为自适应约束来增强在线学习，在53个任务上展现出卓越的样本效率和性能。
摘要：
离策略强化学习（RL）通过利用先前收集的数据进行策略学习，在处理许多复杂的现实世界任务中取得了显著成功。然而，大多数现有的离策略RL算法未能最大程度地利用replay buffer中的信息，限制了样本效率和策略性能。在本研究中，我们发现，基于共享的在线replay buffer同时训练一个离线RL策略，有时能优于原始的在线学习策略，尽管这种性能提升的出现仍存在不确定性。这启发了一种新的可能性，即利用涌现的更优离线最优策略来改进在线策略学习。基于这一洞察，我们提出了Offline-Boosted Actor-Critic（OBAC），这是一个无模型在线RL框架，它通过价值比较优雅地识别出表现更优的离线策略，并将其用作自适应约束，以保证更强的策略学习性能。我们的实验表明，在跨越6个任务集的53项任务中，OBAC在样本效率和渐近性能方面优于其他流行的无模型RL基线，并与先进的基于模型的RL方法相媲美。
英文摘要：
Off-policy reinforcement learning (RL) has achieved notable success in tackling many complex real-world tasks, by leveraging previously collected data for policy learning. However, most existing off-policy RL algorithms fail to maximally exploit the information in the replay buffer, limiting sample efficiency and policy performance. In this work, we discover that concurrently training an offline RL policy based on the shared online replay buffer can sometimes outperform the original online learning policy, though the occurrence of such performance gains remains uncertain. This motivates a new possibility of harnessing the emergent outperforming offline optimal policy to improve online policy learning. Based on this insight, we present Offline-Boosted Actor-Critic (OBAC), a model-free online RL framework that elegantly identifies the outperforming offline policy through value comparison, and uses it as an adaptive constraint to guarantee stronger policy learning performance. Our experiments demonstrate that OBAC outperforms other popular model-free RL baselines and rivals advanced model-based RL methods in terms of sample efficiency and asymptotic performance across 53 tasks spanning 6 task suites.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
