Reward-agnostic Fine-tuning：混合强化学习的可证明统计优势
Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning
AI总结：
本文研究混合强化学习设置，提出一种三阶段算法，在数据收集时无需奖励信息，通过单策略部分集中性概念指导离线与在线数据结合，在样本复杂度上超越纯离线和纯在线RL。
摘要：
本文研究表格型强化学习（RL）的混合设置，假设可访问离线数据集和与未知环境的在线交互。核心问题归结为如何高效利用在线数据收集来增强并补充离线数据集，从而实现有效的策略微调。利用reward-agnostic探索和基于模型的离线RL的最新进展，我们设计了一种三阶段混合RL算法，在样本复杂度上胜过两个纯世界——纯离线RL和纯在线RL。所提算法在数据收集期间不需要任何奖励信息。我们的理论基于名为单策略部分集中性的新概念，它刻画了分布不匹配与覆盖不足之间的权衡，并指导离线与在线数据间的相互作用。
英文摘要：
This paper studies tabular reinforcement learning (RL) in the hybrid setting, which assumes access to both an offline dataset and online interactions with the unknown environment. A central question boils down to how to efficiently utilize online data collection to strengthen and complement the offline dataset and enable effective policy fine-tuning. Leveraging recent advances in reward-agnostic exploration and model-based offline RL, we design a three-stage hybrid RL algorithm that beats the best of both worlds -- pure offline RL and pure online RL -- in terms of sample complexities. The proposed algorithm does not require any reward information during data collection. Our theory is developed based on a new notion called single-policy partial concentrability, which captures the trade-off between distribution mismatch and miscoverage and guides the interplay between offline and online data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
