利用Q条件状态熵探索改进离线到在线强化学习
Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Exploration
AI总结：
针对离线到在线强化学习中的分布偏移问题，提出Q条件状态熵（QCSE）作为内在奖励，通过最大化状态熵并隐式实现状态边际匹配，提升在线微调性能，实验显示在CQL和Cal-QL上分别提升约13%和8%。
摘要：
研究如何微调离线强化学习（RL）预训练策略，对于提高强化学习算法的样本效率具有深远意义。然而，直接微调预训练策略往往会导致次优性能，这主要是由于离线预训练与在线微调阶段之间的分布偏移所致。具体而言，分布偏移限制了有效在线样本的获取，最终影响了在线微调的性能。为了缩小离线与在线阶段之间的分布偏移，我们提出了Q条件状态熵（QCSE）作为内在奖励。具体来说，QCSE分别考虑每个样本的Q值，最大化所有样本的状态熵。这种方法鼓励探索低频样本，同时惩罚高频样本，并隐式实现状态边际匹配（SMM），从而确保最优性能，解决了基于约束方法的渐近次优性问题。此外，QCSE可以无缝集成到各种强化学习算法中，提升在线微调性能。为验证我们的主张，我们进行了大量实验，并观察到QCSE带来的显著改进（CQL约提升13%，Cal-QL约提升8%）。此外，我们将实验测试扩展到其他算法，证实了QCSE的通用性。
英文摘要：
Studying how to fine-tune offline reinforcement learning (RL) pre-trained policy is profoundly significant for enhancing the sample efficiency of RL algorithms. However, directly fine-tuning pre-trained policies often results in sub-optimal performance. This is primarily due to the distribution shift between offline pre-training and online fine-tuning stages. Specifically, the distribution shift limits the acquisition of effective online samples, ultimately impacting the online fine-tuning performance. In order to narrow down the distribution shift between offline and online stages, we proposed Q conditioned state entropy (QCSE) as intrinsic reward. Specifically, QCSE maximizes the state entropy of all samples individually, considering their respective Q values. This approach encourages exploration of low-frequency samples while penalizing high-frequency ones, and implicitly achieves State Marginal Matching (SMM), thereby ensuring optimal performance, solving the asymptotic sub-optimality of constraint-based approaches. Additionally, QCSE can seamlessly integrate into various RL algorithms, enhancing online fine-tuning performance. To validate our claim, we conduct extensive experiments, and observe significant improvements with QCSE (about 13% for CQL and 8% for Cal-QL). Furthermore, we extended experimental tests to other algorithms, affirming the generality of QCSE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
