学习信任贝尔曼更新：离线强化学习的选择性状态自适应正则化
Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
AI总结：
针对离线强化学习中固定正则化强度难以平衡外推误差与策略性能的问题，提出选择性状态自适应正则化方法，通过状态级信任与选择性约束高质量动作，在D4RL基准上显著优于现有方法。
摘要：
离线强化学习旨在从静态数据集中学习有效的策略。为缓解外推误差，现有研究通常对所有状态统一地正则化价值函数或策略更新。然而，由于数据质量存在显著差异，固定的正则化强度常导致两难困境：较弱的正则化强度无法解决外推误差和价值过估计问题，而较强的正则化强度则使策略学习偏向行为克隆，阻碍了贝尔曼更新可能带来的性能提升。为解决这一问题，我们提出了用于离线强化学习的选择性状态自适应正则化方法。具体而言，我们引入状态自适应正则化系数以信任状态层面的贝尔曼驱动结果，同时选择性地对高质量动作应用正则化，旨在避免对低质量动作的严格约束所导致的性能下降。通过建立代表性价值正则化方法CQL与显式策略约束方法之间的联系，我们有效地将选择性状态自适应正则化扩展到这两种主流的离线强化学习方法中。大量实验表明，所提出的方法在D4RL基准上的离线及离线到在线设置中均显著优于最先进的方法。
英文摘要：
Offline reinforcement learning (RL) aims to learn an effective policy from a static dataset. To alleviate extrapolation errors, existing studies often uniformly regularize the value function or policy updates across all states. However, due to substantial variations in data quality, the fixed regularization strength often leads to a dilemma: Weak regularization strength fails to address extrapolation errors and value overestimation, while strong regularization strength shifts policy learning toward behavior cloning, impeding potential performance enabled by Bellman updates. To address this issue, we propose the selective state-adaptive regularization method for offline RL. Specifically, we introduce state-adaptive regularization coefficients to trust state-level Bellman-driven results, while selectively applying regularization on high-quality actions, aiming to avoid performance degradation caused by tight constraints on low-quality actions. By establishing a connection between the representative value regularization method, CQL, and explicit policy constraint methods, we effectively extend selective state-adaptive regularization to these two mainstream offline RL approaches. Extensive experiments demonstrate that the proposed method significantly outperforms the state-of-the-art approaches in both offline and offline-to-online settings on the D4RL benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
