彭的Q(λ)在离线强化学习中的保守价值估计
Peng's Q($λ$) for Conservative Value Estimation in Offline Reinforcement Learning
AI总结：
本文提出了一种基于离线多步强化学习的算法CPQL，通过保守价值估计替代贝尔曼算子，实现了更优的性能保证和鲁棒性改进。
摘要：
我们提出了一种模型无关的离线多步强化学习（RL）算法，保守彭的Q(λ)（CPQL）。我们的算法将彭的Q(λ)（PQL）算子适应为保守价值估计的替代品，作为贝尔曼算子的替代。到目前为止，这是首个在离线RL中理论和实验证明多步算子在保守价值估计有效性的工作。离线RL中PQL算子的固定点更接近行为策略的价值函数，从而自然诱导隐式的行为正则化。CPQL同时缓解了过度悲观的价值估计，实现了性能优于（或等于）行为策略的性能，并提供了近最优性能保证——这是之前保守方法无法达到的里程碑。在D4RL基准上的广泛数值实验表明，CPQL在一致性和显著性上均优于现有的离线单步基线。除了在离线RL中的贡献外，我们提出的方法还对离线到在线学习框架做出了贡献。使用由CPQL在离线设置中预训练的Q函数，使在线PQL代理能够避免通常在微调开始时观察到的性能下降，并实现鲁棒的性能改进。我们的代码可在https://github.com/oh-lab/CPQL获取。
英文摘要：
We propose a model-free offline multi-step reinforcement learning (RL) algorithm, Conservative Peng's Q($λ$) (CPQL). Our algorithm adapts the Peng's Q($λ$) (PQL) operator for conservative value estimation as an alternative to the Bellman operator. To the best of our knowledge, this is the first work in offline RL to theoretically and empirically demonstrate the effectiveness of conservative value estimation with a \textit{multi-step} operator by fully leveraging offline trajectories. The fixed point of the PQL operator in offline RL lies closer to the value function of the behavior policy, thereby naturally inducing implicit behavior regularization. CPQL simultaneously mitigates over-pessimistic value estimation, achieves performance greater than (or equal to) that of the behavior policy, and provides near-optimal performance guarantees -- a milestone that previous conservative approaches could not achieve. Extensive numerical experiments on the D4RL benchmark demonstrate that CPQL consistently and significantly outperforms existing offline single-step baselines. In addition to the contributions of CPQL in offline RL, our proposed method also contributes to the offline-to-online learning framework. Using the Q-function pre-trained by CPQL in offline settings enables the online PQL agent to avoid the performance drop typically observed at the start of fine-tuning and to attain robust performance improvements. Our code is available at https://github.com/oh-lab/CPQL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
