离线强化学习的轻度保守正则化评估
Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning
AI总结：
针对离线强化学习中的分布偏移与过估计问题，提出MCRE框架和MCRQ算法，通过在Bellman backup中结合TD误差与行为克隆项平衡保守性与性能，在基准数据集上优于现有强基线算法。
摘要：
离线强化学习（RL）旨在从静态数据集中学习最优策略，无需进一步环境交互。核心挑战是学习策略与行为策略间的分布偏移，导致分布外（OOD）动作和过估计。为防止严重过估计，价值函数须保持保守；但过度保守会阻碍性能提升。为此，我们提出轻度保守正则化评估（MCRE）框架，通过在Bellman backup中结合时序差分（TD）误差与行为克隆项，平衡保守性与性能。基于此，我们开发了轻度保守正则化Q学习（MCRQ）算法，将MCRE集成至离线actor-critic框架中。实验表明，MCRQ在基准数据集上优于强基线与最先进离线RL算法。
英文摘要：
Offline reinforcement learning (RL) seeks to learn optimal policies from static datasets without further environment interaction. A key challenge is the distribution shift between the learned and behavior policies, leading to out-of-distribution (OOD) actions and overestimation. To prevent gross overestimation, the value function must remain conservative; however, excessive conservatism may hinder performance improvement. To address this, we propose the mildly conservative regularized evaluation (MCRE) framework, which balances conservatism and performance by combining temporal difference (TD) error with a behavior cloning term in the Bellman backup. Building on this, we develop the mildly conservative regularized Q-learning (MCRQ) algorithm, which integrates MCRE into an off-policy actor-critic framework. Experiments show that MCRQ outperforms strong baselines and state-of-the-art offline RL algorithms on benchmark datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
