基于闭式策略改进算子的离线强化学习
Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
AI总结：
针对离线强化学习的行为约束优化问题，提出基于一阶泰勒近似和高斯混合行为策略建模的闭式策略改进算子，在D4RL基准上性能优于现有最优算法。
摘要：
行为约束策略优化已被证明是解决离线强化学习问题的成功范式。通过利用历史转移数据，智能体训练策略以最大化学习到的价值函数，同时受行为策略约束以避免显著的分布偏移。本文提出了闭式策略改进算子。我们有一项新颖发现：行为约束天然促使使用一阶泰勒近似，从而得到策略目标的线性近似。此外，由于实际数据集通常由异构策略收集，我们将行为策略建模为高斯混合模型，并利用LogSumExp下界和Jensen不等式克服了由此产生的优化难题，最终得到闭式策略改进算子。我们用所提出的新型策略改进算子构建了离线强化学习算法，并在标准D4RL基准上通过实验证明，其性能优于当前最优算法。我们的代码可在https://cfpi-icml23.github.io/获取。
英文摘要：
Behavior constrained policy optimization has been demonstrated to be a successful paradigm for tackling Offline Reinforcement Learning. By exploiting historical transitions, a policy is trained to maximize a learned value function while constrained by the behavior policy to avoid a significant distributional shift. In this paper, we propose our closed-form policy improvement operators. We make a novel observation that the behavior constraint naturally motivates the use of first-order Taylor approximation, leading to a linear approximation of the policy objective. Additionally, as practical datasets are usually collected by heterogeneous policies, we model the behavior policies as a Gaussian Mixture and overcome the induced optimization difficulties by leveraging the LogSumExp's lower bound and Jensen's Inequality, giving rise to a closed-form policy improvement operator. We instantiate offline RL algorithms with our novel policy improvement operators and empirically demonstrate their effectiveness over state-of-the-art algorithms on the standard D4RL benchmark. Our code is available at https://cfpi-icml23.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
