离线强化学习中的策略内Q函数正则化
Offline Reinforcement Learning with On-Policy Q-Function Regularization
AI总结：
本文提出通过SARSA式估计行为策略的Q函数并以其作为正则化目标，替代难以估计的行为策略本身，以解决离线强化学习中的外推误差问题，并在D4RL基准上验证了两种算法的强性能。
摘要：
离线强化学习（RL）的核心挑战在于处理由历史数据集与期望策略之间的分布偏移所引发的（潜在灾难性的）外推误差。先前的大量工作通过隐式或显式地将学习策略正则化至行为策略来应对这一挑战，但在实践中，行为策略难以可靠估计。在本工作中，我们提出转而正则化至行为策略的Q函数，而非行为策略本身，其前提是Q函数可通过SARSA式估计更可靠、更轻松地获得，并能更直接地处理外推误差。我们提出了两种利用估计Q函数进行正则化的算法，并证明它们在D4RL基准上表现出强劲的性能。
英文摘要：
The core challenge of offline reinforcement learning (RL) is dealing with the (potentially catastrophic) extrapolation error induced by the distribution shift between the history dataset and the desired policy. A large portion of prior work tackles this challenge by implicitly/explicitly regularizing the learning policy towards the behavior policy, which is hard to estimate reliably in practice. In this work, we propose to regularize towards the Q-function of the behavior policy instead of the behavior policy itself, under the premise that the Q-function can be estimated more reliably and easily by a SARSA-style estimate and handles the extrapolation error more straightforwardly. We propose two algorithms taking advantage of the estimated Q-function through regularizations, and demonstrate they exhibit strong performance on the D4RL benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
