SelfBC：面向离线强化学习的自行为克隆
SelfBC: Self Behavior Cloning for Offline Reinforcement Learning
AI总结：
针对离线强化学习静态策略约束导致策略过度保守的问题，本文提出 SelfBC 动态自约束机制，并从理论与 D4RL MuJoCo 实验验证其性能。
摘要：
离线强化学习中的策略约束方法采用额外的正则化技术，以约束所学策略与离线数据集之间的差异。然而，这些方法往往会产生过于保守、且与行为策略相似的策略，从而限制其性能。我们研究了这一局限，并将其归因于传统约束的静态性质。本文提出一种新颖的动态策略约束，将所学策略限制在由先前所学策略的指数移动平均所生成的样本上。通过将这种自约束机制集成到离策略方法中，该方法在离线设置下有助于学习非保守策略，同时避免策略崩溃。理论结果表明，我们的方法可带来近乎单调改进的参考策略。在 D4RL MuJoCo 领域上的大量实验表明，所提出方法在策略约束方法中达到了最先进性能。
英文摘要：
Policy constraint methods in offline reinforcement learning employ additional regularization techniques to constrain the discrepancy between the learned policy and the offline dataset. However, these methods tend to result in overly conservative policies that resemble the behavior policy, thus limiting their performance. We investigate this limitation and attribute it to the static nature of traditional constraints. In this paper, we propose a novel dynamic policy constraint that restricts the learned policy on the samples generated by the exponential moving average of previously learned policies. By integrating this self-constraint mechanism into off-policy methods, our method facilitates the learning of non-conservative policies while avoiding policy collapse in the offline setting. Theoretical results show that our approach results in a nearly monotonically improved reference policy. Extensive experiments on the D4RL MuJoCo domain demonstrate that our proposed method achieves state-of-the-art performance among the policy constraint methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
