面向离线强化学习的支撑信任区域优化
Supported Trust Region Optimization for Offline Reinforcement Learning
AI总结：
针对离线强化学习的分布外问题，提出支撑信任区域优化（STR），在行为策略支撑内约束策略，保证策略改进，并在 MuJoCo 和 AntMaze 上取得最优性能。
摘要：
离线强化学习面临分布外问题和外推误差。大多数策略约束方法将训练策略的密度正则化到行为策略，这在多数情况下过于严格。我们提出支撑信任区域优化（STR），在策略被约束在行为策略支撑内的条件下进行信任区域策略优化，从而享有更宽松的支撑约束。我们证明，在假设无近似和采样误差时，STR 保证策略严格改进直至收敛到数据集中的最优支撑约束策略。进一步，在包含两种误差的情况下，STR 仍保证每一步的安全策略改进。实验结果验证了 STR 的理论，并展示了其在 MuJoCo 运动域和更具挑战性的 AntMaze 域上的最先进性能。
英文摘要：
Offline reinforcement learning suffers from the out-of-distribution issue and extrapolation error. Most policy constraint methods regularize the density of the trained policy towards the behavior policy, which is too restrictive in most cases. We propose Supported Trust Region optimization (STR) which performs trust region policy optimization with the policy constrained within the support of the behavior policy, enjoying the less restrictive support constraint. We show that, when assuming no approximation and sampling error, STR guarantees strict policy improvement until convergence to the optimal support-constrained policy in the dataset. Further with both errors incorporated, STR still guarantees safe policy improvement for each step. Empirical results validate the theory of STR and demonstrate its state-of-the-art performance on MuJoCo locomotion domains and much more challenging AntMaze domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
