生成式OOD正则化的基于模型的策略优化
Generative OOD-regularized Model-based Policy Optimization
AI总结：
提出GORMPO算法，利用生成式密度估计在稀疏状态-动作空间中限制策略更新到高密度区域，以解决离线强化学习中的分布外动作问题，并在真实医疗数据集和离线RL数据集上优于基线方法。
摘要：
我们研究使用离线强化学习的序贯决策。传统离线RL策略在训练仅依赖稀疏离线表示时可能导致分布外（OOD）动作。为确保在稀疏状态-动作空间中的安全离线策略，我们探索如何将密度估计模型集成到基于模型的RL方法中以避免OOD区域。生成式模型能够显式建模稀疏状态-动作空间中的密度。基于此，我们引入生成式OOD正则化的基于模型的策略优化（GORMPO），一种密度正则化的离线RL算法，使用生成式密度建模将策略更新限制在数据集的高密度区域。此外，我们考察更好的OOD检测是否对应更好的基于模型的离线策略。我们比较了（1）各种密度估计器的OOD检测能力，以及（2）它们在GORMPO框架内在真实医疗数据集和稀疏离线RL数据集上的性能。我们在温和假设下理论上保证了GORMPO的性能。实验上，GORMPO在真实医疗数据集上比最先进的基线方法提升17%，并在离线RL数据集上增强了基础模型。我们的实证发现表明，在动态稳定的环境中，更好的OOD检测通常导致改进的策略，而当动态不确定时，带有保守惩罚的较差密度估计更受青睐。
英文摘要：
We study sequential decision-making with offline reinforcement learning (RL). Traditional offline RL policies may result in out-of-distribution (OOD) actions when training relies only on sparse offline representations. To ensure safe offline policies in a sparse state-action space, we explore how density estimation models can be integrated into model-based RL methods to avoid the OOD regions. Generative models are capable of explicitly modeling the density in sparse state-action spaces. Building on this, we introduce Generative OOD-regularized Model-based Policy Optimization (GORMPO), a density-regularized offline RL algorithm that uses generative density modeling to restrict policy updates to high-density areas of the dataset. Furthermore, we examine whether better OOD detection corresponds to better model-based offline policies. We compare (1) the OOD detection capabilities of various density estimators and (2) their performance within the GORMPO framework on a real-world medical dataset and sparse offline RL datasets. We theoretically guarantee GORMPO's performance under mild assumptions. Empirically, GORMPO outperforms state-of-the-art baselines by 17% on a real-world medical dataset and enhances the base model on the offline RL datasets. Our empirical findings show that better OOD detection generally results in improved policies in environments with stable dynamics, while conservative penalties with poor density estimation are favored when dynamics are uncertain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
