arXiv 2405.199092024-07-15cs.LGcs.AIcs.RO

面向离线强化学习的自适应优势引导策略正则化

Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning

National University of Defense Technology（国防科技大学）；Shanghai Institute of Microsystem and Information Technology（上海微系统与信息技术研究所）；Chinese Academy of Sciences（中国科学院）；Delft University of Technology（代尔夫特理工大学）

Tenglong Liu, Yang Li, Yixing Lan, Hao Gao, Wei Pan, Xin Xu

备注：: ICML 2024, 19 pages

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对离线强化学习中分布外问题导致现有策略正则化方法过度保守的缺陷，提出结合VAE的自适应优势引导策略正则化方法A2PR，理论上保障策略提升，在D4RL等基准上取得最优性能。

摘要：

在离线强化学习中，分布外（OOD）问题十分突出。为解决该问题，现有方法通常通过策略正则化约束所学策略，但这类方法常存在不必要的保守性问题，阻碍策略提升，原因是它们无差别地将生成离线数据集的行为策略的所有动作都用作约束，该问题在数据集质量欠佳时尤为明显。为此，我们提出自适应优势引导策略正则化（Adaptive Advantage-guided Policy Regularization，A2PR），该方法从结合VAE（变分自编码器）的增强行为策略中获取高优势动作，以引导所学策略。A2PR能够筛选出有别于数据集中现有动作的高优势动作，同时仍能有效维持对分布外动作的保守性，这是通过利用VAE生成匹配数据点分布样本的能力实现的。我们从理论上证明了行为策略的提升是有保障的，此外该方法还能有效缓解价值高估问题，且性能差距有界。在实验方面，我们在D4RL基准上开展了一系列实验，A2PR展现出了当前最优的性能。此外，在额外的次优混合数据集上的实验结果表明，A2PR表现出更优异的性能。代码可在https://github.com/ltlhuuu/A2PR获取。

英文摘要：

In offline reinforcement learning, the challenge of out-of-distribution (OOD) is pronounced. To address this, existing methods often constrain the learned policy through policy regularization. However, these methods often suffer from the issue of unnecessary conservativeness, hampering policy improvement. This occurs due to the indiscriminate use of all actions from the behavior policy that generates the offline dataset as constraints. The problem becomes particularly noticeable when the quality of the dataset is suboptimal. Thus, we propose Adaptive Advantage-guided Policy Regularization (A2PR), obtaining high-advantage actions from an augmented behavior policy combined with VAE to guide the learned policy. A2PR can select high-advantage actions that differ from those present in the dataset, while still effectively maintaining conservatism from OOD actions. This is achieved by harnessing the VAE capacity to generate samples matching the distribution of the data points. We theoretically prove that the improvement of the behavior policy is guaranteed. Besides, it effectively mitigates value overestimation with a bounded performance gap. Empirically, we conduct a series of experiments on the D4RL benchmark, where A2PR demonstrates state-of-the-art performance. Furthermore, experimental results on additional suboptimal mixed datasets reveal that A2PR exhibits superior performance. Code is available at https://github.com/ltlhuuu/A2PR.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。