arXiv 2512.201152025-12-23cs.LG

基于样本过滤的高效策略约束离线深度强化学习

Sample-Efficient Policy Constraint Offline Deep Reinforcement Learning based on Sample Filtering

Guangdong Key Laboratory of Intelligent Morphing Mechanisms（广东智能变形机制与自适应机器人重点实验室）；the Harbin Institute of Technology Shenzhen（哈尔滨工业大学深圳研究院）；Faculty of Robot Science and Engineering（机器人科学与工程学院）；Northeastern University（东北大学）；School of Systems Engineering（系统工程学院）；City University of Hong Kong（香港城市大学）

Yuanhao Chen, Qi Liu, Pengbin Chen, Zhongjian Qiao, Yanjie Li

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出一种基于样本过滤的策略约束离线深度强化学习方法，通过筛选高分转移样本提升样本效率和性能。

摘要：

离线强化学习（RL）旨在使用给定的静态转移数据集学习一个最大化预期回报的策略。然而，离线RL面临分布偏移问题。提出的策略约束离线RL方法旨在解决分布偏移问题。在策略约束离线RL训练过程中，确保所学策略与行为策略之间的差异在给定阈值内是重要的。因此，所学策略严重依赖于行为策略的质量。然而，现有策略约束方法存在一个问题：如果数据集中包含许多低回报转移，所学策略将被包含在次优参考策略中，导致学习速度慢、样本效率低和表现差。本文表明，策略约束离线RL中使用数据集所有转移的采样方法可以改进。提出了一种简单但高效的样本过滤方法以提高样本效率和最终表现。首先，通过数据集中的episode的平均回报和平均折扣回报评估转移的分数，并提取高分的转移样本。其次，利用高分转移样本训练离线RL算法。我们在一系列离线RL算法和基准任务中验证了所提出的方法。实验结果表明，所提出的方法优于基线。

英文摘要：

Offline reinforcement learning (RL) aims to learn a policy that maximizes the expected return using a given static dataset of transitions. However, offline RL faces the distribution shift problem. The policy constraint offline RL method is proposed to solve the distribution shift problem. During the policy constraint offline RL training, it is important to ensure the difference between the learned policy and behavior policy within a given threshold. Thus, the learned policy heavily relies on the quality of the behavior policy. However, a problem exists in existing policy constraint methods: if the dataset contains many low-reward transitions, the learned will be contained with a suboptimal reference policy, leading to slow learning speed, low sample efficiency, and inferior performances. This paper shows that the sampling method in policy constraint offline RL that uses all the transitions in the dataset can be improved. A simple but efficient sample filtering method is proposed to improve the sample efficiency and the final performance. First, we evaluate the score of the transitions by average reward and average discounted reward of episodes in the dataset and extract the transition samples of high scores. Second, the high-score transition samples are used to train the offline RL algorithms. We verify the proposed method in a series of offline RL algorithms and benchmark tasks. Experimental results show that the proposed method outperforms baselines.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。