可达性加权的离线目标条件重采样
Reachability Weighted Offline Goal-conditioned Resampling
AI总结：
本文针对离线目标条件强化学习中均匀采样产生大量不可达样本的问题，提出可达性加权采样（RWS），利用PU学习训练可达性分类器为采样分配优先级，在多个机器人操作任务上显著提升策略性能。
摘要：
离线目标条件强化学习（RL）依赖于固定数据集，其中许多潜在目标共享相同的状态和动作空间。然而，这些潜在目标并未在收集到的轨迹中被显式表示。为了学习可泛化的目标条件策略，通常使用动态规划方法（如 Q-learning）对目标和状态-动作对进行均匀采样。然而，均匀采样需要一个难以处理的大规模数据集来覆盖所有可能的组合，并且会产生大量不可达的状态-目标-动作对，从而降低策略性能。我们的关键见解是，采样应偏向那些能够实现目标的转移。为此，我们提出了可达性加权采样（Reachability Weighted Sampling，RWS）。RWS 使用一个可达性分类器，该分类器通过正无标记（positive-unlabeled，PU）学习在目标条件状态-动作值上进行训练。该分类器将这些值映射为可达性分数，然后将该分数用作采样优先级。RWS 是一个即插即用模块，可与标准离线 RL 算法无缝集成。在六个复杂的模拟机器人操作任务上的实验（包括使用机械臂和灵巧手的任务）表明，RWS 显著提升了性能。在一个值得注意的案例中，HandBlock-Z 任务上的性能相对于基线提高了近 50%。这些结果表明了可达性加权采样的有效性。
英文摘要：
Offline goal-conditioned reinforcement learning (RL) relies on fixed datasets where many potential goals share the same state and action spaces. However, these potential goals are not explicitly represented in the collected trajectories. To learn a generalizable goal-conditioned policy, it is common to sample goals and state-action pairs uniformly using dynamic programming methods such as Q-learning. Uniform sampling, however, requires an intractably large dataset to cover all possible combinations and creates many unreachable state-goal-action pairs that degrade policy performance. Our key insight is that sampling should favor transitions that enable goal achievement. To this end, we propose Reachability Weighted Sampling (RWS). RWS uses a reachability classifier trained via positive-unlabeled (PU) learning on goal-conditioned state-action values. The classifier maps these values to a reachability score, which is then used as a sampling priority. RWS is a plug-and-play module that integrates seamlessly with standard offline RL algorithms. Experiments on six complex simulated robotic manipulation tasks, including those with a robot arm and a dexterous hand, show that RWS significantly improves performance. In one notable case, performance on the HandBlock-Z task improved by nearly 50 percent relative to the baseline. These results indicate the effectiveness of reachability-weighted sampling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
