GOPlan: 基于学习模型规划的目标条件离线强化学习
GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models
AI总结：
针对无模型离线目标条件强化学习在数据有限和泛化未见目标上的局限，提出基于模型的GOPlan框架，通过优势加权条件GAN预训练先验策略、用学习模型规划生成想象数据微调策略，在多任务中取得最优性能且小数据和分布外目标泛化能力更强。
摘要：
离线目标条件强化学习（GCRL）为从多样化多任务离线数据集中学习通用策略提供了可行范式。尽管近期取得显著进展，但主流的离线GCRL方法（主要是无模型方法）在处理有限数据和泛化到未见目标方面仍面临限制。本研究提出目标条件离线规划（GOPlan），这是一种新型基于模型的框架，包含两个关键阶段：（1）预训练一个能够捕捉多目标数据集内多模态动作分布的先验策略；（2）采用带规划的重分析方法生成想象轨迹以微调策略。具体而言，我们将先验策略建立在优势加权条件生成对抗网络之上，该网络有助于实现清晰的模态分离，缓解分布外（OOD）动作的缺陷。为进一步优化策略，重分析方法通过使用学习到的模型为轨迹内和轨迹间目标进行规划，生成高质量的想象数据。通过全面的实验评估，我们证明GOPlan在多种离线多目标导航和操作任务上取得了最优性能。此外，结果凸显了GOPlan在处理小数据量和泛化到分布外目标方面的卓越能力。
英文摘要：
Offline Goal-Conditioned RL (GCRL) offers a feasible paradigm for learning general-purpose policies from diverse and multi-task offline datasets. Despite notable recent progress, the predominant offline GCRL methods, mainly model-free, face constraints in handling limited data and generalizing to unseen goals. In this work, we propose Goal-conditioned Offline Planning (GOPlan), a novel model-based framework that contains two key phases: (1) pretraining a prior policy capable of capturing multi-modal action distribution within the multi-goal dataset; (2) employing the reanalysis method with planning to generate imagined trajectories for funetuning policies. Specifically, we base the prior policy on an advantage-weighted conditioned generative adversarial network, which facilitates distinct mode separation, mitigating the pitfalls of out-of-distribution (OOD) actions. For further policy optimization, the reanalysis method generates high-quality imaginary data by planning with learned models for both intra-trajectory and inter-trajectory goals. With thorough experimental evaluations, we demonstrate that GOPlan achieves state-of-the-art performance on various offline multi-goal navigation and manipulation tasks. Moreover, our results highlight the superior ability of GOPlan to handle small data budgets and generalize to OOD goals.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
