面向离线强化学习的目标条件预测编码
Goal-Conditioned Predictive Coding for Offline Reinforcement Learning
AI总结：
本文提出目标条件预测编码（GCPC），通过两阶段框架利用序列模型编码轨迹表示并学习目标条件策略，在三个离线强化学习基准上验证了序列建模对决策任务的显著提升作用。
摘要：
近期工作表明，将决策问题形式化为对离线收集轨迹的监督学习是有效的。强大的序列模型（如GPT或BERT）常被用于编码轨迹。然而，对轨迹数据进行序列建模的益处仍不明确。本文研究序列建模是否具备将轨迹压缩为有用表示以增强策略学习的能力。我们采用两阶段框架：首先利用序列模型编码轨迹级表示，然后学习一个以编码表示为输入的目标条件策略。该形式化使我们能够将许多现有的监督式离线强化学习方法视为本框架的特例。在此框架内，我们提出目标条件预测编码（GCPC），这是一种序列建模目标，可产生强大的轨迹表示并带来高性能策略。通过在AntMaze、FrankaKitchen和Locomotion环境上的广泛实证评估，我们观察到序列建模对具有挑战性的决策任务可产生显著影响。此外，我们证明GCPC学习到的目标条件潜在表示编码了未来轨迹，从而在全部三个基准上取得了具有竞争力的性能。
英文摘要：
Recent work has demonstrated the effectiveness of formulating decision making as supervised learning on offline-collected trajectories. Powerful sequence models, such as GPT or BERT, are often employed to encode the trajectories. However, the benefits of performing sequence modeling on trajectory data remain unclear. In this work, we investigate whether sequence modeling has the ability to condense trajectories into useful representations that enhance policy learning. We adopt a two-stage framework that first leverages sequence models to encode trajectory-level representations, and then learns a goal-conditioned policy employing the encoded representations as its input. This formulation allows us to consider many existing supervised offline RL methods as specific instances of our framework. Within this framework, we introduce Goal-Conditioned Predictive Coding (GCPC), a sequence modeling objective that yields powerful trajectory representations and leads to performant policies. Through extensive empirical evaluations on AntMaze, FrankaKitchen and Locomotion environments, we observe that sequence modeling can have a significant impact on challenging decision making tasks. Furthermore, we demonstrate that GCPC learns a goal-conditioned latent representation encoding the future trajectory, which enables competitive performance on all three benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
