面向决策Transformer的未来条件无监督预训练
Future-conditioned Unsupervised Pretraining for Decision Transformer
AI总结：
本文提出预训练决策Transformer（PDT），利用未来轨迹信息作为特权上下文进行无监督离线RL预训练，并通过在线微调可控采样高回报行为，在次优数据上优于监督预训练方法。
摘要：
离线强化学习（RL）的最新研究表明，以回报为条件的监督学习是解决决策问题的一种强大范式。尽管前景可观，但回报条件化仅限于使用带有奖励标签的训练数据，因此在从无监督数据中学习时面临挑战。本文旨在利用广义未来条件化，实现从无奖励和次优离线数据中进行高效的无监督预训练。我们提出了预训练决策Transformer（Pretrained Decision Transformer，PDT），这是一种概念上简单的无监督RL预训练方法。PDT利用未来轨迹信息作为特权上下文，在训练期间预测动作。基于当前和未来因素进行决策的能力增强了PDT的泛化能力。此外，这一特性可以很容易地整合到以回报为条件的框架中进行在线微调，方法是为可能的未来分配回报值，并根据各自的回报值采样未来嵌入。实验表明，PDT优于或达到与其监督预训练对应方法相当的性能，尤其是在处理次优数据时。进一步分析表明，PDT能够从离线数据中提取多样化行为，并通过在线微调可控地采样高回报行为。代码可在以下地址获取。
英文摘要：
Recent research in offline reinforcement learning (RL) has demonstrated that return-conditioned supervised learning is a powerful paradigm for decision-making problems. While promising, return conditioning is limited to training data labeled with rewards and therefore faces challenges in learning from unsupervised data. In this work, we aim to utilize generalized future conditioning to enable efficient unsupervised pretraining from reward-free and sub-optimal offline data. We propose Pretrained Decision Transformer (PDT), a conceptually simple approach for unsupervised RL pretraining. PDT leverages future trajectory information as a privileged context to predict actions during training. The ability to make decisions based on both present and future factors enhances PDT's capability for generalization. Besides, this feature can be easily incorporated into a return-conditioned framework for online finetuning, by assigning return values to possible futures and sampling future embeddings based on their respective values. Empirically, PDT outperforms or performs on par with its supervised pretraining counterpart, especially when dealing with sub-optimal data. Further analysis reveals that PDT can extract diverse behaviors from offline data and controllably sample high-return behaviors by online finetuning. Code is available at here.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
