IPD: 通过离线规划蒸馏提升序列策略
IPD: Boosting Sequential Policy with Imaginary Planning Distillation in Offline Reinforcement Learning
AI总结：
IPD通过引入离线规划蒸馏技术，提升离线强化学习中序列策略的决策稳定性和性能。
摘要：
基于决策变换器的序列策略已成为离线强化学习（RL）中的强大范式，但其有效性仍受到静态数据集质量和固有架构限制。具体而言，这些模型往往难以有效整合次优经验，并未能显式规划最优策略。为弥合这一差距，我们提出了Imaginary Planning Distillation（IPD），一种新颖的框架，能够无缝地将离线规划整合到数据生成、监督训练和在线推理中。我们的框架首先从离线数据中学习一个具有不确定性度量和近似最优价值函数的世界模型。这些组件用于识别次优轨迹，并通过模型预测控制（MPC）为其补充可靠的、想象出的最优轨迹。然后在增强的数据集上训练基于Transformer的序列策略，并辅以价值引导的目标，促进最优策略的蒸馏。通过将传统手动调优的回报到去替换为学习的近似最优价值函数，IPD提高了推理中的决策稳定性和性能。在D4RL基准上的实证评估表明，IPD在多样任务中显著优于几种最先进的基于价值和基于变换器的离线RL方法。
英文摘要：
Decision transformer based sequential policies have emerged as a powerful paradigm in offline reinforcement learning (RL), yet their efficacy remains constrained by the quality of static datasets and inherent architectural limitations. Specifically, these models often struggle to effectively integrate suboptimal experiences and fail to explicitly plan for an optimal policy. To bridge this gap, we propose \textbf{Imaginary Planning Distillation (IPD)}, a novel framework that seamlessly incorporates offline planning into data generation, supervised training, and online inference. Our framework first learns a world model equipped with uncertainty measures and a quasi-optimal value function from the offline data. These components are utilized to identify suboptimal trajectories and augment them with reliable, imagined optimal rollouts generated via Model Predictive Control (MPC). A Transformer-based sequential policy is then trained on this enriched dataset, complemented by a value-guided objective that promotes the distillation of the optimal policy. By replacing the conventional, manually-tuned return-to-go with the learned quasi-optimal value function, IPD improves both decision-making stability and performance during inference. Empirical evaluations on the D4RL benchmark demonstrate that IPD significantly outperforms several state-of-the-art value-based and transformer-based offline RL methods across diverse tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
