通过奖励感知一致性轨迹蒸馏加速离线强化学习中的扩散规划器
Accelerating Diffusion Planners in Offline RL via Reward-Aware Consistency Trajectory Distillation
AI总结：
本文提出了一种奖励感知一致性轨迹蒸馏方法，通过解耦训练和无噪声奖励信号，在离线强化学习中实现高效扩散规划器，提升性能并加快推理速度。
摘要：
尽管扩散模型在决策任务中取得了显著成果，但其缓慢的推理速度仍然是一个关键限制。一致性模型提供了一种潜在的解决方案，但现有应用在决策任务中要么在行为克隆下难以获得次优演示，要么依赖于在actor-critic框架下复杂的同时训练多个网络。在本工作中，我们提出了一种新的一致性蒸馏方法用于离线强化学习，该方法直接将奖励优化纳入蒸馏过程。我们的方法通过解耦训练和无噪声奖励信号，在单步采样时生成高奖励动作轨迹。在Gym MuJoCo、FrankaKitchen和长 horizon规划基准上的实验证明，我们的方法在状态-of-the-art方法上实现了9.7%的提升，并在推理时间上比扩散方法快了142倍。
英文摘要：
Although diffusion models have achieved strong results in decision-making tasks, their slow inference speed remains a key limitation. While consistency models offer a potential solution, existing applications to decision-making either struggle with suboptimal demonstrations under behavior cloning or rely on complex concurrent training of multiple networks under the actor-critic framework. In this work, we propose a novel approach to consistency distillation for offline reinforcement learning that directly incorporates reward optimization into the distillation process. Our method achieves single-step sampling while generating higher-reward action trajectories through decoupled training and noise-free reward signals. Empirical evaluations on the Gym MuJoCo, FrankaKitchen, and long horizon planning benchmarks demonstrate that our approach can achieve a 9.7% improvement over previous state-of-the-art while offering up to 142x speedup over diffusion counterparts in inference time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
