PlanDQ:通过 D-Conductor 和 Q-Performer 实现分层规划编排
PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer
AI总结:
本文提出面向离线 RL 的分层规划器 PlanDQ,以扩散规划器 D-Conductor 生成子目标、Q-Performer 执行子目标,从而兼顾长视野稀疏奖励与短视野密集奖励任务。
摘要:
尽管离线 RL 近期取得了进展,但尚没有一种统一算法能够在广泛任务中实现卓越性能。尤其是离线价值函数学习,在稀疏奖励、长视野任务中面临困难,原因是难以解决信用分配问题,以及随着任务视野增长而累积的外推误差。另一方面,能够在长视野任务中表现良好的模型是专门为目标条件任务设计的,在短视野、密集奖励场景中通常不如价值函数学习方法。为弥合这一差距,我们提出了一种为离线 RL 设计的分层规划器 PlanDQ。PlanDQ 在高层纳入了一个名为 D-Conductor 的基于扩散的规划器,该规划器通过子目标引导低层策略。在低层,我们使用了一种名为 Q-Performer 的基于 Q-learning 的方法来完成这些子目标。实验结果表明,PlanDQ 在 D4RL 连续控制基准任务以及作为长视野任务的 AntMaze、Kitchen 和 Calvin 上均能取得卓越或具有竞争力的性能。
英文摘要:
Despite the recent advancements in offline RL, no unified algorithm could achieve superior performance across a broad range of tasks. Offline \textit{value function learning}, in particular, struggles with sparse-reward, long-horizon tasks due to the difficulty of solving credit assignment and extrapolation errors that accumulates as the horizon of the task grows.~On the other hand, models that can perform well in long-horizon tasks are designed specifically for goal-conditioned tasks, which commonly perform worse than value function learning methods on short-horizon, dense-reward scenarios. To bridge this gap, we propose a hierarchical planner designed for offline RL called PlanDQ. PlanDQ incorporates a diffusion-based planner at the high level, named D-Conductor, which guides the low-level policy through sub-goals. At the low level, we used a Q-learning based approach called the Q-Performer to accomplish these sub-goals. Our experimental results suggest that PlanDQ can achieve superior or competitive performance on D4RL continuous control benchmark tasks as well as AntMaze, Kitchen, and Calvin as long-horizon tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。