arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.06793cs.LGcs.AI
收藏

PlanDQ:通过 D-Conductor 和 Q-Performer 实现分层规划编排

PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer

Rutgers University(罗格斯大学);KAIST(韩国科学技术院);National University of Singapore(新加坡国立大学);EPFL(洛桑联邦理工学院)

Chang Chen, Junyeob Baek, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, Sungjin Ahn

AI总结:

本文提出面向离线 RL 的分层规划器 PlanDQ,以扩散规划器 D-Conductor 生成子目标、Q-Performer 执行子目标,从而兼顾长视野稀疏奖励与短视野密集奖励任务。

摘要:

尽管离线 RL 近期取得了进展,但尚没有一种统一算法能够在广泛任务中实现卓越性能。尤其是离线价值函数学习,在稀疏奖励、长视野任务中面临困难,原因是难以解决信用分配问题,以及随着任务视野增长而累积的外推误差。另一方面,能够在长视野任务中表现良好的模型是专门为目标条件任务设计的,在短视野、密集奖励场景中通常不如价值函数学习方法。为弥合这一差距,我们提出了一种为离线 RL 设计的分层规划器 PlanDQ。PlanDQ 在高层纳入了一个名为 D-Conductor 的基于扩散的规划器,该规划器通过子目标引导低层策略。在低层,我们使用了一种名为 Q-Performer 的基于 Q-learning 的方法来完成这些子目标。实验结果表明,PlanDQ 在 D4RL 连续控制基准任务以及作为长视野任务的 AntMaze、Kitchen 和 Calvin 上均能取得卓越或具有竞争力的性能。

英文摘要:

Despite the recent advancements in offline RL, no unified algorithm could achieve superior performance across a broad range of tasks. Offline \textit{value function learning}, in particular, struggles with sparse-reward, long-horizon tasks due to the difficulty of solving credit assignment and extrapolation errors that accumulates as the horizon of the task grows.~On the other hand, models that can perform well in long-horizon tasks are designed specifically for goal-conditioned tasks, which commonly perform worse than value function learning methods on short-horizon, dense-reward scenarios. To bridge this gap, we propose a hierarchical planner designed for offline RL called PlanDQ. PlanDQ incorporates a diffusion-based planner at the high level, named D-Conductor, which guides the low-level policy through sub-goals. At the low level, we used a Q-learning based approach called the Q-Performer to accomplish these sub-goals. Our experimental results suggest that PlanDQ can achieve superior or competitive performance on D4RL continuous control benchmark tasks as well as AntMaze, Kitchen, and Calvin as long-horizon tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑