arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.07730cs.LGcs.AIcs.RO
收藏

DEAS:面向可扩展离线强化学习的动作序列解耦值学习

DEAS: DEtached value learning with Action Sequence for Scalable Offline RL

KAIST(韩国科学技术院);UC Berkeley(伯克利大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);NVIDIA(英伟达)

Changyeon Kim, Haeone Lee, Younggyo Seo, Kimin Lee, Yuke Zhu

AI总结:

DEAS提出利用动作序列进行解耦值学习,解决离线强化学习长时程决策中的值高估问题,在OGBench及大规模VLA模型任务上显著提升性能。

摘要:

离线强化学习(RL)提供了一种无需昂贵的在线交互即可训练智能体的有吸引力的范式。然而,当前方法在处理复杂、长时程的序列决策方面仍然存在困难。在这项工作中,我们引入了动作序列解耦值学习(DEAS),一个简单而有效的离线强化学习框架,利用动作序列进行值学习。这些时间上扩展的动作比单步动作提供更丰富的信息,并且可以通过半马尔可夫决策过程Q学习在选项框架下进行解释,从而通过一次考虑更长的序列来缩短有效规划范围。然而,在演员-评论家算法中直接采用这样的序列会导致过度的值高估,我们通过解耦值学习来解决这个问题,该学习将值估计引导至离线数据集中实现高回报的分布内动作。我们证明DEAS在OGBench的复杂、长时程任务上持续优于基线,并且可以应用于增强预测动作序列的大规模视觉-语言-动作模型的性能,在RoboCasa Kitchen模拟任务和真实世界操作任务中显著提升性能。

英文摘要:

Offline reinforcement learning (RL) presents an attractive paradigm for training intelligent agents without expensive online interactions. However, current approaches still struggle with complex, long-horizon sequential decision making. In this work, we introduce DEtached value learning with Action Sequence (DEAS), a simple yet effective offline RL framework that leverages action sequences for value learning. These temporally extended actions provide richer information than single-step actions and can be interpreted through the options framework via semi-Markov decision process Q-learning, enabling reduction of the effective planning horizon by considering longer sequences at once. However, directly adopting such sequences in actor-critic algorithms introduces excessive value overestimation, which we address through detached value learning that steers value estimates toward in-distribution actions that achieve high return in the offline dataset. We demonstrate that DEAS consistently outperforms baselines on complex, long-horizon tasks from OGBench and can be applied to enhance the performance of large-scale Vision-Language-Action models that predict action sequences, significantly boosting performance in both RoboCasa Kitchen simulation tasks and real-world manipulation tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑