离线目标条件强化学习的选项感知时间抽象值
Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning
AI总结：
针对离线目标条件强化学习在长视野任务中高级策略生成子目标困难及优势估计符号错误的问题，提出选项感知时间抽象值学习（OTA），通过将时间抽象融入时间差分学习缩短有效视野，显著提升了高级策略在复杂基准任务上的性能。
摘要：
离线目标条件强化学习（GCRL）提供了一种实用的学习范式，在该范式中，无需额外的环境交互，即可从丰富的状态-动作轨迹数据集中训练到达目标的策略。然而，即使采用了诸如 HIQL 等分层策略结构的最新进展，离线 GCRL 在长视野任务中仍然面临困难。为了识别这一挑战的根本原因，我们得出了以下观察见解。首先，性能瓶颈主要源于高级策略无法生成合适的子目标。其次，在长视野情况下学习高级策略时，优势估计的符号经常变得不正确。因此，我们认为改进值函数以产生清晰的优势估计来学习高级策略是至关重要的。在本文中，我们提出了一种简单而有效的解决方案：选项感知时间抽象值学习（简称 OTA），它将时间抽象整合到时间差分学习过程中。通过将值更新修改为选项感知的，我们的方法缩短了有效的视野长度，即使在长视野情况下也能实现更好的优势估计。我们在实验中表明，使用 OTA 值函数学习到的高级策略在 OGBench（一个最近提出的离线 GCRL 基准）的复杂任务上取得了强劲的性能，包括迷宫导航和视觉机器人操作环境。
英文摘要：
Offline goal-conditioned reinforcement learning (GCRL) offers a practical learning paradigm in which goal-reaching policies are trained from abundant state-action trajectory datasets without additional environment interaction. However, offline GCRL still struggles with long-horizon tasks, even with recent advances that employ hierarchical policy structures, such as HIQL. Identifying the root cause of this challenge, we observe the following insight. Firstly, performance bottlenecks mainly stem from the high-level policy's inability to generate appropriate subgoals. Secondly, when learning the high-level policy in the long-horizon regime, the sign of the advantage estimate frequently becomes incorrect. Thus, we argue that improving the value function to produce a clear advantage estimate for learning the high-level policy is essential. In this paper, we propose a simple yet effective solution: Option-aware Temporally Abstracted value learning, dubbed OTA, which incorporates temporal abstraction into the temporal-difference learning process. By modifying the value update to be option-aware, our approach contracts the effective horizon length, enabling better advantage estimates even in long-horizon regimes. We experimentally show that the high-level policy learned using the OTA value function achieves strong performance on complex tasks from OGBench, a recently proposed offline GCRL benchmark, including maze navigation and visual robotic manipulation environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
