面向长视界离线目标条件强化学习的自适应由粗到细子目标细化
Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
AI总结：
提出CFHRL框架，通过自适应递归细化子目标并基于可学习可达性成本停止细化，解决长视界离线目标条件强化学习中的弱监督和累积误差问题。
摘要：
离线目标条件强化学习（GCRL）在长视界任务中具有挑战性，其中遥远的状态-目标对提供弱监督，且价值估计容易受到累积自举误差的影响。分层方法通过引入中间子目标来缓解这一困难，但固定的时间抽象或固定的层次深度可能与具有不同可达性视界的状态-目标对不匹配。我们提出由粗到细分层目标强化学习（CFHRL），一种完全离线的GCRL框架，在执行前自适应地细化遥远目标。从最终目标开始，CFHRL递归地提出中间目标，这些目标由重放支持的候选训练，并在当前目标被估计为可通过学习的可达性成本局部执行时停止细化。关键思想是，子目标不必是精确的中点或全局最优路径点；它只需要提供可靠的进展并减少剩余到达难度，从而能够在更短的视界上进行后续细化。一个风格化的分析进一步支持近似递归收缩的鲁棒性。在OGBench上的实验表明，在多个长视界任务上取得了显著收益，消融实验验证了所提出的细化和停止机制。
英文摘要：
Offline goal-conditioned reinforcement learning (GCRL) is challenging in long-horizon tasks, where distant state--goal pairs provide weak supervision and value estimates become vulnerable to accumulated bootstrapping errors. Hierarchical methods mitigate this difficulty by introducing intermediate subgoals, but fixed temporal abstractions or fixed hierarchy depths can be mismatched to state--goal pairs with different reachability horizons. We propose Coarse-to-Fine Hierarchical Goal Reinforcement Learning (CFHRL), a fully offline GCRL framework that adaptively refines distant goals before execution. Starting from the final goal, CFHRL recursively proposes intermediate targets, trained from replay-supported candidates, and stops refinement once the current target is estimated to be locally executable by a learned reachability cost. The key idea is that a subgoal need not be an exact midpoint or globally optimal waypoint; it only needs to provide reliable progress and reduce the remaining reaching difficulty, enabling subsequent refinement over shorter horizons. A stylized analysis further supports the robustness of approximate recursive contraction. Experiments on OGBench show substantial gains on several long-horizon tasks, with ablations validating the proposed refinement and stopping mechanisms
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
