HIQL：以隐状态为动作的离线目标条件强化学习
HIQL: Offline Goal-Conditioned RL with Latent States as Actions
AI总结：
针对离线目标条件强化学习中远距离目标价值函数难估计的问题，提出分层算法HIQL，通过高低层策略利用子目标结构提升鲁棒性，在长时序、高维图像等任务上表现优于现有方法。
摘要：
无监督预训练近来已成为计算机视觉与自然语言处理的基石。在强化学习（RL）领域，目标条件强化学习有望提供一种类似的自监督方法，以利用大量无标注（无奖励）数据。然而，构建能直接从多样的离线数据中学习的有效目标条件强化学习算法颇具挑战，因为难以准确估计远距离目标的精确价值函数。尽管如此，目标达成问题存在固有结构：抵达远距离目标需要先经过更近的子目标。这种结构极具价值，因为评估近目标下动作的质量通常比评估远目标下的更容易。基于这一思路，我们提出一种适用于离线数据的目标条件强化学习分层算法。借助一个无动作价值函数，我们学习两个策略来利用该结构：高层策略将状态视为动作，预测子目标的（隐空间表示）；低层策略预测抵达该子目标的动作。通过理论分析与教学示例，我们证明这种分层分解使我们的方法对价值函数估计中的噪声具有鲁棒性。随后我们将该方法应用于离线目标达成绩准测试，结果表明：我们的方法能解决此前方法难以应对的长时序任务，可扩展至高维图像观测输入，且能轻松利用无动作数据。我们的代码发布于https://seohong.me/projects/hiql/
英文摘要：
Unsupervised pre-training has recently become the bedrock for computer vision and natural language processing. In reinforcement learning (RL), goal-conditioned RL can potentially provide an analogous self-supervised approach for making use of large quantities of unlabeled (reward-free) data. However, building effective algorithms for goal-conditioned RL that can learn directly from diverse offline data is challenging, because it is hard to accurately estimate the exact value function for faraway goals. Nonetheless, goal-reaching problems exhibit structure, such that reaching distant goals entails first passing through closer subgoals. This structure can be very useful, as assessing the quality of actions for nearby goals is typically easier than for more distant goals. Based on this idea, we propose a hierarchical algorithm for goal-conditioned RL from offline data. Using one action-free value function, we learn two policies that allow us to exploit this structure: a high-level policy that treats states as actions and predicts (a latent representation of) a subgoal and a low-level policy that predicts the action for reaching this subgoal. Through analysis and didactic examples, we show how this hierarchical decomposition makes our method robust to noise in the estimated value function. We then apply our method to offline goal-reaching benchmarks, showing that our method can solve long-horizon tasks that stymie prior methods, can scale to high-dimensional image observations, and can readily make use of action-free data. Our code is available at https://seohong.me/projects/hiql/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
