多步准度量学习用于可扩展的目标导向强化学习
Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning
AI总结：
本文提出了一种基于多步准度量学习的离线目标导向强化学习方法，通过多步蒙特卡洛回报提升长视野任务的性能，并在现实世界机器人操作中实现了多步拼接。
摘要：
在人工智能中，学习如何在环境中达到目标是一个长期挑战，然而在长时间范围内进行推理仍然是现代方法的挑战。关键问题是如何估计观察对之间的时序距离。虽然时序差分方法利用局部更新来提供最优性保证，但它们通常比进行全局更新（例如，使用多步回报）的蒙特卡洛方法表现更差，后者缺乏此类保证。我们展示了如何将这些方法整合到一个实用的离线GCRL方法中，该方法通过多步蒙特卡洛回报来拟合准度量距离。我们展示了我们的方法在长达4000步的模拟任务上优于现有的离线GCRL方法，即使使用视觉观察。我们还证明了我们的方法可以在现实世界的机器人操作领域中实现拼接（Bridge设置）。我们的方法是首个端到端的离线GCRL方法，能够在无标签的视觉观察离线数据集中实现多步拼接，从而在现实世界的操作领域中实现稳健的视野泛化。
英文摘要：
Learning how to reach goals in an environment is a longstanding challenge in AI, yet reasoning over long horizons remains a challenge for modern methods. The key question is how to estimate the temporal distance between pairs of observations. While temporal difference methods leverage local updates to provide optimality guarantees, they often perform worse than Monte Carlo methods that perform global updates (e.g., with multi-step returns), which lack such guarantees. We show how these approaches can be integrated into a practical offline GCRL method that fits a quasimetric distance using a multistep Monte-Carlo return. We show our method outperforms existing offline GCRL methods on long-horizon simulated tasks with up to 4000 steps, even with visual observations. We also demonstrate that our method can enable stitching in the real-world robotic manipulation domain (Bridge setup). Our approach is the first end-to-end offline GCRL method that enables multistep stitching in this real-world manipulation domain from an unlabeled offline dataset of visual observations and demonstrate robust horizon generalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
