离线目标条件强化学习与准度量表示
Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations
AI总结：
本文提出了一种结合对比表示和时间距离的离线目标条件强化学习方法，利用准度量表示空间结构和约束，实现最优目标到达，提升缝合任务和高维环境下的性能。
摘要：
针对目标条件强化学习（GCRL）的方法通常利用学习的状态表示来提取达到目标的策略。两种表示结构框架已产生了特别有效的GCRL算法：（1）对比表示，其中方法通过对比目标进行推理来学习“后继特征”；（2）时间距离，将表示空间中的（准度量）距离与状态到目标的传输时间联系起来。我们提出了一种方法，通过准度量表示空间的结构（三角不等式）和适当的附加约束，学习能够实现最优目标到达的后继表示。与以往工作不同，我们的方法能够利用准度量距离参数化来学习最优的目标到达距离，即使在数据不完全和随机环境中也能做到。这使我们获得了两者的最佳结合：我们保留了蒙特卡洛对比RL方法的稳定性和长周期能力，同时获得了准度量网络参数化的自由缝合能力。在现有的离线GCRL基准上，我们的表示学习目标在缝合任务中提升了性能，这些任务中基于对比学习的方法表现不佳，在噪声大、高维环境中的表现也优于基于准度量网络的方法。
英文摘要：
Approaches for goal-conditioned reinforcement learning (GCRL) often use learned state representations to extract goal-reaching policies. Two frameworks for representation structure have yielded particularly effective GCRL algorithms: (1) *contrastive representations*, in which methods learn "successor features" with a contrastive objective that performs inference over future outcomes, and (2) *temporal distances*, which link the (quasimetric) distance in representation space to the transit time from states to goals. We propose an approach that unifies these two frameworks, using the structure of a quasimetric representation space (triangle inequality) with the right additional constraints to learn successor representations that enable optimal goal-reaching. Unlike past work, our approach is able to exploit a **quasimetric** distance parameterization to learn **optimal** goal-reaching distances, even with **suboptimal** data and in **stochastic** environments. This gives us the best of both worlds: we retain the stability and long-horizon capabilities of Monte Carlo contrastive RL methods, while getting the free stitching capabilities of quasimetric network parameterizations. On existing offline GCRL benchmarks, our representation learning objective improves performance on stitching tasks where methods based on contrastive learning struggle, and on noisy, high-dimensional environments where methods based on quasimetric networks struggle.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
