超越成功率：离线目标条件强化学习的可训练性和可提取性
Beyond Success Rates: Trainability and Extractability for Offline GCRL
AI总结：
研究离线目标条件强化学习中各方法的可训练性和可提取性差距，通过构建可训练性景观及配对提取诊断，发现仅成功率不能确立广泛可提取行为，可训练性景观揭示差距，提取诊断提供低成本视角观察信号成策略的过程。
摘要：
离线目标条件强化学习（GCRL）通常以各方法的最佳调优成功率为基准。该分数衡量可实现的性能，但未揭示学到的目标条件信号能多可靠地提取到策略中：一种方法可能在多种价值学习和提取设置中成功，或仅在狭窄、难找到的配置中成功。我们在共享优势加权回归（AWR）提取器下研究了GCIQL、GCIVL、QRL和CRL这四种方法的这种差距。对于每种方法，我们在影响价值学习和智能体优化的优化器学习率以及控制智能体模仿高优势转换选择性的AWR温度上构建可训练性景观。在AntMaze、Cube和Scene上，我们观察到不同情况：高分方法可能广泛适用或脆弱，而宽泛的相对盆地可能仍低于低绝对上限。为解释这些差异，我们将景观与未来与随机目标区分及AWR权重集中的事后诊断配对。它们与下游成功的关系取决于任务。在AntMaze上，这些诊断解释了景观情况。在Cube和Scene上，目标排序和操纵控制解耦：方法能很好地对目标排序但在下游失败，或尽管未来与随机分离较弱仍通过动作条件优势成功。这些结果表明仅峰值调优成功率并不能确立广泛可提取的目标条件行为。可训练性景观揭示了这一差距，而提取诊断提供了一个低成本视角来观察学到的信号如何成为策略。
英文摘要：
Offline goal-conditioned reinforcement learning (GCRL) is typically benchmarked by the best tuned success rate of each method. This score measures attainable performance, but it does not reveal how reliably a learned goal-conditioned signal can be extracted into a policy: a method could succeed across many value-learning and extraction settings, or only at a narrow, hard-to-find configuration. We study this gap across four methods, GCIQL, GCIVL, QRL, and CRL, under a shared advantage-weighted regression (AWR) extractor. For each method, we construct trainability landscapes over the optimizer learning rate, which affects value learning and actor optimization, and AWR temperature, which controls how selectively the actor imitates high-advantage transitions. Across AntMaze, Cube, and Scene, we observe distinct regimes: high-scoring methods may be broadly accessible or brittle, while broad relative basins may still sit below low absolute ceilings. To interpret these differences, we pair landscapes with post-hoc diagnostics of future-vs-random goal discrimination and AWR weight concentration. Their relationship to downstream success is task-dependent. On AntMaze, where future goals align with path-like progress, these diagnostics explain landscape regimes. On Cube and Scene, goal ranking and manipulation control decouple: methods can rank goals well while failing downstream, or succeed through action-conditioned advantages despite weak future-vs-random separation. These results show that peak tuned success alone does not establish broadly extractable goal-conditioned behavior. Trainability landscapes expose this gap, while extraction diagnostics offer a lower-cost lens on how learned signals become policies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
