物理信息值函数学习器用于离线目标条件强化学习
Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning
AI总结：
针对离线目标条件强化学习在数据覆盖有限和长时程任务中的挑战，提出基于程函偏微分方程的物理信息正则化损失，结合分层隐式Q学习形成Eik-HIQL方法，显著提升性能与泛化能力。
摘要：
离线目标条件强化学习（GCRL）在自主导航和移动等领域中具有巨大潜力，这些领域中收集交互数据成本高昂且不安全。然而，由于需要从状态-动作空间覆盖有限的数据集中学习，并需在长时程任务中进行泛化，该领域在实践中仍具挑战性。为应对这些挑战，我们提出了一种基于物理信息（Pi）正则化的值函数学习损失，该损失源自程函偏微分方程（PDE），并在学习到的值函数中引入几何归纳偏置。与主要用于稳定训练的通用梯度惩罚不同，我们的公式基于连续时间最优控制，并鼓励值函数与到达代价结构对齐。所提出的正则化器与基于时间差分（TD）的值函数学习广泛兼容，并可集成到现有离线GCRL算法中。当与分层隐式Q学习（HIQL）结合时，所得到的方法——程函正则化HIQL（Eik-HIQL）——在性能和泛化方面均带来显著提升，尤其在拼接（stitching）场景和大规模导航任务中表现突出。
英文摘要：
Offline Goal-Conditioned Reinforcement Learning (GCRL) holds great promise for domains such as autonomous navigation and locomotion, where collecting interactive data is costly and unsafe. However, it remains challenging in practice due to the need to learn from datasets with limited coverage of the state-action space and to generalize across long-horizon tasks. To improve on these challenges, we propose a \emph{Physics-informed (Pi)} regularized loss for value learning, derived from the Eikonal Partial Differential Equation (PDE) and which induces a geometric inductive bias in the learned value function. Unlike generic gradient penalties that are primarily used to stabilize training, our formulation is grounded in continuous-time optimal control and encourages value functions to align with cost-to-go structures. The proposed regularizer is broadly compatible with temporal-difference-based value learning and can be integrated into existing Offline GCRL algorithms. When combined with Hierarchical Implicit Q-Learning (HIQL), the resulting method, Eikonal-regularized HIQL (Eik-HIQL), yields significant improvements in both performance and generalization, with pronounced gains in stitching regimes and large-scale navigation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
