基于状态空间划分的目标条件离线强化学习
Goal-conditioned Offline Reinforcement Learning through State Space Partitioning
AI总结：
针对离线目标条件强化学习中的分布偏移与长视野多模态问题，提出基于状态空间价值划分的双优势加权方法DAWOG，性能优于现有基准算法，且保证策略不弱于行为策略。
摘要：
离线强化学习（RL）旨在仅使用离线数据集推导序贯决策策略。这是一种难度极大的设定，尤其是在仅含稀疏奖励的给定场景下学习实现多个不同目标或结果时。对于通过监督学习进行目标条件策略的离线学习，已有研究表明，优势加权对数似然损失可保证策略单调提升。本文认为，该方法虽有优势，但仍不足以完全解决分布偏移与多模态问题。后者在长视野任务中尤为严重：由于可能存在多种甚至相互冲突的解决方案，很难找到从某一状态抵达期望目标的唯一最优策略。为应对这些挑战，我们提出一种互补的基于优势的加权方案，引入额外的归纳偏置来源：在对状态空间进行基于价值的划分后，相较于最终目标，那些预计能导向更易抵达的目标区域的动作，其贡献会被进一步提升。实验方面，我们证明所提方法Dual-Advantage Weighted Offline Goal-conditioned RL（双优势加权离线目标条件强化学习，简称DAWOG）在常用基准上优于多种主流离线算法。理论分析方面，我们给出保证：学习到的策略绝不会差于底层行为策略。
英文摘要：
Offline reinforcement learning (RL) aims to infer sequential decision policies using only offline datasets. This is a particularly difficult setup, especially when learning to achieve multiple different goals or outcomes under a given scenario with only sparse rewards. For offline learning of goal-conditioned policies via supervised learning, previous work has shown that an advantage weighted log-likelihood loss guarantees monotonic policy improvement. In this work we argue that, despite its benefits, this approach is still insufficient to fully address the distribution shift and multi-modality problems. The latter is particularly severe in long-horizon tasks where finding a unique and optimal policy that goes from a state to the desired goal is challenging as there may be multiple and potentially conflicting solutions. To tackle these challenges, we propose a complementary advantage-based weighting scheme that introduces an additional source of inductive bias: given a value-based partitioning of the state space, the contribution of actions expected to lead to target regions that are easier to reach, compared to the final goal, is further increased. Empirically, we demonstrate that the proposed approach, Dual-Advantage Weighted Offline Goal-conditioned RL (DAWOG), outperforms several competing offline algorithms in commonly used benchmarks. Analytically, we offer a guarantee that the learnt policy is never worse than the underlying behaviour policy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
