Q-WSL：通过动态规划利用加权监督学习优化目标条件强化学习
Q-WSL: Optimizing Goal-Conditioned RL with Weighted Supervised Learning via Dynamic Programming
AI总结：
针对GCWSL缺乏轨迹拼接能力且次优数据下难以学习最优策略的问题，本文提出融合Q-learning动态规划的Q-WSL框架，在性能、样本效率及鲁棒性上优于现有目标条件方法。
摘要：
最近出现了一类被称为目标条件加权监督学习（GCWSL）的新型先进算法，用于应对目标条件强化学习（RL）中稀疏奖励所带来的挑战。由于其简单性、有效性和稳定性，GCWSL在多种目标到达任务中持续表现出强劲性能。然而，GCWSL方法缺少一项被称为轨迹拼接的关键能力；当测试过程中面对未见过的技能时，该能力对于学习最优策略至关重要。当回放缓冲区主要由次优轨迹填充时，这一局限会变得尤为突出。相比之下，传统的基于时序差分（TD）的RL方法（例如利用动态规划的Q-learning）不会遇到这一问题，但由于价值函数近似的内在困难，它们往往会出现不稳定。本文提出Q-learning Weighted Supervised Learning（Q-WSL），这是一个旨在通过吸收Q-learning中动态规划优势来克服GCWSL局限的新颖框架。Q-WSL利用动态规划结果，在回放缓冲区的不同轨迹之间输出（state, goal）对的最优动作。该方法协同结合Q-learning与GCWSL的优势，有效缓解它们各自的弱点并提升整体性能。在具有挑战性的目标到达任务上的实证评估表明，Q-WSL在性能和样本效率方面均优于其他目标条件方法。此外，在以二元奖励结构和环境随机性为特征的环境中，Q-WSL表现出显著的鲁棒性。
英文摘要：
A novel class of advanced algorithms, termed Goal-Conditioned Weighted Supervised Learning (GCWSL), has recently emerged to tackle the challenges posed by sparse rewards in goal-conditioned reinforcement learning (RL). GCWSL consistently delivers strong performance across a diverse set of goal-reaching tasks due to its simplicity, effectiveness, and stability. However, GCWSL methods lack a crucial capability known as trajectory stitching, which is essential for learning optimal policies when faced with unseen skills during testing. This limitation becomes particularly pronounced when the replay buffer is predominantly filled with sub-optimal trajectories. In contrast, traditional TD-based RL methods, such as Q-learning, which utilize Dynamic Programming, do not face this issue but often experience instability due to the inherent difficulties in value function approximation. In this paper, we propose Q-learning Weighted Supervised Learning (Q-WSL), a novel framework designed to overcome the limitations of GCWSL by incorporating the strengths of Dynamic Programming found in Q-learning. Q-WSL leverages Dynamic Programming results to output the optimal action of (state, goal) pairs across different trajectories within the replay buffer. This approach synergizes the strengths of both Q-learning and GCWSL, effectively mitigating their respective weaknesses and enhancing overall performance. Empirical evaluations on challenging goal-reaching tasks demonstrate that Q-WSL surpasses other goal-conditioned approaches in terms of both performance and sample efficiency. Additionally, Q-WSL exhibits notable robustness in environments characterized by binary reward structures and environmental stochasticity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
