arXiv 2506.007952025-09-11cs.LG

弥合时序差分学习与监督学习之间的差距：基于$Q$条件最大化的方法

Closing the Gap between TD Learning and Supervised Learning with $Q$-Conditioned Maximization

Xi’an Jiaotong University（西安交通大学）；Westlake University（西湖大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Georgia Institute of Technology（佐治亚理工学院）；National University of Singapore（新加坡国立大学）；Aalto University School of Electrical Engineering, Aalto University（阿alto大学电气工程学院，阿alto大学）

Xing Lei, Zifeng Zhuang, Shentao Yang, Sheng Xu, Yunhao Luo, Fei Shen, Wenyan Yang, Xuetao Zhang, Donglin Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出GCReinSL，通过Q条件最大化和归一化流估计，赋予监督学习轨迹拼接能力，在离线目标条件强化学习中缩小与TD学习的性能差距。

摘要：

近年来，监督学习（SL）方法因其简单性、稳定性和高效性，已成为离线强化学习（RL）的一种有效方法。然而，近期研究表明，监督学习方法缺乏通常与时序差分（TD）方法相关联的轨迹拼接能力。一个自然的问题浮现出来：\textit{我们如何赋予监督学习方法拼接能力，并缩小其与TD学习之间的性能差距？}为回答这一问题，我们引入了用于离线目标条件强化学习的$Q$条件最大化监督学习，该方法通过$Q$条件策略和$Q$条件最大化，赋予监督学习拼接能力。具体而言，我们提出了\textbf{目标条件强化监督学习}（\textbf{GC\textit{Rein}SL}），其包括：（1）通过归一化流从离线数据集中估计$Q$函数；（2）通过将$Q$函数最大化与期望回归相结合，在数据支持内寻找最大$Q$值。在推理时，我们的策略基于该最大$Q$值选择最优动作。在离线强化学习数据集上的拼接评估实验结果表明，我们的方法优于先前具有拼接能力和目标数据增强技术的监督学习方法。

英文摘要：

Recently, supervised learning (SL) methodology has emerged as an effective approach for offline reinforcement learning (RL) due to their simplicity, stability, and efficiency. However, recent studies show that SL methods lack the trajectory stitching capability, typically associated with temporal difference (TD)-based approaches. A question naturally surfaces: \textit{How can we endow SL methods with stitching capability and close its performance gap with TD learning?} To answer this question, we introduce $Q$-conditioned maximization supervised learning for offline goal-conditioned RL, which enhances SL with the stitching capability through $Q$-conditioned policy and $Q$-conditioned maximization. Concretely, we propose \textbf{G}oal-\textbf{C}onditioned \textbf{\textit{Rein}}forced \textbf{S}upervised \textbf{L}earning (\textbf{GC\textit{Rein}SL}), which consists of (1) estimating the $Q$-function by Normalizing Flows from the offline dataset and (2) finding the maximum $Q$-value within the data support by integrating $Q$-function maximization with Expectile Regression. In inference time, our policy chooses optimal actions based on such a maximum $Q$-value. Experimental results from stitching evaluations on offline RL datasets demonstrate that our method outperforms prior SL approaches with stitching capabilities and goal data augmentation techniques.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。