arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.02017cs.LG
收藏

自适应Q-辅助用于离线强化学习中的条件监督学习

Adaptive $Q$-Aid for Conditional Supervised Learning in Offline Reinforcement Learning

Jeonghye Kim, Suyoung Lee, Woojun Kim, Youngchul Sung

AI总结:

本文提出QCS方法,结合RCSL的稳定性与Q函数的缝合能力,通过自适应整合Q-辅助到损失函数中,提升离线强化学习性能。

摘要:

离线强化学习(RL)通过返回条件监督学习(RCSL)取得进展,但缺乏缝合能力仍是一个限制。我们引入Q-辅助条件监督学习(QCS),有效结合RCSL的稳定性与Q函数的缝合能力。通过分析Q函数过泛化,损害稳定缝合,QCS根据轨迹返回自适应地将Q-辅助整合到RCSL的损失函数中。实验证明,QCS显著优于RCSL和基于价值的方法,在多样化的离线RL基准测试中一致实现或超过最大轨迹返回。

英文摘要:

Offline reinforcement learning (RL) has progressed with return-conditioned supervised learning (RCSL), but its lack of stitching ability remains a limitation. We introduce $Q$-Aided Conditional Supervised Learning (QCS), which effectively combines the stability of RCSL with the stitching capability of $Q$-functions. By analyzing $Q$-function over-generalization, which impairs stable stitching, QCS adaptively integrates $Q$-aid into RCSL's loss function based on trajectory return. Empirical results show that QCS significantly outperforms RCSL and value-based methods, consistently achieving or exceeding the maximum trajectory returns across diverse offline RL benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑