arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2102.10960cs.LG
收藏

强化学习中基于返回的对比表示学习

Return-Based Contrastive Representation Learning for Reinforcement Learning

University of Science and Technology of China(中国科学技术大学);IIIS, Tsinghua University(清华大学信息科学技术学院);Microsoft Research(微软研究院)

Guoqing Liu, Chuheng Zhang, Li Zhao, Tao Qin, Jinhua Zhu, Jian Li, Nenghai Yu, Tie-Yan Liu

摘要:

最近,已经提出了各种辅助任务来加速深度强化学习(RL)中的表示学习并提高样本效率。然而,现有的辅助任务没有考虑到RL问题的特点,是无监督的。通过利用RL中最重要的反馈信号返回,我们提出了一种新的辅助任务,该任务迫使学习的表示区分具有不同返回的状态-动作对。我们的辅助损失在理论上是合理的学习表示,捕捉一种新形式的状态-动作抽象的结构,在这种结构下,具有相似的回报分布的状态-动作对聚集在一起。在低数据状态下,我们的算法在Atari游戏和DeepMind Control套件中的复杂任务上的表现优于强大的基线,并且在与现有辅助任务相结合时实现了更好的性能。

英文摘要:

Recently, various auxiliary tasks have been proposed to accelerate representation learning and improve sample efficiency in deep reinforcement learning (RL). However, existing auxiliary tasks do not take the characteristics of RL problems into consideration and are unsupervised. By leveraging returns, the most important feedback signals in RL, we propose a novel auxiliary task that forces the learnt representations to discriminate state-action pairs with different returns. Our auxiliary loss is theoretically justified to learn representations that capture the structure of a new form of state-action abstraction, under which state-action pairs with similar return distributions are aggregated together. In low data regime, our algorithm outperforms strong baselines on complex tasks in Atari games and DeepMind Control suite, and achieves even better performance when combined with existing auxiliary tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑