arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.09056cs.LGcs.AIcs.CVcs.ROstat.ML
收藏

ReCoRe:世界模型的正则化对比表示学习

ReCoRe: Regularized Contrastive Representation Learning of World Model

Toshiba Europe Ltd(东芝欧洲有限公司);University of Cambridge(剑桥大学)

Rudra P. K. Poudel, Harit Pandya, Stephan Liwicki, Roberto Cipolla

AI总结:

本文提出ReCoRe世界模型,通过对比无监督学习与干预不变辅助正则化学习不变表示,提升视觉导航的样本效率、泛化与sim-to-real迁移能力。

摘要:

尽管近期的无模型强化学习(RL)方法已在游戏环境中展现出人类水平的有效性,但它们在视觉导航等日常任务中的成功仍然有限,尤其是在外观发生显著变化时。这一局限源于:(i)样本效率低下,以及(ii)对训练场景的过拟合。为应对这些挑战,我们提出一种世界模型,利用(i)对比无监督学习和(ii)干预不变正则化器来学习不变特征。学习世界动态的显式表示(即世界模型)可提高样本效率,而对比学习则隐式地促使模型学习不变特征,从而提升泛化能力。然而,将对比损失简单整合到世界模型中并不足够,因为基于世界模型的RL方法会独立优化表示学习和智能体策略。为克服这一问题,我们以辅助任务的形式提出干预不变正则化器,例如深度预测、图像去噪、图像分割等,显式地促使模型对风格干预保持不变性。我们的方法优于当前最先进的基于模型和无模型RL方法,并在iGibson基准上评估的分布外点导航任务中取得显著提升。在仅使用视觉观测的情况下,我们进一步证明该方法在点导航方面优于近期由语言引导的基础模型,这对于部署在计算能力有限的机器人上至关重要。最后,我们证明所提出模型在Gibson基准上的感知模块具备出色的仿真到现实迁移能力。

英文摘要:

While recent model-free Reinforcement Learning (RL) methods have demonstrated human-level effectiveness in gaming environments, their success in everyday tasks like visual navigation has been limited, particularly under significant appearance variations. This limitation arises from (i) poor sample efficiency and (ii) over-fitting to training scenarios. To address these challenges, we present a world model that learns invariant features using (i) contrastive unsupervised learning and (ii) an intervention-invariant regularizer. Learning an explicit representation of the world dynamics i.e. a world model, improves sample efficiency while contrastive learning implicitly enforces learning of invariant features, which improves generalization. However, the naïve integration of contrastive loss to world models is not good enough, as world-model-based RL methods independently optimize representation learning and agent policy. To overcome this issue, we propose an intervention-invariant regularizer in the form of an auxiliary task such as depth prediction, image denoising, image segmentation, etc., that explicitly enforces invariance to style interventions. Our method outperforms current state-of-the-art model-based and model-free RL methods and significantly improves on out-of-distribution point navigation tasks evaluated on the iGibson benchmark. With only visual observations, we further demonstrate that our approach outperforms recent language-guided foundation models for point navigation, which is essential for deployment on robots with limited computation capabilities. Finally, we demonstrate that our proposed model excels at the sim-to-real transfer of its perception module on the Gibson benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑