arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.05080cs.LGcs.AI
收藏

基于语言抽象与预训练表示的语义探索

Semantic Exploration from Language Abstractions and Pretrained Representations

DeepMind(深度思维)

Allison C. Tam, Neil C. Rabinowitz, Andrew K. Lampinen, Nicholas A. Roy, Stephanie C. Y. Chan, DJ Strouse, Jane X. Wang, Andrea Banino, Felix Hill

AI总结:

本文提出利用自然图像描述数据预训练的视觉-语言表示构建语义状态抽象以定义新颖性,从而改善高维3D环境中的强化学习探索,并通过两类任务与消融实验验证其有效性。

摘要:

有效探索是强化学习(RL)中的一项挑战。基于新颖性的探索方法在高维状态空间中可能表现不佳,例如连续的、部分可观测的3D环境。我们通过利用具有语义意义的状态抽象来定义新颖性,从而应对这一挑战;这些抽象可以从受自然语言塑造的学习表示中获得。具体而言,我们评估了在自然图像描述数据集上预训练的视觉-语言表示。我们表明,这些预训练表示能够推动有意义且与任务相关的探索,并提升在3D模拟环境中的性能。我们还通过考察使用来自预训练模型、语言预言机以及若干消融实验的表示所产生的影响,刻画了语言为何以及如何为探索提供有用的抽象。我们在两个截然不同的任务领域中证明了本方法的优势:一个强调识别和操作日常物体,另一个则需要在广阔世界中进行导航式探索。我们的结果表明,使用受语言塑造的表示可以改进各种算法和智能体在具有挑战性的环境中的探索。

英文摘要:

Effective exploration is a challenge in reinforcement learning (RL). Novelty-based exploration methods can suffer in high-dimensional state spaces, such as continuous partially-observable 3D environments. We address this challenge by defining novelty using semantically meaningful state abstractions, which can be found in learned representations shaped by natural language. In particular, we evaluate vision-language representations, pretrained on natural image captioning datasets. We show that these pretrained representations drive meaningful, task-relevant exploration and improve performance on 3D simulated environments. We also characterize why and how language provides useful abstractions for exploration by considering the impacts of using representations from a pretrained model, a language oracle, and several ablations. We demonstrate the benefits of our approach in two very different task domains -- one that stresses the identification and manipulation of everyday objects, and one that requires navigational exploration in an expansive world. Our results suggest that using language-shaped representations could improve exploration for various algorithms and agents in challenging environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑