一条新路径：利用合成指令与模仿学习扩展视觉-语言导航
A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning
AI总结：
本文提出利用合成导航指令和GAN生成新视角图像构建大规模VLN数据集，并通过模仿学习训练Transformer智能体，在RxR上显著超越现有RL方法。
摘要：
视觉-语言导航（VLN）领域近期的研究训练强化学习（RL）智能体在逼真的环境中执行自然语言导航指令，以此作为迈向能够遵循人类指令的机器人的一步。然而，由于人类指令数据稀缺且训练环境多样性有限，这些智能体在复杂语言接地和空间语言理解方面仍然面临困难。在大规模网络文本和图像-文本数据集上进行预训练已被广泛探索，但改进有限。我们研究利用合成指令进行大规模数据增强。我们获取了500多个以密集采样的360度全景图捕获的室内环境，在这些全景图之间构建导航轨迹，并使用Marky——一个高质量的多语言导航指令生成器——为每条轨迹生成视觉接地的指令。我们还使用图像到图像的GAN从新视角合成图像观测。由此产生的包含420万条指令-轨迹对的数据集比现有的人工标注数据集大两个数量级，并且包含更广泛的环境和视角种类。为了高效利用这种规模的数据，我们训练了一个简单的基于Transformer的智能体，采用模仿学习。在具有挑战性的RxR数据集上，我们的方法优于所有现有的RL智能体，在已见环境中将最先进的NDTW从71.1提高到79.1，在未见测试环境中从64.6提高到66.8。我们的工作指出了一条改进指令遵循智能体的新路径，强调大规模模仿学习和合成指令生成能力的发展。
英文摘要：
Recent studies in Vision-and-Language Navigation (VLN) train RL agents to execute natural-language navigation instructions in photorealistic environments, as a step towards robots that can follow human instructions. However, given the scarcity of human instruction data and limited diversity in the training environments, these agents still struggle with complex language grounding and spatial language understanding. Pretraining on large text and image-text datasets from the web has been extensively explored but the improvements are limited. We investigate large-scale augmentation with synthetic instructions. We take 500+ indoor environments captured in densely-sampled 360 degree panoramas, construct navigation trajectories through these panoramas, and generate a visually-grounded instruction for each trajectory using Marky, a high-quality multilingual navigation instruction generator. We also synthesize image observations from novel viewpoints using an image-to-image GAN. The resulting dataset of 4.2M instruction-trajectory pairs is two orders of magnitude larger than existing human-annotated datasets, and contains a wider variety of environments and viewpoints. To efficiently leverage data at this scale, we train a simple transformer agent with imitation learning. On the challenging RxR dataset, our approach outperforms all existing RL agents, improving the state-of-the-art NDTW from 71.1 to 79.1 in seen environments, and from 64.6 to 66.8 in unseen test environments. Our work points to a new path to improving instruction-following agents, emphasizing large-scale imitation learning and the development of synthetic instruction generation capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
