通过生成未来视图语义改进视觉-语言导航
Improving Vision-and-Language Navigation by Generating Future-View Image Semantics
AI总结：
本文提出VLN-SIG方法，通过预训练代理任务让智能体生成未来导航视图语义，并利用辅助损失进行微调，在Room-to-Room和CVDN数据集上取得最先进性能，同时提升可解释性和长路径导航能力。
摘要：
视觉-语言导航（VLN）是一项要求智能体根据自然语言指令在环境中进行导航的任务。在每一步中，智能体通过从一组可导航位置中进行选择来采取下一步行动。在本文中，我们旨在更进一步，探索智能体是否能够从在导航过程中生成潜在未来视图中获益。直观上，人类会基于自然语言指令和周围视图对未来环境的样子产生预期，这将有助于正确导航。因此，为了使智能体具备生成未来导航视图语义的能力，我们首先在智能体的域内预训练期间提出三个代理任务：掩码全景建模（MPM）、掩码轨迹建模（MTM）和带图像生成的动作预测（APIG）。这三个目标分别教会模型预测全景图中缺失的视图（MPM）、预测完整轨迹中缺失的步骤（MTM），以及基于完整指令和导航历史生成下一个视图（APIG）。然后，我们在VLN任务上对智能体进行微调，并引入一个辅助损失，该损失最小化智能体生成的视图语义与下一步真实视图语义之间的差异。实验表明，我们的VLN-SIG在Room-to-Room数据集和CVDN数据集上都取得了新的最先进性能。我们进一步表明，我们的智能体学会了在定性层面填补未来视图中的缺失图像块，这为智能体预测的动作带来了更强的可解释性。最后，我们证明学习预测未来视图语义还能使智能体在更长路径上取得更好的性能。
英文摘要：
Vision-and-Language Navigation (VLN) is the task that requires an agent to navigate through the environment based on natural language instructions. At each step, the agent takes the next action by selecting from a set of navigable locations. In this paper, we aim to take one step further and explore whether the agent can benefit from generating the potential future view during navigation. Intuitively, humans will have an expectation of how the future environment will look like, based on the natural language instructions and surrounding views, which will aid correct navigation. Hence, to equip the agent with this ability to generate the semantics of future navigation views, we first propose three proxy tasks during the agent's in-domain pre-training: Masked Panorama Modeling (MPM), Masked Trajectory Modeling (MTM), and Action Prediction with Image Generation (APIG). These three objectives teach the model to predict missing views in a panorama (MPM), predict missing steps in the full trajectory (MTM), and generate the next view based on the full instruction and navigation history (APIG), respectively. We then fine-tune the agent on the VLN task with an auxiliary loss that minimizes the difference between the view semantics generated by the agent and the ground truth view semantics of the next step. Empirically, our VLN-SIG achieves the new state-of-the-art on both the Room-to-Room dataset and the CVDN dataset. We further show that our agent learns to fill in missing patches in future views qualitatively, which brings more interpretability over agents' predicted actions. Lastly, we demonstrate that learning to predict future view semantics also enables the agent to have better performance on longer paths.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
