VELMA：用于街景中视觉与语言导航的 LLM 智能体言语化具身框架
VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View
AI总结：
本文提出 VELMA，通过轨迹与全景视觉观察的言语化提示让 LLM 执行街景 VLN，并在两个数据集上较此前最优方法实现 25%–30% 的相对任务完成率提升。
摘要：
在真实环境中进行增量式决策，是具身人工智能领域最具挑战性的任务之一。一个要求尤其高的场景是视觉与语言导航（Vision and Language Navigation，VLN），它需要视觉和自然语言理解能力，以及空间与时间推理能力。具身智能体必须把自己对导航指令的理解，落实到对 Street View 这类真实环境的观察中。尽管 LLM 在其他研究领域已取得令人瞩目的成果，但如何以最佳方式将其与交互式视觉环境连接起来，仍是一个尚未解决的问题。本文提出 VELMA，这是一个具身 LLM 智能体，它将轨迹和视觉环境观察言语化，并以此作为预测下一动作的上下文提示。视觉信息由一条流水线进行言语化：该流水线从人工撰写的导航指令中提取地标，并使用 CLIP 判断这些地标在当前全景视图中的可见性。我们表明，仅使用两个上下文示例，VELMA 就能在 Street View 中成功遵循导航指令。我们进一步在数千个示例上微调该 LLM 智能体，并在两个数据集上使任务完成率相比此前的最优水平取得 25%–30% 的相对提升。
英文摘要：
Incremental decision making in real-world environments is one of the most challenging tasks in embodied artificial intelligence. One particularly demanding scenario is Vision and Language Navigation~(VLN) which requires visual and natural language understanding as well as spatial and temporal reasoning capabilities. The embodied agent needs to ground its understanding of navigation instructions in observations of a real-world environment like Street View. Despite the impressive results of LLMs in other research areas, it is an ongoing problem of how to best connect them with an interactive visual environment. In this work, we propose VELMA, an embodied LLM agent that uses a verbalization of the trajectory and of visual environment observations as contextual prompt for the next action. Visual information is verbalized by a pipeline that extracts landmarks from the human written navigation instructions and uses CLIP to determine their visibility in the current panorama view. We show that VELMA is able to successfully follow navigation instructions in Street View with only two in-context examples. We further finetune the LLM agent on a few thousand examples and achieve 25%-30% relative improvement in task completion over the previous state-of-the-art for two datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
