arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.10670cs.CLcs.RO
收藏

OpenFMNav:基于视觉-语言基础模型的开放集零样本目标导航

OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models

Peking University(北京大学);University of Notre Dame(圣母大学)

Yuxuan Kuang, Hai Lin, Meng Jiang

AI总结:

本文提出OpenFMNav框架,利用LLM提取目标、VLM构建语义得分图,实现开放集零样本目标导航,在HM3D基准上超越所有基线并验证了真实世界泛化能力。

摘要:

目标导航(ObjectNav)要求智能体在未见过的环境中导航以找到指定目标。许多先前的方法试图通过依赖监督学习或强化学习来解决该任务,它们在具有闭集物体的有限家庭数据集上进行训练。然而,两个关键挑战仍未解决:理解需要开放集物体的自由格式自然语言指令,以及以零样本方式泛化到新环境。为了解决这两个挑战,本文提出了OpenFMNav,一个基于开放集基础模型的零样本目标导航框架。我们首先释放大型语言模型(LLMs)的推理能力,从自然语言指令中提取满足用户需求的候选目标。然后,我们利用大型视觉语言模型(VLMs)的泛化能力,主动发现并检测场景中的候选物体,构建通用语义得分图(VSSM)。接着,通过在VSSM上进行常识推理,我们的方法能够执行有效的语言引导探索与利用,并最终到达目标。通过利用基础模型的推理和泛化能力,我们的方法能够理解自由格式的人类指令,并在多样环境中执行有效的开放集零样本导航。在HM3D ObjectNav基准上的大量实验表明,我们的方法在所有指标上均超越了所有强基线,证明了方法的有效性。此外,我们进行了真实机器人演示,以验证方法的开放集特性及对真实世界环境的泛化能力。

英文摘要:

Object navigation (ObjectNav) requires an agent to navigate through unseen environments to find queried objects. Many previous methods attempted to solve this task by relying on supervised or reinforcement learning, where they are trained on limited household datasets with close-set objects. However, two key challenges are unsolved: understanding free-form natural language instructions that demand open-set objects, and generalizing to new environments in a zero-shot manner. Aiming to solve the two challenges, in this paper, we propose OpenFMNav, an Open-set Foundation Model based framework for zero-shot object Navigation. We first unleash the reasoning abilities of large language models (LLMs) to extract proposed objects from natural language instructions that meet the user's demand. We then leverage the generalizability of large vision language models (VLMs) to actively discover and detect candidate objects from the scene, building a Versatile Semantic Score Map (VSSM). Then, by conducting common sense reasoning on VSSM, our method can perform effective language-guided exploration and exploitation of the scene and finally reach the goal. By leveraging the reasoning and generalizing abilities of foundation models, our method can understand free-form human instructions and perform effective open-set zero-shot navigation in diverse environments. Extensive experiments on the HM3D ObjectNav benchmark show that our method surpasses all the strong baselines on all metrics, proving our method's effectiveness. Furthermore, we perform real robot demonstrations to validate our method's open-set-ness and generalizability to real-world environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑