arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.18892cs.CVcs.RO
收藏

将知识图谱与视觉感知对齐以用于目标导航

Aligning Knowledge Graph with Visual Perception for Object-goal Navigation

ZhejiangLab(浙江实验室);Dalhousie University(达尔豪斯大学)

Nuo Xu, Wen Wang, Rong Yang, Mengjie Qin, Zheyuan Lin, Wei Song, Chunlong Zhang, Jason Gu, Chao Li

AI总结:

针对目标导航中知识图谱与视觉图像不对齐的问题,提出AKGVP方法,通过连续建模分层场景架构并结合视觉-语言预训练实现多模态特征对齐,赋予导航器卓越的零样本导航能力。

摘要:

目标导航是一项具有挑战性的任务,需要基于第一人称视觉观察引导智能体到达特定物体。智能体理解周围环境的能力对于成功找到物体起着至关重要的作用。然而,现有的基于知识图谱的导航器通常依赖离散的类别独热向量和投票计数策略来构建场景的图表示,这导致与视觉图像不对齐。为了提供更准确和连贯的场景描述并解决这一不对齐问题,我们提出了用于目标导航的将知识图谱与视觉感知对齐(AKGVP)方法。在技术上,我们的方法引入了对分层场景架构的连续建模,并利用视觉-语言预训练将自然语言描述与视觉感知对齐。连续知识图谱架构和多模态特征对齐的整合赋予了导航器卓越的零样本导航能力。我们使用 AI2-THOR 模拟器对我们的方法进行了广泛评估,并进行了一系列实验以证明我们导航器的有效性和效率。代码可用:https://github.com/nuoxu/AKGVP。

英文摘要:

Object-goal navigation is a challenging task that requires guiding an agent to specific objects based on first-person visual observations. The ability of agent to comprehend its surroundings plays a crucial role in achieving successful object finding. However, existing knowledge-graph-based navigators often rely on discrete categorical one-hot vectors and vote counting strategy to construct graph representation of the scenes, which results in misalignment with visual images. To provide more accurate and coherent scene descriptions and address this misalignment issue, we propose the Aligning Knowledge Graph with Visual Perception (AKGVP) method for object-goal navigation. Technically, our approach introduces continuous modeling of the hierarchical scene architecture and leverages visual-language pre-training to align natural language description with visual perception. The integration of a continuous knowledge graph architecture and multimodal feature alignment empowers the navigator with a remarkable zero-shot navigation capability. We extensively evaluate our method using the AI2-THOR simulator and conduct a series of experiments to demonstrate the effectiveness and efficiency of our navigator. Code available: https://github.com/nuoxu/AKGVP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑