arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.05413cs.CVcs.RO
收藏

从词语到姿态:利用视觉语言模型增强新物体姿态估计

From Words to Poses: Enhancing Novel Object Pose Estimation with Vision Language Models

TU Wien(维也纳技术大学);UAS Technikum Vienna(维也纳技术高等专业学院)

Tessa Pulli, Stefan Thalhammer, Simon Schwaiger, Markus Vincze

AI总结:

本文提出利用 VLM 语言嵌入与 NeRF 相关性图实现可提示零样本 6D 物体姿态估计,并结合点云配准分析 LERF 在实例级和类别级开放集姿态估计中的适用性。

摘要:

机器人越来越被期望在真实世界场景中进行交互,在这些场景中它们必须不断适应新情况。为了检测并抓取新物体,零样本姿态估计器在没有先验知识的情况下确定姿态。最近,视觉语言模型(VLM)通过在语言输入与图像输入之间建立理解,在机器人应用中取得了显著进展。在我们的工作中,我们利用 VLM 的零样本能力,并将这种能力迁移到 6D 物体姿态估计。我们提出了一种使用语言嵌入进行可提示零样本 6D 物体姿态估计的新颖框架。其思路是基于语言嵌入的 NeRF 重建的相关性图得出物体的粗略位置,并使用点云配准方法计算姿态估计。此外,我们还分析了 LERF 对开放集物体姿态估计的适用性。我们检查了诸如相关性图激活阈值之类的超参数,并研究了实例级和类别级的零样本能力。此外,我们计划在真实世界环境中开展机器人抓取实验。

英文摘要:

Robots are increasingly envisioned to interact in real-world scenarios, where they must continuously adapt to new situations. To detect and grasp novel objects, zero-shot pose estimators determine poses without prior knowledge. Recently, vision language models (VLMs) have shown considerable advances in robotics applications by establishing an understanding between language input and image input. In our work, we take advantage of VLMs zero-shot capabilities and translate this ability to 6D object pose estimation. We propose a novel framework for promptable zero-shot 6D object pose estimation using language embeddings. The idea is to derive a coarse location of an object based on the relevancy map of a language-embedded NeRF reconstruction and to compute the pose estimate with a point cloud registration method. Additionally, we provide an analysis of LERF's suitability for open-set object pose estimation. We examine hyperparameters, such as activation thresholds for relevancy maps and investigate the zero-shot capabilities on an instance- and category-level. Furthermore, we plan to conduct robotic grasping experiments in a real-world setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑