HiFi-CS：利用视觉-语言模型实现面向机器人抓取的开放词汇视觉定位
HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Models
AI总结：
本文提出HiFi-CS，通过分层FiLM融合图文嵌入并结合冻结VLM与轻量解码器，提升复杂场景下机器人抓取的闭集与开放词汇视觉定位性能。
摘要：
通过自然语言与人类交互的机器人可以开启大量应用，例如指代表达抓取合成（Referring Grasp Synthesis，RGS）。给定一个文本查询，RGS会确定一个稳定的抓取位姿，以操作机器人工作空间中被指代的物体。RGS包含两个步骤：视觉定位和抓取位姿估计。近期研究利用强大的视觉-语言模型（Vision-Language Models，VLMs），在真实机器人执行过程中对自由形式的自然语言进行视觉定位。然而，目前仍缺乏在复杂、杂乱环境中且同一物体存在多个实例情况下的比较。本文提出HiFi-CS，其特点是分层应用特征线性调制（Featurewise Linear Modulation，FiLM）来融合图像与文本嵌入，从而针对机器人抓取中遇到的复杂、属性丰富的文本查询增强视觉定位。视觉定位将2D/3D空间中的物体与自然语言输入关联起来，并在闭集词汇和开放词汇两种场景下进行研究。HiFi-CS采用轻量级解码器并结合冻结的VLM，在闭集词汇设置中优于具有竞争力的基线，同时模型尺寸缩小100倍。我们的模型能够有效引导GroundedSAM等开放集物体检测器，以提升开放词汇性能。我们通过使用7自由度机械臂开展真实世界RGS实验验证了该方法，在15个桌面场景中实现了90.33%的视觉定位准确率。我们的代码库位于：https://github.com/vineet2104/hifics
英文摘要：
Robots interacting with humans through natural language can unlock numerous applications such as Referring Grasp Synthesis (RGS). Given a text query, RGS determines a stable grasp pose to manipulate the referred object in the robot's workspace. RGS comprises two steps: visual grounding and grasp pose estimation. Recent studies leverage powerful Vision-Language Models (VLMs) for visually grounding free-flowing natural language in real-world robotic execution. However, comparisons in complex, cluttered environments with multiple instances of the same object are lacking. This paper introduces HiFi-CS, featuring hierarchical application of Featurewise Linear Modulation (FiLM) to fuse image and text embeddings, enhancing visual grounding for complex attribute rich text queries encountered in robotic grasping. Visual grounding associates an object in 2D/3D space with natural language input and is studied in two scenarios: Closed and Open Vocabulary. HiFi-CS features a lightweight decoder combined with a frozen VLM and outperforms competitive baselines in closed vocabulary settings while being 100x smaller in size. Our model can effectively guide open-set object detectors like GroundedSAM to enhance open-vocabulary performance. We validate our approach through real-world RGS experiments using a 7-DOF robotic arm, achieving 90.33\% visual grounding accuracy in 15 tabletop scenes. Our codebase is provided here: https://github.com/vineet2104/hifics
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
