利用视觉-语言模型进行开放词汇实例分割与跟踪
Leveraging Vision-Language Models for Open-Vocabulary Instance Segmentation and Tracking
AI总结：
提出一种结合 VLM 结构化描述、开放词汇检测与视频分割跟踪的框架，实现动态环境中非标准对象的实时开放词汇实例分割、属性提取与在线轨迹更新。
摘要：
视觉-语言模型（VLM）在视觉理解方面表现出色，但往往缺乏可靠的 grounding 能力和可执行的推理速率。将它们与开放词汇目标检测（OVD）、实例分割和跟踪相结合，可发挥其优势并缓解这些缺陷。我们利用 VLM 生成的结构化描述来识别可见对象实例，收集与应用相关的属性，并提示开放词汇检测器提取相应的边界框；这些边界框被传递给视频分割模型，由其提供分割掩码和跟踪。初始化完成后，该模型直接提取分割掩码，以极小的计算开销实时处理图像流。必要时，可通过生成新的结构化描述和检测结果在线更新轨迹。这结合了 VLM 的描述能力、OVD 的 grounding 能力，以及视频分割的像素级理解和速度。我们跨数据集和机器人平台的评估证明了该方法的广泛适用性，展示了其在动态环境中从非标准对象提取任务特定属性的能力。代码、数据、视频和基准可在 https://vlm-gist.github.io 获取。
英文摘要：
Vision-language models (VLMs) excel in visual understanding but often lack reliable grounding capabilities and actionable inference rates. Integrating them with open-vocabulary object detection (OVD), instance segmentation, and tracking leverages their strengths while mitigating these drawbacks. We utilize VLM-generated structured descriptions to identify visible object instances, collect application-relevant attributes, and inform an open-vocabulary detector to extract corresponding bounding boxes that are passed to a video segmentation model providing segmentation masks and tracking. Once initialized, this model directly extracts segmentation masks, processing image streams in real time with minimal computational overhead. Tracks can be updated online as needed by generating new structured descriptions and detections. This combines the descriptive power of VLMs with the grounding capability of OVD and the pixel-level understanding and speed of video segmentation. Our evaluation across datasets and robotics platforms demonstrates the broad applicability of this approach, showcasing its ability to extract task-specific attributes from non-standard objects in dynamic environments. Code, data, videos, and benchmarks are available at https://vlm-gist.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
