利用大型预训练模型进行零样本开放词汇跟踪
Zero-Shot Open-Vocabulary Tracking with Large Pre-Trained Models
AI总结：
本文提出将开放词汇检测器、分割器与光流估计器重构为零样本视频跟踪框架，结合框传播、细化、分割、轨迹终止和重识别，在 UVO 与 BURST 上超越此前最优方法。
摘要：
目标跟踪是机器人感知和场景理解的核心。长期以来，先检测后跟踪一直是特定物体类别目标跟踪的主导范式。近年来，大规模预训练模型在自然场景中检测和分割二维静态图像中的物体及部件方面展现出可喜进展。这就引出了一个问题：我们能否将这些大规模预训练的静态图像模型重新用于开放词汇视频跟踪？本文将一个开放词汇检测器、分割器和密集光流估计器改造为一个可在二维视频中跟踪并分割任意类别物体的模型。我们的方法在单目视频中预测带有相应语言描述的物体和部件轨迹，利用面向静态图像检测和分割的现代大型预训练模型重建了 Tractor 的流程：我们检测开放词汇物体实例，并使用基于光流的运动模型在帧间传播其边界框，利用视觉检测器的边界框回归模块细化传播后的边界框，再用细化后的边界框提示一个开放世界分割器以分割物体。我们根据传播边界框的物体性分数以及前向-后向光流一致性来决定物体轨迹是否终止。我们使用深度特征匹配来跨遮挡重新识别物体。结果表明，我们的模型在多个公认的视频物体分割和跟踪基准上取得了强劲性能，并能在操作数据中生成合理的轨迹。特别是，尽管从未针对跟踪进行显式训练，我们的模型在开放世界物体跟踪和分割基准 UVO 和 BURST 上仍优于此前的最先进方法。我们希望该方法能成为未来研究的一个简单且可扩展的框架。
英文摘要：
Object tracking is central to robot perception and scene understanding. Tracking-by-detection has long been a dominant paradigm for object tracking of specific object categories. Recently, large-scale pre-trained models have shown promising advances in detecting and segmenting objects and parts in 2D static images in the wild. This begs the question: can we re-purpose these large-scale pre-trained static image models for open-vocabulary video tracking? In this paper, we re-purpose an open-vocabulary detector, segmenter, and dense optical flow estimator, into a model that tracks and segments objects of any category in 2D videos. Our method predicts object and part tracks with associated language descriptions in monocular videos, rebuilding the pipeline of Tractor with modern large pre-trained models for static image detection and segmentation: we detect open-vocabulary object instances and propagate their boxes from frame to frame using a flow-based motion model, refine the propagated boxes with the box regression module of the visual detector, and prompt an open-world segmenter with the refined box to segment the objects. We decide the termination of an object track based on the objectness score of the propagated boxes, as well as forward-backward optical flow consistency. We re-identify objects across occlusions using deep feature matching. We show that our model achieves strong performance on multiple established video object segmentation and tracking benchmarks, and can produce reasonable tracks in manipulation data. In particular, our model outperforms previous state-of-the-art in UVO and BURST, benchmarks for open-world object tracking and segmentation, despite never being explicitly trained for tracking. We hope that our approach can serve as a simple and extensible framework for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
