用于指称多目标跟踪与分割的时序增强多模态Transformer
Temporal-Enhanced Multimodal Transformer for Referring Multi-Object Tracking and Segmentation
AI总结：
本文提出TenRMOT，通过在编码与解码阶段进行跨模态融合并引入查询更新模块利用时序先验，同时构建Ref-KITTI Segmentation数据集，提升指称多目标跟踪与分割性能。
摘要：
指称多目标跟踪（RMOT）是一项新兴的跨模态任务，旨在定位任意数量的目标对象，并在视频中维持由语言表达所指称对象的身份。这一复杂任务涉及语言与视觉模态的推理，以及目标对象的时序关联。然而，开创性工作仅采用松散的特征融合，并忽视了对被跟踪对象长期信息的利用。在本研究中，我们提出了一种紧凑的基于Transformer的方法，称为TenRMOT。我们在编码和解码阶段都进行特征融合，以充分利用Transformer架构的优势。具体而言，我们在编码阶段逐层增量执行跨模态融合。在解码阶段，我们利用语言引导的查询来探查记忆特征，以准确预测所需对象。此外，我们引入了一个查询更新模块，该模块显式利用被跟踪对象的时序先验信息，以增强其轨迹的一致性。另外，我们提出了一项名为指称多目标跟踪与分割（RMOTS）的新任务，并构建了一个名为Ref-KITTI Segmentation的新数据集。我们的数据集包含18个视频和818条表达，每条表达平均对应10.7个掩码，与大多数现有指称视频分割数据集中通常的单一掩码相比，这构成了更大的挑战。TenRMOT在指称多目标跟踪和分割任务上均表现出优越的性能。
英文摘要：
Referring multi-object tracking (RMOT) is an emerging cross-modal task that aims to locate an arbitrary number of target objects and maintain their identities referred by a language expression in a video. This intricate task involves the reasoning of linguistic and visual modalities, along with the temporal association of target objects. However, the seminal work employs only loose feature fusion and overlooks the utilization of long-term information on tracked objects. In this study, we introduce a compact Transformer-based method, termed TenRMOT. We conduct feature fusion at both encoding and decoding stages to fully exploit the advantages of Transformer architecture. Specifically, we incrementally perform cross-modal fusion layer-by-layer during the encoding phase. In the decoding phase, we utilize language-guided queries to probe memory features for accurate prediction of the desired objects. Moreover, we introduce a query update module that explicitly leverages temporal prior information of the tracked objects to enhance the consistency of their trajectories. In addition, we introduce a novel task called Referring Multi-Object Tracking and Segmentation (RMOTS) and construct a new dataset named Ref-KITTI Segmentation. Our dataset consists of 18 videos with 818 expressions, and each expression averages 10.7 masks, which poses a greater challenge compared to the typical single mask in most existing referring video segmentation datasets. TenRMOT demonstrates superior performance on both the referring multi-object tracking and the segmentation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
