DRMOT：用于RGBD参照多目标跟踪的数据集和框架
DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking
AI总结：
本文提出DRMOT任务，通过融合RGB、深度和语言信息，构建DRSet数据集并提出DRTrack框架，提升多目标跟踪的3D感知能力。
摘要：
参照多目标跟踪（RMOT）旨在基于语言描述来跟踪特定目标，对于交互式人工智能系统如机器人和自动驾驶至关重要。然而，现有RMOT模型仅依赖2D RGB数据，这使得准确检测和关联具有复杂空间语义（例如，'离相机最近的人'）的目标以及在严重遮挡下保持可靠身份变得具有挑战性，因为缺乏显式的3D空间信息。在本工作中，我们提出了一项新的任务，即RGBD参照多目标跟踪（DRMOT），该任务明确要求模型融合RGB、深度（D）和语言（L）模态以实现3D感知跟踪。为了推进DRMOT任务的研究，我们构建了一个定制的RGBD参照多目标跟踪数据集，名为DRSet，旨在评估模型的空间-语义接地和跟踪能力。具体而言，DRSet包含来自187个场景的RGB图像和深度图，以及240个语言描述，其中56个描述包含与深度相关的信息。此外，我们提出了DRTrack，一种基于大规模语言模型（MLLM）的深度参照跟踪框架。DRTrack从联合RGB-D-L输入中执行深度感知的目标接地，并通过结合深度线索来强制鲁棒的轨迹关联。在DRSet数据集上的广泛实验证明了我们框架的有效性。
英文摘要：
Referring Multi-Object Tracking (RMOT) aims to track specific targets based on language descriptions and is vital for interactive AI systems such as robotics and autonomous driving. However, existing RMOT models rely solely on 2D RGB data, making it challenging to accurately detect and associate targets characterized by complex spatial semantics (e.g., ``the person closest to the camera'') and to maintain reliable identities under severe occlusion, due to the absence of explicit 3D spatial information. In this work, we propose a novel task, RGBD Referring Multi-Object Tracking (DRMOT), which explicitly requires models to fuse RGB, Depth (D), and Language (L) modalities to achieve 3D-aware tracking. To advance research on the DRMOT task, we construct a tailored RGBD referring multi-object tracking dataset, named DRSet, designed to evaluate models' spatial-semantic grounding and tracking capabilities. Specifically, DRSet contains RGB images and depth maps from 187 scenes, along with 240 language descriptions, among which 56 descriptions incorporate depth-related information. Furthermore, we propose DRTrack, a MLLM-guided depth-referring tracking framework. DRTrack performs depth-aware target grounding from joint RGB-D-L inputs and enforces robust trajectory association by incorporating depth cues. Extensive experiments on the DRSet dataset demonstrate the effectiveness of our framework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
