TubeRMC: 基于互约束的管状重建用于弱监督空间-时间视频定位
TubeRMC: Tube-conditioned Reconstruction with Mutual Constraints for Weakly-supervised Spatio-Temporal Video Grounding
AI总结：
TubeRMC通过引入互约束的管状重建方法，在弱监督条件下提升空间-时间视频定位的准确性和一致性。
摘要：
空间-时间视频定位（STVG）旨在在一个未剪辑的视频中定位一个对应于给定语言查询的时空管状区域。这是一个具有挑战性的任务，因为它涉及复杂的视觉-语言理解和时空推理。最近的研究在STVG中探索了弱监督设置，以消除对精细标注（如边界框或时间戳）的依赖。然而，这些方法通常采用简单的后期融合方式，生成与文本描述无关的管状区域，往往导致目标识别失败和目标跟踪不一致。为了解决这一限制，我们提出了一种基于互约束的管状重建（TubeRMC）框架，该框架利用预训练的视觉定位模型生成文本条件的候选管状区域，并通过管状条件重建和时空约束进一步优化。具体来说，我们从时间、空间和时空角度设计了三种重建策略，以全面捕捉丰富的管状-文本对应关系。每种策略都配备了管状条件重建器，利用时空管状作为条件来重建查询中的关键线索。我们进一步在空间和时间提案之间引入互约束，以提高其重建质量。TubeRMC在两个公开基准VidSTG和HCSTVG上优于现有方法。进一步的可视化显示，TubeRMC有效缓解了目标识别错误和不一致的跟踪问题。
英文摘要：
Spatio-Temporal Video Grounding (STVG) aims to localize a spatio-temporal tube that corresponds to a given language query in an untrimmed video. This is a challenging task since it involves complex vision-language understanding and spatiotemporal reasoning. Recent works have explored weakly-supervised setting in STVG to eliminate reliance on fine-grained annotations like bounding boxes or temporal stamps. However, they typically follow a simple late-fusion manner, which generates tubes independent of the text description, often resulting in failed target identification and inconsistent target tracking. To address this limitation, we propose a Tube-conditioned Reconstruction with Mutual Constraints (\textbf{TubeRMC}) framework that generates text-conditioned candidate tubes with pre-trained visual grounding models and further refine them via tube-conditioned reconstruction with spatio-temporal constraints. Specifically, we design three reconstruction strategies from temporal, spatial, and spatio-temporal perspectives to comprehensively capture rich tube-text correspondences. Each strategy is equipped with a Tube-conditioned Reconstructor, utilizing spatio-temporal tubes as condition to reconstruct the key clues in the query. We further introduce mutual constraints between spatial and temporal proposals to enhance their quality for reconstruction. TubeRMC outperforms existing methods on two public benchmarks VidSTG and HCSTVG. Further visualization shows that TubeRMC effectively mitigates both target identification errors and inconsistent tracking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
